I wrote the recommendation system at Netflix (still in use after 5 years). Primary problem was company politics. Many groups were not happy that one person could write a system that was better in A/B test, had more uptime and cheaper to run. All of it (ML, production, monitoring), was custom code.
I don't keep track of time. That system is old technology. I did confirm from a Netflix employee that they still use it a few months ago. Deep learning, LDA (even one of Xavier's pet projects - k-means), did not do better.
This is what working in a big company is like sometimes. Imagine a person like MaxLeiter working in Netflix. You would think that all staff would be happy that their work became easier. But a minority seem to have a zero-sum mindset.
When I finish my current work I will talk about that also. Some of my older work is speech recognition. Download my thesis.
I always wondered how do you find the best artwork for the movies, is it multi armed bandits with thompson sampling?
In my company navigating politics is always the hard part, the marketing team would love to spam everyone all the time and the product and sales team would love to sell some kind of upgraded recommendation, its hard to push back but with metrics of coverage, ctr, precision, etc we usually kept them quiet with this metrics
@sadikkapadia - Any idea why Recommendation Engine as a Service has not picked up? I realize that building a use-case specific recommendation engine is unique.
However, I am wondering is there a recommendation engine as a service, which is similar to algolia available/possible.
I worked for a recommendation engine as a service company called aggregate knowledge in 2007 when they got their second round of funding, 25m from kleiner. It was a remarkably lousy business, and they didn't do well.
There was really nothing wrong with the concept. A little JavaScript on the page, a bunch of back end magic, the ability to use a larger data pool because you are collecting from multiple sites.... But people wouldn't pay, and the engagement of recommendations was never that good.
The secret we learned was, after a bunch of math and research, was that 'best in this category and 3 closest adjacent categories works so well in retail that a naive algo did very well.
Better than the fancy math, which once associated the Koran with the sports illustrated swim suit edition ( and vice versa ).
There is a massive data sufficiency problem, and no company with the real data would go into this low end business. Small companies can't get enough data to be relevent.
The world is 10 years later, ml is better understood, so it might all be different now.... Pm me if you are interested in further detail
> Any idea why Recommendation Engine as a Service has not picked up?
These kind of services aren't so much exposed to the public and likely don't start at <100 bucks a month, which could be why those services are not that visible. However, there are some e-commerce services going in that direction, such like AgilOne...
It is hard to sell technology to companies when they have their own teams (often using free libraries). Embedded teams are always experts and will often discredit better technology.
Often the only method of testing is A/B. These can easily be manipulated. For instance at Netflix (ignoring more blatant practices), P-hacking (run thousands of simulations and report ones that worked), and HARKing (come up with a hypothesis after the results are known) are rampant. That is part of the reason the recommender has been degrading over the years.
> That is part of the reason the recommender has been degrading over the years.
Netflix is clearly promoting its original shows. Do you think your system is still in use now that they've moved to thumbs rating and percent match score?
I feel for you. It must be a huge disappointment to see what they've done to your work. Netflix recommendation system used to be very good a few years ago.
Amazing story! I don't know if it's true or if you're delusional, but I'm inclined to believe you as it would explain why Netflix recommendation system went downhill instead of improving.
I wrote the recommender at Netflix about 5 years ago (every line of code). Netflix has been degrading it since then. The problem is that many companies are hotbeds of politics over expertise. Recommender, UI, A/B tests, etc are an excellent venue for politics at the expense of the product.
Another example is the "Netflix prize team" at Verizon/Yahoo. They refuse to share data with any other groups (they are afraid of being discovered), leaving other groups literally nothing to do.
In my opinion Youtube recommendations are improving. They are still pretty bad but they are trying interesting approaches.
Can you share any ideas on how you would implement a new recommender these days? Any papers you like? I am trying to learn more about this topic, but good information is hard to come by. The best resources I found so far are presentations from Netflix and Spotify, but they skip over so many details and assume so much knowledge that it's hard to get good results without being able to consult someone with experience.
The future is a long time. Eventually faster computation, larger memory would allow taking smaller and smaller steps during training (coupled with avoiding "bad optima" with stochastic training). All of this would improve robustness of training.
The domain dictates whether degree of opacity (or other attributes), would rule out deep learning.
Netflix recommender does not use deep learning (which is pretty amazing given how badly they have messed it up). From the conversations I had (a couple of years ago), they gave up with it. I'm sure the Youtube team could do a better job on the Netflix data then they managed to do.
5 years ago I wrote the recommendation system that Netflix uses (and has degraded since then). One major problem is in the past certain senior Netflix managers are only interested in self promotion (I would hate to extrapolate to the current ones - even if the extrapolation is reasonable). A/B tests are a perfect device for this. What is a better recommender? They were not interested in improving the product. It is easier to win by politics/lying/obfuscation/omission/plagiarism then come up with better ideas. If the company goes down they move on with a good resume and the games they played (for instance USPTO fraud), are hidden.
Some companies are like this. For instance the "Netflix prize team" at Verizon/Yahoo refuse to share recommendations data with other teams leaving them nothing to do. They work in a bubble and will actively try and remove anyone who might be a competitor.
It's sad that Netflix decided to pursue this work (+ other even more brain-dead projects like rewriting the command-line parser, "switching to OO" -
examples of Xavier's initiatives). They could of 5 years ago pushed the beta system that has 40% extra performance. (I'm currently at a factor of 3 better performance).
Netflix is no different to any other company. Your happiness depends on your boss. And even single teams go through bad patches.
Some background. I wrote Netflix's recommendation system. Netflix later went on to take part in fraud against USPTO which now led to Netflix with no ownership of it's "most important technology". Xavier Amatrain https://www.linkedin.com/in/xamatriain/ was my boss. Xavier went on to hire lots of staff to try and replace my system and was unable to do so. In the end he started claiming credit for systems I had written without understanding the technology. He still presents his incorrect knowledge in public conferences. The result is that for 5 years Netflix has been running my alpha system without improving the technology (various techniques that Xavier tried and k-means, LDA and deep-learning). Netflix never did push a version I had written with 40% greater "click-through" rate, 3 times greater speed and order of magnitude less memory.
One example conversation I had with Xavier after he left Netflix (para-phrasing of course):
Me: How is my system doing?
Xavier: We don't use it anymore. We stopped using your system 1 week after you left.
Me: Really what do you use?
Xavier: I cannot tell you it is secret.
Me: Who wrote it?
Xavier: You don't know him. We hired him after you and he wrote and pushed the new algorithm.
Me: So that new person, wrote a new recommendation system; pushed it into A/B test; got test results and then pushed it into production in 1 week.
Xavier: Yes - he is better than you.
All was this is a lie (and not out of character), because I have been in touch with other engineers post-Netflix and in fact I've talked with Justin on a number of occasions on the phone about how to keep my system running.
Later Xavier invited me to work for him at Quora. He offered in his own words "an unbelievable salary". I of course refused. I met Xavier sometime later and tried to talk to him about a particular bad presentation of his. He got upset and shouted "Fuck Off, I'm not interested in listening to anything you have to say" multiple times. We were on the Facebook campus at the time. He left at that point.
