I do technical consulting for the grocery business. The grocery industry is a lot of fun -- lots of extremely hard technical challenges and interesting CS problems, but also lots of low hanging fruit in the way of painful manual process. Couple this with the fact that the grocery marketplace is dynamic in the extreme. Prices and availability of items in categories like produce change on a daily if not hourly cadence. And it's all about relationships, which is something nearly every startup in this space misses, which is why nearly all of them fail faster and harder than they do in other industries. You won't get far if you're thinking in terms of replacing people with APIs.
Anyway, I'm excited to see what the next ten years will bring here.
> And it's all about relationships, which is something nearly every startup in this space misses, which is why nearly all of them fail faster and harder than they do in other industries.
Could you expand on this? It's not clear from the outside how this would be such a central part of the industry.
Not even just that, in CPG store managers typically have a lot of autonomy in what SKUs get carried, how many placements of each, what shelf locations, etc.
It's not easy for a small to mid-sized brand to try to get their product into stores when each physical brick and mortar store is a pitch in and of itself. So what ends up happening is brands partner with Agencies who specialize in building relationships with each of the store managers in a region and advocating for all of the brands in their portfolio.
I realize this is a step removed from Instacart and delivery services, but the whole industry tends to operate in a similar interpersonal/relationship-based way.
Where are they getting "petabytes" of daily data from? Even if you took every single transaction anywhere near the industry, in all its details, and represented it in a verbose format like JSON or XML, that's still only kilobytes of data times billions of transactions = terabytes.
If they were gathering tons of video, maybe. Or vast numbers of high-precision sensors generating data continuously. But I didn't see anything like that.
It's just hard for any human activity to generate that much data. Is "petabytes" just some kind of hype, or is there something I'm missing?
To me it seems to be more of a volume data point rather than storage. So, it might only be terabytes of data, but for instance in ML algorithms, you could be running that data through the algorithm many times every two minutes. So by the end of the day, you've processed "petabytes" of data. This is just my interpretation of it. I could be wrong to.
1 user searches for a product, and then clicks on 1: although you estimate this transaction generates 1kb of data, in fact this is more on the order of 1kb x the number of candidates, so a few MB.
This is a elegant writeup that explains how instacart works with detail that I feel like I've never seen from other companies that face similar challenges (uber, doordash). Thank you to the authors!
I think this can only be solved by taking humans out of the equation as much as possible in order to lower costs and boost productivity.
To have one person shopping in store and then another one to deliver to your door is low productivity and almost guaranteed to cost more than most people are willing to pay for the service, unless this is in a developing country like India and China where you have both people with Western purchasing power and people that can be paid peanuts.
Same for services like Deliveroo, a driver cannot make that many deliveries per hour in most locations. It has historically worked for things like pizzas because pizzas are very simple, quick, and cheap so you can have someone on minimum wage do deliveries and still be in the black but as soon as you move to things with more overhead and lower margins you hit the wall.
For picking it's already doable. It's possible to build automated warehouses that could pick and pack orders ready for delivery or "click and collect".
Delivery is indeed the difficulty and the solution depends on the coming of age of self-driving cars (level 5). My experience of ordering online from supermarkets is that a guy drives a van around and takes your order to your front door. In suburban areas (i.e. houses not flats) we can imagine self-driving vans stopping in front of your houses and letting you collect your order like a vending machine. This is not possible now because there are no self-driving cars but the instant they become commercially available I bet this will quickly become the norm.
Automated warehouses plus self-driving cars enable a fully automated process from order to delivery. I'm sure we'll get there.
Drones are simpler to fly autonomously but I don't think they are practical for delivering 10s of kilos of stuff to residential areas.
Self driving will take decades before it becomes viable in developing countries. The state of driving there is very different than what the algorithms are being trained for in the US and Germany. We might have to wait until quantum computing is a thing.
I am surprised Uber Eats is losing money given how expensive it is. Is there a source on that? Could it be due to expansion costs in new cities/countries rather than losing on per-order basis?
The moment a competitor starts offering self-driving option for the general public, there won't be any reason to use Uber Car.
A single 10 mile trip back and forth to downtown will easily cost you 100$+ without tips and 8 times out of 10 it'll be in a filthy and smelly car with a driver who doesn't care about the speed limit or the volume or genre of music he's playing in the car.
I don’t understand the purpose of the Item Availability Model. If you’re getting inventory updates from a store at least once a day, can’t you use a more simple approach to stock prediction than a machine learning model? I.e. they can calculate the historic sale velocity of an item and with the current stock number decide whether the item will be there when the shopper arrives? They give the example of seasonal items, but if a customer tried to order egg nog in July, wouldn’t they have inventory data that the nog hasn’t been in stock for months? Maybe I’m overestimating the fidelity or completeness of the data they receive from retailers, and I guess ML is a way to do all this math without expressing it in code.
There's a great aversion to coding up any heuristic or rule system when you can learn about ML and apply it. It's more interesting, flexible and pads resumes. And sometimes it works as well or better than the rule systems, though it's rare.
I asked a similar question of someone who worked on these systems at Instacart, and the answer was that story inventory is often innacurate or incomplete. The stores just haven't needed the degree of accuracy before.
There's also a fun integration problem: food manufacturers essentially lease shelf space and then the manufacturer handles all the inventory/restocking, and the store itself doesn't know or care what's there. So now you have to contact a bunch of third parties to learn what's in the store.
Yep. The inventory drift from wrong shipments and damaged items varies from store to store, but in the store I work at inventory numbers are often off by 5-10%. And much higher numbers aren't unheard of. Most stores completely re-count every item they have quarterly.
Inventory management is something that's easy in theory, and very very hard in practice.
Totally agree, I work on the COVID vaccine distribution/administration for one state in the US. Inventory tracking is very hard, not technically so much as operationally. For example, healthcare workers didn't go to school to learn how to maintain inventory levels, they went to school to deliver healthcare. Our systems have to constantly remind and nag the users to keep reported inventory at least close to reality.
the store does know, or can figure out if they really wanted to.
every vendor is assigned some shelf space and has some restocking day (like every morning for bread) when the shelves are expected to be 100% full.
then they have purchases data from point of sale terminals/online orders.
We deal with a similar problem in construction materials, and it's not that simple.
Situation:
1)Customer picks up product, other falls on the ground and becomes damaged, product -x
Equation: a-b-x where you don't know how much product is damaged
2)Product arrives in a bad batch, x number is affected, and requires manual adjustment, this doesn't happen, or happens incorrectly
Equation: a-b-x where you don't know how much product is damaged
3)Customer picks up X amount, however x-y was registered as a sale
Equation: a-b-y where you don't know how much product is unaccounted for
4)Delivery is expected on x day, however due to traffic/sickness/equipment failure delivery is delayed
Equation: a-0, stock isn't available as it didn't arrive, however the assumption was that product arrived (trivial to fix this one, but I'm laying out scenarios).
You now have four scenarios that are guaranteed to happen around %10 of the time. Issues can be expanded to the manufacturer/border/trade agreements/ thousands of other potential scenarios that disrupt sourcing.
In terms of taking stock, it's not a trivial task to take accurate inventory on a regular basis. It's a manual problem that can only be done in a reliable fashion in most cases through estimation (therefore inaccurate).
The reason why they provide availability ratings is that it provides a clearer picture of what a customer can purchase, and in the event it has a low rating, prompt for potential replacements. It's not binary, it's a case of 'probably' or 'probably not'.
I've seen stock that should have lasted a week disappear in a day, stock mis-allocated(multiple times for the same item from multiple vendors in the same day), large volumes sold incorrectly resulting in stock adjustments, wastage from random occurrences, etc.
I hope this provides a level of insight into the complexities of
There’s usually a fair few features that can be incorporated into demand prediction: trend, seasonality (daily, weekly, monthly..), special events (e.g. Easter eggs selling faster near Easter), weather, restocking time etc.
I imagine you could get decent improvements in prediction accuracy using a more sophisticated ML model.
That said even the simple sale velocity model you describe has learnable parameters (e.g. historical data window length). This would probably be best done using an ML rather than ad hoc approach. An ML approach might be as simple as a logistic model indicating whether an item will be in stock or out of stock based on time of day + stock levels at the start of the day.
In the same way that I find Mealpal infinitely more satisfying than any food delivery, I am over the moon about shopping for groceries online and picking them up in person.
The interaction with the business I am paying is valuable to me. I can - and do - give direct feedback about the service, which improves it.
And if I forgot something, I can also run into the store and grab it, COVID circumstances permitting.
The _same guy_ puts groceries in my car every time. A guy with a job that has health insurance. That matters.
can't highlight this enough. We've tried instacart when the pandemic started, but ended up with our stores express lane service. The online order works pretty well and the grocery shoppers are less likely (by a lot) to forget something and are pretty good at replacing out of stock items. With instacart, it always felt like babystitting the
shopper.
And lastly yes, it helps, that I feel like the ones with the job are less exploited than the IC shoppers....
I stopped using Instacart when they were unable to deliver an order in the time windows I selected and then unilaterally changed the time to a time when I was not going to be at home. I complained and they insisted and eventually the food was left outside for hours until it went bad.
A shortsighted decision on their part. Because right after they took that stupid decision I stopped using their services forever and have told everyone I know to do the same.
Plus, sometimes you want to buy one thing and add extra stuff to your cart in order to meet the minimum order value, and they do not have the one thing you were interested in on stock and you end up wasting your time.
Amazon is much better at estimating stock and much more reliable. Amazon customer support is better and prioritizes long term relationships with customers over short term revenue.
I've had similar issues. I made an instacart order on Sat. night that I selected a 9a->11a Sunday AM window for. I was unexpectedly kept up until 5am due to a personal emergency, and I was unable to adjust the window later, so I was woken up by the delivery at 9a sharp. (and actually by the incessant notifications at 8a sharp).
I've also often wished for a "cancel the entire order if you don't have this item" button. Especially when shopping for a specific recipe where if one ingredient is missing, the rest are useless.
Neat content but the “secret sauce” is really the willingness/ability to take advantages of the employees they call contractors. Throughout human history exploiting people has been one of the most effective business models imaginable and yet somehow Instacart (and others) have figured out a way to lose absolutely astounding amounts of money doing it.
Though quite an interesting solution and an appropriate one for ML, I think, given the sporadic and temporally limited nature of the availability data they receive from the stores. After all, that a store has 100 cans of soda right now doesn't mean they'll have any when my order is being picked in 2 days time.
Another point on this - even if current inventory levels were known precisely (generally they aren’t), whether an item is currently available is not what is of interest. Rather what is of interest is whether an item will be available when an order is filled.
Let's talk about the hard problem here. Control and Flexibility. I'm a regular Instacart user and I don't think I've ever had a perfect order. There's always something wrong in terms of size, quality, or availability. Most of the time it's no big deal but it begs the question, can these problems be eliminated?
Take the perspective of a 3 star chef. You've got two major strategies to work with 1. Control and 2. Flexibility.
The control strategy means you have deep trust in reliable suppliers. You've spent years finding the people who can get you what you need when you need it. You build your menu in consultation with your suppliers and then you check, check, and recheck every single order to make sure the standard doesn't slip. It's incredibly hard.
The flexible strategy means that you adapt your menu daily or nearly daily to the supply you select yourself. Your talent is taking what is available, finding the best, and creating surprise and delight in a clientele who like that kind of thing. This is also incredibly hard.
Basically nowhere in the high-quality-fresh-food world can you 100% rely on supply. And that's the challenge that Instacart faces. It doesn't have any strong relationships with suppliers (it's all third party or arms-reach) and it has almost no ability to be flexible because the orders are set by people who are hard to reach and haven't explained their needs.
There's some light in this tunnel. For example plantoeat.com (recipe and meal planning service) now integrates with instacart, but it doesn't work well at all. Even when it does populate the cart shoppers don't know the recipe associated with what they are buying and aren't incentivized to care.
Throwing layers of ML on top of unpredictable systems isn't a terrible idea, but it's a bandaid. The problems as I've described above are 1. Lack of control and 2. An inability to be flexible because you don't understand the needs you're fulfilling.
The best answer to this is vertical integration. Goodeggs.com is one vertically integrated grocery delivery service. You could also think of it as a net-first grocery store, ghost grocer, whatever. When I buy from them I'm much more likely to get what I ask for. And in the cases where I am ordering a meal kit and they are out of some component, they know what can and cannot be substituted to fulfill the requirements of the recipe.
I don't think Instacart is going anywhere, it does what it does pretty well and I think it will keep getting better. But Amazon was right to buy Whole Foods and until Instacart owns the entirety of the production and distribution chain they won't get to the kind of experience people have been used to when they plan a meal, visit a store, and select for themselves.
Algorithms aside, I just found out that one of the victims in the Colorado shooting was an Instacart shopper.
It hit me real hard thinking that the app user who made the Instacart order never knew why their order was late and may have even gotten upset and asked for a refund, not having the slightest idea about the tragedy that happened behind the scenes, or the risk to their own life had they gone shopping themselves that day instead of ordering online.
"The mother of two was at the supermarket filling an Instacart order, something she did in retirement to help others, he told the newspaper."
I’m not making any value or policy judgments. Anyone who understands a bit of American politics knows how hard the pro-gun lobby fights anything related to curbing gun ownership. I was merely saying that as an American it saddens me that there are all these senseless killings and that I’m not all that hopeful that real change is possible. Maybe I’m a glass half empty kind of guy I dunno.
It's time for the adults to take care of this. I understand the deeply embedded gun culture and stockpiles. But what would you do if you found out porcelain toilets were killing people at this rate? You'd ban them and start programs to subsidize replacements. No, no, actually you'd have a significant percentage of the country insisting you can have thier toilets when you pry them from thier cold, dead hands. I sincerely wish this wasn't the way it is. The problem is generational, I guess. We need to start today by making guns a subject of ridicule among young people by using stats and studies to show there is no net benefit.
That may be true (ok, it is definitely true) but we have to start somewhere. I know a lot of people my age who grew up (like me!) in a house with guns and hunting that do not hunt or own guns. I don't even have anything against responsible gun owning and hunting. I am talking about getting people to understand that going beyond this (carrying a sidearm around all the time, keeping them in your car, thinking you can be Clint Eastwood when you get mugged, etc) all lead to people dying on purpose or accidentally at far higher rates than if we just didn't do this stuff. Gun OWNERSHIP is not the problem. Tons of households in countries all over the world have guns in them. You simply don't see the same level of gun death as in the US. It is because of their culture (e.g. compulsory military service - I'm not a fan but you will learn how to responsibly own and house a firearm and when you could reasonably expect it to be a useful tool vs something that will go off and kill your kid or be stolen and used in a crime later).
Rant over, sorry, I get emotional when I think about this having been mugged at gun point with a pistol that was undoubtedly stolen from a car glove compartment or something during a routine car break in (assumptions, but you know what I mean). I survived the encounter because I didn't try to fumble in my bag for some gun I barely know how to use (in fact the first thing they did was grab my bag and frisk every inch of me). It was a crew, they were all business, and they got me. I walked away with my house keys and my empty wallet (note, they did NOT steal my ID).
So yeah, that sort of thing has happened to me and I still, somehow, don't have fantasies about shooting a bunch of teenagers. Call me crazy.
I’m with you and personally dislike guns to a degree that many find weird but I think the issue is that gun violence is illegal in the first place and so making laws seeking to prevent it is ineffective at best and really just smacks of legislative hubris. I can get an illegal firearm easily, today. I can get an illegal gun much easier than I can a legal one. Most gun violence is committed by criminals. These problems are often denied by advocates for reform, and that doesn’t help their credibility with bipartisan independents wary of authoritarian government.
Exactly. Outside the US, major gun holders are criminals who get them illegally. Even in the US, most criminals have their guns illegally because they can't get one legally after their first felony.
I say let’s try it! Do a gun buy back. Ban new sales of assault rifles and high capacity magazines; institute longer more invasive (effective?) background checks (have a buyer have to see a shrink before buying? Makes sense to me.)
Before moving to Germany I owned some guns for sport: my friends and I would go into the mountains to designated shooting areas and do target practice. If the hoops to get a gun were increased 10 fold it wouldn’t effect me at all.
> If the hoops to get a gun were increased 10 fold it wouldn’t effect me at all.
I imagine this line is annoying people. That you aren't bothered isn't the issue if others are. That you don't see the issue with a delay, or a background check, says more about your one-size-fits-all view than the complexity/simplicity of the problem. Personally I never have menstrual issues so I don't understand what all the fuss is about.
I come from country people, where even today going to the city is an event. And it's almost a holiday to go to "the big city". So a purchase you might be able to go to walk to the store to start a waiting period for might be between two months-apart trips for my family and much more inconvenient.
You also sound like you live in the city so your needs for protection are a lot different than in the country. In the city a pistol with frangible ammo, or a shotgun at home, are the max you'd need. In the country my driveway was 1.5km so different solutions are needed. What's a scary black gun in the city, when in the country, is a way to both keep coyotes from livestock at great distance and defend yourself from people if needed. Being 500m+ from a problem means you need more bullets (larger magazine) than being at 3' from the bad guy in the city, etc.
They're different worlds with different problems.
Look at Canada where the issue recently become polarized but not across party lines. Some nutcase in a surplus police car with a replica uniform drove around a small community gunning people down while the police holed up in the city for safety. He preyed on country people many kilometers from anyone else, with a rifle he'd smuggled up from the USA. Now the city people want to ban scary guns despite that a ban wouldn't have interfered with this shooter, but people from the country want to buy exactly the same gun he used because it's the best-in-class for home defense in those same scenarios.
I apologize for my flippant response. You’re right I didn’t think about the “country folks”. I was merely highlighting that more steps need to be taken at the burden of lawful, sane, well meaning, no-harm-to-anyone gun owners to try to stem the flow of guns into the hands of those that aren’t mentally fit to own them.
A psychologist likely won’t be able to figure out things very quickly you’re right. Perhaps a discussion could be had to find a solution there to prevent more who would do harm with them from getting said guns.
I also said nothing about gun-shows. They perform the same background checks but have been very lax in the past. And then private gun sales on recently required further vetting at least in my home state of Oregon.
In the countryside do you really need an AR15 for self defense? Surely a shotgun at home makes more sense. And then to protect livestock a high powered rifle?
I don’t know what all the solutions might be but I do know that I am willing to pay in much longer lead times to get a gun if it means the next mass shooter is less likely to.
> I don’t know what all the solutions might be but I do know that I am willing to pay in much longer lead times to get a gun if it means the next mass shooter is less likely to.
I wonder how many shooters would have been stopped. I know one recently went shooting the day he received his guns but I don't know if he had enough guns before that. Had he been denied, what would he have done?
> In the countryside do you really need an AR15 for self defense? Surely a shotgun at home makes more sense. And then to protect livestock a high powered rifle?
Defense, against animals, likely no. Almost all animals except moose and bears are smart enough to avoid people. But defense of livestock and property, yes.
Are you saying an AR-15 isn't powerful enough for animals? It'd a moose at defensive range, but isn't what I'd want for hunting. But it'll kill any cougar-sized animal or smaller in a single hit, even at great range.
If bears are an issue you might want a 30-06, or an AR-10 to get the benefit of modern gear.
For defense against people it depends. I think the crazed shooter like in Canada is a pretty rare circumstance and the likely scenario is your wife defending the family against opportunistic robbers. In that case you want something quick and reliable, like a semi-auto battle rifle, with a high-capacity magazine. You aren't shooting at 600m, but perforating a car at 30m. But country defense is handled by the idea of weapons. Simply by being armed you and your neighbors probably won't need to use it, making the tactics of what is ideal somewhat irrelevant. Being that a shotgun is just an anti-human defensive weapon (it'll just anger a bear, very badly) it might not be worth the cost unless you use it for birding as well. The main benefit is the frangibility of the shot and lower wall-penetration which is more of a city concern.
The Canadian shooting happened and took so long to stop because even country people are largely disarmed there. You have magazine size limits, gun limits, have to store the gun locked and unloaded, and you can't intend to use the gun for defense!?
(I don't own a gun, not have intention of getting one myself)
I get that for big cities with large police presence, but the US is very wide and has countless suburbs and small towns with very little law enforcement present. Please look up YouTube, you'll find many single moms and small families whose lives were saved because of guns, where it took law enforcement forever to reach them.
I don't think seeing a 'shrink' would make much difference except for those with very obvious conditions than the gun shop would refuse selling to them anyway. Most psychiatrists are nothing like the movies portray them to be, and they can't deduce everything about you from a couple hours of talking unless you're in a real dire state of mind. Hell, most of the good ones even refuse to take new patients or have 6 months long waiting list, and only the new inexperienced ones are easy to get hold of. Source: Tried to find a psychiatrist myself couple years ago for depression.
You seem like a reasonable mentally stable person with a respect for the power that a firearm brings. The issue is that with all tools be they an AR15 (before moving to the Germany I wanted to build an AR10 with a 7.62 caliber barrel — that would have been fun to go plinking with) or nuclear weapons or anything in between it’s how do we prevent the less sane, those that would do the innocent harm?
I don’t envy the politicians here. Their job is very hard. They have to weigh legitimate uses for self defense, recreation, hunting, etc., against those that would use them to kill many on a mass scale.
One thing that did cause alarm in your most recent post was this: “ You aren't shooting at 600m, but perforating a car at 30m.” Why is anyone shooting at a car? In the US some states if the suspect is fleeing you can’t pursue, but this is getting into the weeds of things. Suffice it to say that this is a thorny issue with no real easy fix or silver bullet (pun not intended). I do maintain that if the time to get a gun went from same day to a month I personally would be willing to pay that cost in waiting.
Feel free to continue this over email: Email (b64 encoded) YWxleGFuZGFyIGRvdCBuYXJheWFuIGF0IGhleSBkb3QgY29t
HN curiosity: I post the most practical gun-control things I can think of that veer rather draconian yet get down-voted because they're not enough? If you disagree, perhaps state why?
Yes. Each time shooting happens I see 'analysts' thinking everything is full-auto only because of how it looks, and that automatically discredits them.
Anyway, I'm excited to see what the next ten years will bring here.