Software estimation is a joke because there's no penalty for underestimating.
Compare movie production estimation. There are companies which will sell a film production a completion bond. The bonding company guarantees that a releasable film will come out, or your money back.
Here's an overview of completion bonds.[1] A completion bond costs about 2% of production cost. It's typically used for indie films with budgets between US$3 million and US$70 million. The completion bond company will cover some overrun, but if it's too big, they will fire the director and take over the production. That's what gives this real teeth. Movies where this happens tend not to be great, but tend to break even, so everybody gets paid.
So how do completion bond companies do estimates? History.
They have the actual expenses from many completed films. This info is at a highly detailed level, with costs for each shot. So, when they get a script with a car chase scene, they go to their database, look up "car chase", and pull out how much the last hundred movie car chases cost.
They also have info about directors, producers, and actors, and how much they tend to cost to shoot a scene relative to their peers.
It's not rocket science. It's insurance underwriting.
> Software estimation is a joke because there's no penalty for underestimating.
In my experience, while this might be true in some companies, I noticed that most of the time the issue with estimations is the lack of clarity (=what do I really want?) and enough context (what's the impact in terms of scope to reach what I want?).
Most of commercial software, for whatever weird reason, keeps on being compared to other industries and repeatedly this fails.
It's not like building a car, it's not like building a house, it's not like shooting a movie, etc.
> I noticed that most of the time the issue with estimations is the lack of clarity (=what do I really want?) and enough context (what's the impact in terms of scope to reach what I want?).
From what I've seen, this up front design ends up being a good portion of the design time. The cheat is that people don't include this upfront design in the time estimate. They say "the specs are still being set, we can't start yet" rather than "we're delayed in setting these new specs", even though the end date moves similarly.
This is because of the way most software is built. If you are using event modeling, it's very easy to estimate feature costs. The only exception is when you are truly innovating new algorithms so there is not a historical precedent you can refer to. This is not common work in the industry however, most software projects are variations of previous themes.
Sorry but I don't buy that a specific process can solve that problem. DDD can actually add quite some unnecessary complexity, not only because it requires that everyone is skilled to know how to apply it, but also because sometimes you really feel you're overenginerring things just to follow ... a model.
I know for certain that a specific way of doing things can and will improve a process. Nothing to say there. But "solving" a problem? Not sure about it.
Plus, sometimes you as a company don't know really what decision to make. PMs try to figure it out, get stuck, unstuck, and then suddenly a big company comes up with something new and your market is shaken. So again "what do I want?" which has an impact over everyone in the chain, because you must design something but keeping in mind that "it might be different in 6 months". So extensible, scalable, but hell please don't overengineer it, and yet I want it to run on K8s, but if a customer needs it in their data centers, we need to find a way to make it work there too, in a way that scales, but we can't give them a k8s cluster. It has to be easy to install, offer great UX and also be Fedramp and super secure, which most of the time means "either ... or".
Some industries are brutal.
Sorry I don't believe that a process can solve that problem.
Maybe in some specific industries where you have a lot of time, very well defined and predictable roadmaps it can work too.
> Software estimation is a joke because there's no penalty for underestimating.
Comparing something as cookie cutter as movie production, to "sofware" doesn't seem fair. It's comparing the problems of tool users to tool makers.
Movies are mostly cookie cutter. You can go rent a full crew, with equipment and catering, for a weekend with a 30 minute phone call. Try hiring a crew of software developers, for your specific problem, in 30 minutes.
Software isn't cookie cutter, because software isn't fundamentally "software", it's a means to implement a solution for a specific problem. The difference in problem space a software developer can run into is massive. Unless you've already done it before, or are doing something incredibly cookie cutter, time estimates are based on gut feeling and past, probably incorrect, assumptions.
If you're doing something cookie cutter in software, then you can get great time estimates.
If you're doing something unusual with movies, then you can go WAY over time (see anything James Cameron touches).
And like James Cameron innovating underwater shooting with "The Abyss", software developers may innovate a domain on the way to solving a business problem
The problem with software estimation is that the requirements change constantly because customers often don't know what they want until they see a prototype or a mock of the system. Then, you constantly get unspecified, subtle requirements in endless emails starting with "I would also like to ...", without the customer realizing what they are asking for and how that might effect the deadlines.
This is interesting. I've never heard of this service, but it totally makes sense.
It makes me muse: what would it take to have a completion bond service in the software development market? If you were running one of these insurance companies, what would you require in order to be comfortable to put your own money on the line? Presumably it would be similar to what you listed for the movies:
1. everything that will be delivered down to every single detail. No room to change the scope midstream.
2. some historical table listing how long typical features take to develop.
3. personnel involved and their past performance.
I am guessing the first requirement would be the deal breaker, but probably not in every case.
It's the historical data that makes it work. Collect all that data, compute median and standard deviation, and estimate as median plus one sigma or so as a safety margin.
There's outside monitoring. The completion bonding company will usually have someone on set watching every day, checking expenditures and progress.. "The completion bonder usually has a right of approval of the producer, director, and lead cast. These are not creative approvals. Rather, such approvals relate to reliability, tendency to cause over-budget results, history of substance abuse or other personal habits that have raised other issues as to reliability. ... Completion guarantors keep track of the on-budget track record of producers and directors and, it seems, cross-check with each other as to substance abuse or other extreme conduct relating to actors".
It's not that they need every single detail. Changes can be made during production. It's expensive changes that are a concern. Directors who tend to make expensive changes during production are known to completion bond companies, and that gets priced into the estimate. Not necessarily rejected, but priced in.
An interesting analogy. But note that movie output is a fixed blob with NO degrees of freedom, makers have fewer combinatorial problems, and staffing according to a well-defined set of fixed roles with little or no pressure to change that. In contrast, the output of a software project is an interactive machine with enormous degrees of freedom, makers deal with 3 OOM more combinatorial problems (by dint of logical dependencies changing 1000x faster than physical ones), and staffing roles are ALWAYS subject to change with enormous pressure to "innovate" those roles, particularly by smushing them together (e.g. full-stack, devops roles, the end of sysadmins and db admins). What you suggest may be possible within a large and static consultancy that fixes the roles and dependencies for 5-10 years at a time, but such a firm might have a tough sell. Even success stories like 37signals who've been using the same language consistently over its 25 years probably could not do what you suggest.
Movies have much higher risk than software development.
- Shipping cast, crew, and equipment to Outer Nowhere is common. Lots of things can go wrong. Software developers sit at a desk.
- Weather affects shooting schedules in a big way.
- Film production has a lot of moving parts to be kept in sync. This can be botched.
- It's possible to get close to the end of production and see that the plot doesn't work. Most of Woody Allen's movies had serious problems with the ending, and had to be rescued by the film editor. Or that it's a total bomb with test audiences, as with the 2024 Cinderella.
Not good comparision between artistic products vs hard requirement software product.
- If your movie is running out of time, you slash scenes, production etc.
- If your bussines software dev is running out of time and does not meet the minimun requirement, you can NOT slash it. Specially if it mission critical.
In How Big Things Get Done the author Bent is an advocate of reference class forecasting, which is basically what OP describes.
Bent, who studies megaprojects, who goes on to say that the big mistake everyone makes when coming up with estimates is to think that their project is "special". People come up with a million reasons why their project is unique, and then they produce an incorrect estimate because they try to consider each facet of the project and add or subtract time based on it.
In contrast, reference class forcasting "bakes in" all the various factors and averages them. If you're trying to build a house and want to estimate how long it will take, take the average time it takes to build a house of similar square footage in your state. This will factor in things like risk of a storm causing delays or cost overruns, and you don't have to consider that or other unknown unknowns.
If you think some aspect is important, find a reference class that can index on that aspect. The estimate you produce will be more empirical and data driven that way.
>Software estimation is a joke because there's no penalty for underestimating.
Yeah, the other problem is that the company is constantly pressuring the developer to underestimate, and all the costs are on the developer if they can't deliver on that absurd date.
Compare movie production estimation. There are companies which will sell a film production a completion bond. The bonding company guarantees that a releasable film will come out, or your money back.
Here's an overview of completion bonds.[1] A completion bond costs about 2% of production cost. It's typically used for indie films with budgets between US$3 million and US$70 million. The completion bond company will cover some overrun, but if it's too big, they will fire the director and take over the production. That's what gives this real teeth. Movies where this happens tend not to be great, but tend to break even, so everybody gets paid.
So how do completion bond companies do estimates? History. They have the actual expenses from many completed films. This info is at a highly detailed level, with costs for each shot. So, when they get a script with a car chase scene, they go to their database, look up "car chase", and pull out how much the last hundred movie car chases cost. They also have info about directors, producers, and actors, and how much they tend to cost to shoot a scene relative to their peers.
It's not rocket science. It's insurance underwriting.
[1] https://www.mediaservices.com/blog/how-to-bond-a-film-a-defi...