>There would be a benefit to thinking that way. Instead of doing a ridiculous sprint planning, instead just keep pulling from the backlog.
The purpose of sprints is not to divide up the work---this is a common misunderstanding. The purpose of the sprint is to set incremental goals based on incremental learning. So instead of setting a goal to create the complete software system from the beginning, you set a goal that can be accomplished within a fixed timeframe that people can make reasonable estimations in and that will test some of your key assumptions about the software.
Simply throwing everything into a massive backlog and grinding through it indefinitely is a recipe for failure, since there is no opportunity to reflect on the larger picture: the team is too busy focusing on the assembly line and not paying attention to the product. And of course you can still have "priority inversions" in kanban, but unlike the finite sprint, you may go months before you realize what you've done wrong.
As I said, there are appropriate times to do kanban, but the situation you describe is not one of them.
> So instead of setting a goal to create the complete software system from the beginning, you set a goal that can be accomplished within a fixed timeframe that people can make reasonable estimations in and that will test some of your key assumptions about the software.
In Kanban you can do all of that, except faster. Instead of waiting for the arbitrary sprint boundary, you can immediately evaluate an item that has landed in production and immediately re-prioritize the backlog based on any learnings. That would be compared to: we will deploy this in sprint A, then in B we will see the results, and then in sprint C we will work on our updated items. The latter is a many week delay compared to assigning tasks in strict sequence of "ship/measure/adjust".
In the worst case, you ship at the beginning of a sprint and then instead of working on the next priority thing (that you learn from having shipped), you instead keep working on the other sprint items. At that point, it could be waterfall, working on items that you now know are pointless, but you still have to anyway because it was planned some time prior to do so.
Kanban doesn't forbid regular retrospectives. You can still pause frequently, reflect, and update your process accordingly.
The purpose of sprints is to try to estimate the completion of a release -- and as progress is made the drift from that original prediction. Sprints in theory let a manager estimate a team velocity. Together with estimates for the task in a release, that, in theory, gives one the ability to predict when a set of features (a release) will be complete. I have yet to see this actually work, but it could just be reflective of where I have worked.
>The purpose of sprints is to try to estimate the completion of a release -- and as progress is made the drift from that original prediction
This is not the purpose of sprints. The sprint is a fixed increment of time intended to encourage incremental development of complete software systems. The idea is that while people don't have an intuitive idea about how much work will have to go into developing a complex application, they do have an intuitive feel for how much work they can accomplish in a couple of weeks. So instead of asking, how much time will it take us to implement a fixed set of features, the team will be asking how many features can we implement this sprint?
>Sprints in theory let a manager estimate a team velocity.
A manager is not a consumer or producer of team velocity. The team itself is the only legitimate consumer and producer of team velocity. Agile is about team self-management.
This idea of asking how much can be implemented in a given sprint is an important clarification. Though, this can be impractical and/or sub-optimal for many contexts. The one-size-fits-all approach of scrum, where so often it is a company or a new person joining and just declaring that "we'll do scrum" now is where things can get de-railed because it is not the universal way to do all development.
So, to paraphrase the distinction, instead of death marching for a large set of features without any feedback, the team decides "we won't do more than 2 weeks of work at a time before we ship & deploy some feature."
(I really wish I could find the link) A recent hacker news article looked at this situation and noted how this does not really allow for extensive R&D where there is no deliverable after the end of two weeks. An example was Google BigTable which had no deliverable at all for 8 months. Yet, the investment into BigTable lead to a very huge pay-off, and the point made was that there was no good way to break up the R&D required into two week deliverables, and the ROI of that work was far more than most any 2 week increment.
Overall the issue for scrum is context, it has its places where it works well - but it is not the one-size fit all solution that so many places try to make it out to be. For example, two weeks does not always allow for a lot of rework, experimentation, architecture, and R&D. A team can choose to make a deliverable be "the results of R&D", is this still fungible thinking and is not actually delivering a release.
Personally I would be a lot happier whenever a scrum master and/or product owner/manager that run a scrum team could clarify exactly when scrum is a bad fit (this feels like the expert-beginner problem). If a person knows when scrum is a bad fit, they can at least try to mitigate those problems and/or lean on other development methodologies. It is almost like silo'ing into one programming language and using only that language despite it not being the best tool in some situations (and also of detriment, various alternative techniques are not learned; eg, sometimes a bash script is the right tool to use, and not branching out and trying to solve some problems with a full blown programming language leads to a lot slower of a solution. To illustrate this, Amazon once had a team spend time to find all phone numbers on its website. This took a long time and the program was complex and never worked well, a very competent shell scripter got wind of this and solved it in 15 minutes with a bash one-liner. This became an interview question at Amazon at one point even, in part to check if a candidate will realize that their language of choice might be a bad fit for the problem at hand)
Breaking a story down into smaller stories is not the same as incrementally developing a complete system. This is the difference between reductionism (breaking a large pre-defined project into smaller pieces for execution) and holism (always delivering a complete system, but incrementally extending the functionality as the team learns more). Sprints provide the time increment for the latter. Note that a sprint increment may differ from project to project. But most teams find 2-3 weeks to be a good increment.
Even in a reductionist approach, though, sprints provide a regular opportunity to review the appropriateness of the breakdown or the validity of the parent story.
You make a really good point on the importance of having a complete iteration at the end of the sprint.
In my experience, the fixed time increments have always felt too constraining.
I think a team can accomplish a similar goal of having regular incremental progress in the form of complete systems by relaxing the fixed-length sprints requirement.
The alternative that I favor is Kanban with defined releases. A release, as a collection of features, is a complete system, but it doesn't necessarily have to be finished in the same time that the previous release was. A larger release may take six weeks instead of the smaller previous release that only took one.
You make a lot of good points, thank you for sharing your thoughts.
General reply to those who say, "but you can have planning and retrospective in kanban!":
Sure, you can. But it requires discipline to actually do it. It's very easy to get wrapped up in a task that runs over, and when you multiply that tendency by everyone on the team, the idea that it will actually get done regularly in practice is optimistic, to say the least. And if you are doing it regularly, together as a team, is that really so different from a sprint?
Conversely, the notion that you can't change course mid-sprint is simply wrong---you absolutely can. The concept of sprint commitment is an acknowledgment that there is value in completing something workable, even if it's the wrong thing, as opposed to staying constantly in a state of WIP. But if it's obvious you're doing something no longer needed or technically feasible, nothing in scrum says you have to complete the sprint as planned.
That incremental goal is not mutually exclusive to kanban. The difference is whether you are stating at the beginning what goal you will hit, vs just working towards that goal. It's the same thing, one creates pressure to hit inaccurate estimates, the other just marches towards the goal.
> Simply throwing everything into a massive backlog and grinding through it indefinitely is a recipe for failure
Why is that? Scrum has the same backlog.
> since there is no opportunity to reflect on the larger picture
Why is this the case? Kanban often says that you talk about retrospective items any time they come up. You then even can re-prioritize such items right when needed, meaning you solve todays problem today, not tomorrow. Retrospectives in scrum can often come too late (and personally I've never really seen them work well, too little, too late, and often compete against 'business' priorities that always win out).
> And of course you can still have "priority inversions" in kanban, but unlike the finite sprint, you may go months before you realize what you've done wrong.
Kanban is less susceptible to priority inversion. You keep the highest priority at top, it is evaluated more often, and you pull from the top. Why does this mean that months can go by? Why is that any more the case than say scrum where you are encouraged to fix priority for weeks at a time?
> As I said, there are appropriate times to do kanban, but the situation you describe is not one of them.
I think the burden of proof is on you to state places where scrum uniquely bests kanban. The statement that kanban is best for places where ad-hoc work comes in constantly is a case where scrum spectacularly fails. For regular project work, kanban can be a great fit (I'd say universally better than scrum, since planning is just complete BS. Kanban is often little more than scrum without the planning sessions and with far more frequent prioritization, and you get to keep all of the same retrospective & estimation tools)
I answered your general point in a separate reply, since others brought up the idea of continuously planning and retrospecting in kanban.
But I'm confused by this statement:
>planning is just complete BS
To me, planning is where design meets reality. Planning involves several activities, but probably the most important is prioritization. I'm curious why you think that prioritization happens more frequently in kanban, where the convention is to simply take the next task off the stack. Are you saying that every time someone takes a task off the kanban backlog, they independently re-prioritize the entire stack? Or do you have regular re-prioritization sessions with the team? If it's the former, that seems like a lot of overhead and opportunity for conflict and misunderstanding. If it's the latter, how is that really different from regular sprint planning? Because surely, as part of the prioritization, you would also consider things like estimation of difficulty and risk.
The purpose of sprints is not to divide up the work---this is a common misunderstanding. The purpose of the sprint is to set incremental goals based on incremental learning. So instead of setting a goal to create the complete software system from the beginning, you set a goal that can be accomplished within a fixed timeframe that people can make reasonable estimations in and that will test some of your key assumptions about the software.
Simply throwing everything into a massive backlog and grinding through it indefinitely is a recipe for failure, since there is no opportunity to reflect on the larger picture: the team is too busy focusing on the assembly line and not paying attention to the product. And of course you can still have "priority inversions" in kanban, but unlike the finite sprint, you may go months before you realize what you've done wrong.
As I said, there are appropriate times to do kanban, but the situation you describe is not one of them.