What you've said is very true, but implementing it depends heavily on the client. The fundamental rule of agile is that business needs to participate in software development on daily basis (and I agree with that 100%). Although, it is very difficult. The first big obstacle is that business doesn't understand the very nature of software development which is: it is easy to change and it is reusable. The former one means that you can gradually improve your solution instead of coming up with exhaustive description up front. Instead they think about it like about processes their are familiar with, e.g. building house, where if you build a wall it will be hard to move. The latter one (that it is reusable) is a bit easier to grasp for non-software folks, but then again: someone needs to introduce them to this idea, as they are not familiar with the concept that there are building blocks ready and the hardest part is to find best fitting building blocks and glue them together. Those I would call knowledge gaps, sometimes there is some resistance due to the fact that those guys are doing totally different things, so they don't see a need for learning about software development (which is quite naive, if you ask me, since software is basically everywhere nowadays).
But there are more obstacles, like cultural ones: I can't count how many time I heard "client pays, client demands" without want to hear any feedback or guidance.
There are also obstacles on work organization level, for example: people who have final say how should system look/work have no time to even grasp general ideas about the system.
And god forbid if your client has experience with software developers who were silently and blindly (which means: inefficiently) developing everything that client demanded, because it creates a wall that will be very hard to breakthrough.
I think you're addressing an issue that is related to what I talked about in my comment. Rent seeking behavior of big software corporations that provide most of the base platforms and tooling drives a cottage industry of specialists that thrive in their shadow. There are consulting firms with revenue in the billions just working with 'cloud' platform. There are paas companies out there building on paas owned by a different company, which is in turn based on paas by another company. It's madness.
You're dead on with the business vs software cultural divide. It is both bred by this situation and a moat around it. I am trying to come up with a way to find profit in going against the grain, because I think this is the only way to drive change (or at least preserve my sanity).
> I am trying to come up with a way to find profit in going against the grain, because I think this is the only way to drive change (or at least preserve my sanity).
I know what you mean, but I think the profit is already there, with some good soft skills you can show your value to the client by doing it the right way. Although it won't be always pleasurable or smooth sail as you need to speak up or tell harsh words to others, but, you know, if there is no tension in bicycle chain it means you are not pedaling. Of course, you need to do it in gentle and respectful manner - that's why soft skills are so important.
But there are more obstacles, like cultural ones: I can't count how many time I heard "client pays, client demands" without want to hear any feedback or guidance.
There are also obstacles on work organization level, for example: people who have final say how should system look/work have no time to even grasp general ideas about the system.
And god forbid if your client has experience with software developers who were silently and blindly (which means: inefficiently) developing everything that client demanded, because it creates a wall that will be very hard to breakthrough.