The one thing agile is aimed in the right direction about is that software is for users. Users use it to create value in a business context. Many developers are too far removed from value creation and making sure their features and design changes will actually benefit users in the long run. Tech debt is a thing, but so is business value driven development that works well for users. All of these refactoring side treks need to really be necessary. I do greatly empathize with grandparent, lack of being seen and acknowledged really sucks. And who knows what their situation really was like.
Let's be clear on who the users are, then. In general, I see several different broad groups of stakeholders:
1. The developers who create the software.
2. The rest of the company developers work for, and possibly other people in other companies - who all make money selling or renting the software.
3. The actual users stuck with the software.
I'm not sure if Agile meant 3 > 1 and 2, or just pretended to, but actually meant 2 > 1, but your comment does read as advocating for 2 > 1 > 3, which is arguably the current industry standard.