>If the tests are revealing that the system is "badly designed," the solution is not to throw away the tests in my view - it's to work on gradually refactoring the system to address the difficulties with testing it
I'm mystified that people keep suggesting this given the glaringly obvious chicken-egg problem inherent in doing it.
If you think you can't safely refactor without reliable unit tests and you can't get reliable unit tests without refactoring then you are stuck.
The ironic thing is that on the projects Ive dug out of a "not safe to refactor" quagmire (with high level integration tests), I always used to have an end goal in mind of refactoring to make the code more unit testable in the end. It felt like the "right" thing to do. That was just the industry dogma talking though.
In practice by the time I reached the point where we could do that there was usually no point. Refactoring towards Unit testability had a low/negative ROI at that point, with code that was already safe to change.
> If you think you can't safely refactor without reliable unit tests and you can't get reliable unit tests without refactoring then you are stuck.
Books have been written on exactly this problem; Michael Feathers' "Working Effectively with Legacy Code" comes to mind, where he explicitly names this problem as "The Legacy Code Dilemma:"
> When we change code, we should have tests in place. To put tests in place, we often have to change code.
It's not exactly an easy problem (hence the existence of the book), but there do exist techniques for getting around it - speaking very generally, finding "seams" (places where the behaviour of the system can be modified without changing the source code), breaking dependencies, and gradually getting smaller and smaller units of the system into test harnesses.
Sometimes the code does need to change in order to enable testing:
> Is it safe to do these refactorings without tests? It can be. [...] The trick is to do these initial refactorings very conservatively.
Martin Fowler has catalogued [0] some of these "safe moves" or refactorings one can make; Feathers also recommends the use of tooling and automated refactoring support (provided one understands how safe those tools are, and what guarantees they offer) in order to make these initial refactorings to get the code under test.
Whether or not this is actually worth the time invested, is another matter, and probably one far more complex.
>It's not exactly an easy problem (hence the existence of the book), but there do exist techniques for getting around it - speaking very generally, finding "seams" (places where the behaviour of the system can be modified without changing the source code), breaking dependencies, and gradually getting smaller and smaller units of the system into test harnesses.
These techniques are on the right track but they are outdated. They advocate making changes that are as small as possible (hence the seams thing) while covering the app with unit tests. Most of the advice centers around this maxim: change as little as possible.
What's the smallest amount of change you can make? Zero - i.e. running hermetic end to end testa over the app and changing no more than a couple of lines.
They dont advocate zero though. They advocate unit tests.
To be fair to these authors that option was not really viable when they first wrote the book. Instead of 200 15 second reliable playwright tests run across 20 cloud based workers completing in 3 minutes you were faced with the possibility of 24-48 hour flaky test suites running on one Jenkins server.
So, it probably made sense more often to take slightly larger risks to crack open some of those seams in pursuit of a test that was less flaky and ran in under a second rather than 2 minutes.
> To be fair to these authors that option was not really viable when they first wrote the book. Instead of 200 15 second reliable playwright tests run across 20 cloud based workers completing in 3 minutes you were faced with the possibility of 24-48 hour flaky test suites running on one Jenkins server.
The problem with integration and e2e tests is that they do not give you a measure of the "internal quality" of the system in the way unit tests do. Quoting again from "Growing Object Oriented Software, Guided by Tests:"
> Running end-to-end tests tells us about the external quality of our system, and writing them tells us something about how well we (the whole team) understand the domain, but end-to-end tests don’t tell us how well we’ve written the code. Writing unit tests gives us a lot of feedback about the quality of our code [...]
I don't think compute / test duration were the only motivations behind this approach.
If I inherited a crappy code base I want to get to a place where I can safely refactor as quickly as possible.
I dont need indirect commentary on how crap the code base is in the form of tests that are annoying to write because they require 100 mock objects. It's not telling me anything new and it's annoying the hell out of me while it does it.
If the codebase is good, I also dont need that commentary. I can read.
Indeed maybe the real message that unit tests are sending by being intolerant of bad code is that they are also the bad code.
I'm mystified that people keep suggesting this given the glaringly obvious chicken-egg problem inherent in doing it.
If you think you can't safely refactor without reliable unit tests and you can't get reliable unit tests without refactoring then you are stuck.
The ironic thing is that on the projects Ive dug out of a "not safe to refactor" quagmire (with high level integration tests), I always used to have an end goal in mind of refactoring to make the code more unit testable in the end. It felt like the "right" thing to do. That was just the industry dogma talking though.
In practice by the time I reached the point where we could do that there was usually no point. Refactoring towards Unit testability had a low/negative ROI at that point, with code that was already safe to change.