First of all, I also do not condone the idea that developers shouldn't worry about other tests than unit tests. I think the developers should be responsible for producing working code, end to end, and having automation and QA is just a nice bonus, additional verification.
Anyway, regarding unit testing. Lot of the objections against unit testing becomes clearer once we start talking about it in a formal way, best in functional terms.
Let's have a function y=f(x) that we want to test. In unit testing, we generate some examples (x1,y1),.. that we run this function with and compare the output.
If we have two functions, f and g, where z = g(f(x)), even if we unit tested each of them separately, we still can fail because what we didn't test was if the domain of g is indeed a subset of range of f. In fact, that cannot be unit tested, since unit tests only verify logic, not the domain and range of the functions.
That's the first objection to unit tests, there are holes in integration. This is especially insidious because if two different people wrote the two functions, they can each have different assumption on the domain and range, yet they won't detect it by unit testing, because they both wrote the tests that only operate under their respective assumption. (BTW, this also shows that the code coverage metrics are meaningless, unless you can coverage all the code executed down to libraries, because you can always leak the coverage through the datatype and vice versa.)
Second objection to unit tests is how do you generate the test cases, in particular, the output? Well, the unit to be tested has to be reasonably small. This means more work (more mocking) and more holes in the integration assumptions, as above.
Personally, I believe property based testing is superior in all respects and should replace unit testing. Property based testing forces you to write down assumptions that you have written your code with, and it scales better because you decouple generation of test cases from the assumptions themselves.
So formally in property based testing, we would create a generator of input cases for x, and also a property - a function that verifies what the y looks like (possibly even given x). In fact, this approach completely subsumes unit testing, because for each of the test cases (x1,y1),.. as above we can just write a property that checks if the output of f is y1 given the x1, and so on.
However, property based testing is stronger. We could also produce a property that would state that input to g has to be in its domain. Then we could easily detect the above problem with composing f and g, just by verifying the properties for our generated test cases. So it resolves the first objection.
That's the beauty of the properties, they really test the data, not the logic. I believe that what we really need to test as developers is the assumptions that we put on the data structures we work with, rather than logic that the functions do. If you want to verify the logic, read the code you have written again.
The second objection to unit testing is also somewhat resolved, because we don't have to produce complete test output, we only verify some of it's properties. Also, separating generation of inputs from properties let's you naturally reuse outputs from the other parts of the program for testing. You only to get the input generation right, the other things will sort of test themselves. So for instance to test g, we don't need to create an extra input generator, we can just live with the outputs of f. In essence, property based testing just makes tests themselves to compose better.
It always bothers me a lot when writing unit test, how little bang I get for the buck. I need to come up with all these test examples, and they usually only cover a small piece of code. What often happens is that I actually know (in my head) the testing generator and properties, it's just instead of writing them up so that computer would understand them as well, I just write a few examples. It feels totally wrong.
Ideally, I would love to see framework that would let some of the verification properties (from property based testing) to live in the code as additional runtime assertions. I think that would be much more practical approach than having a lot of unit tests, and it would tie nicely with defensive programming.
First of all, I also do not condone the idea that developers shouldn't worry about other tests than unit tests. I think the developers should be responsible for producing working code, end to end, and having automation and QA is just a nice bonus, additional verification.
Anyway, regarding unit testing. Lot of the objections against unit testing becomes clearer once we start talking about it in a formal way, best in functional terms.
Let's have a function y=f(x) that we want to test. In unit testing, we generate some examples (x1,y1),.. that we run this function with and compare the output.
If we have two functions, f and g, where z = g(f(x)), even if we unit tested each of them separately, we still can fail because what we didn't test was if the domain of g is indeed a subset of range of f. In fact, that cannot be unit tested, since unit tests only verify logic, not the domain and range of the functions.
That's the first objection to unit tests, there are holes in integration. This is especially insidious because if two different people wrote the two functions, they can each have different assumption on the domain and range, yet they won't detect it by unit testing, because they both wrote the tests that only operate under their respective assumption. (BTW, this also shows that the code coverage metrics are meaningless, unless you can coverage all the code executed down to libraries, because you can always leak the coverage through the datatype and vice versa.)
Second objection to unit tests is how do you generate the test cases, in particular, the output? Well, the unit to be tested has to be reasonably small. This means more work (more mocking) and more holes in the integration assumptions, as above.
Personally, I believe property based testing is superior in all respects and should replace unit testing. Property based testing forces you to write down assumptions that you have written your code with, and it scales better because you decouple generation of test cases from the assumptions themselves.
So formally in property based testing, we would create a generator of input cases for x, and also a property - a function that verifies what the y looks like (possibly even given x). In fact, this approach completely subsumes unit testing, because for each of the test cases (x1,y1),.. as above we can just write a property that checks if the output of f is y1 given the x1, and so on.
However, property based testing is stronger. We could also produce a property that would state that input to g has to be in its domain. Then we could easily detect the above problem with composing f and g, just by verifying the properties for our generated test cases. So it resolves the first objection.
That's the beauty of the properties, they really test the data, not the logic. I believe that what we really need to test as developers is the assumptions that we put on the data structures we work with, rather than logic that the functions do. If you want to verify the logic, read the code you have written again.
The second objection to unit testing is also somewhat resolved, because we don't have to produce complete test output, we only verify some of it's properties. Also, separating generation of inputs from properties let's you naturally reuse outputs from the other parts of the program for testing. You only to get the input generation right, the other things will sort of test themselves. So for instance to test g, we don't need to create an extra input generator, we can just live with the outputs of f. In essence, property based testing just makes tests themselves to compose better.
It always bothers me a lot when writing unit test, how little bang I get for the buck. I need to come up with all these test examples, and they usually only cover a small piece of code. What often happens is that I actually know (in my head) the testing generator and properties, it's just instead of writing them up so that computer would understand them as well, I just write a few examples. It feels totally wrong.
Ideally, I would love to see framework that would let some of the verification properties (from property based testing) to live in the code as additional runtime assertions. I think that would be much more practical approach than having a lot of unit tests, and it would tie nicely with defensive programming.
Finally, I would like to point out to my earlier, more general comment I made about testing: https://news.ycombinator.com/item?id=23470259