Hey, friends!
In his great talk “TDD – Where did it all go wrong“, Ian Cooper points out how our common way of unit testing fails the promise that the tests would enable us to refactor the system, by giving us confidence that we didn’t break anything.
That common way of unit testing I’m talking about involves testing classes in isolation by mocking out the classes they depend on. In a microservice like this (simplified)…
… you would usually see unit tests for each class individually, where the ProductControllerTest passes mock objects for the three dependencies to the class under test.
While we do achieve the isolation the book tells us to go for, we observe exactly the drawback Ian describes:
They break all the time, especially the heavily mocked ones.
Although we all agree that we want to test behavior, not implementation details, the tests are tightly coupled to the implementation. In result, it’s tedious to do small changes to the code, because unit tests will fail and we have to adjust all those mocks.
Last month, Philipp Hauer published an inspiring article on that topic. He demonstrates how to reduce the refactoring pain by testing classes in integration more than in isolation, and I have really nothing to add to his opinion. Although I had something like that already in mind after watching Ian’s talk, it was Philipp’s example that motivated me to apply that idea to my latest project and helped me to convince my team to give it a try.
After having the new approach in place for some weeks I can just agree that the cost-benefit-ratio of this kind of tests feels way better than before. Few isolated unit tests for situations that make sense, and integrated tests through the rest API as a default really is our sweet spot of testing, so I give Philipp tribute by stealing citing that wording as my headline.
Since we are using JPA in our services (and I’m not going to argue that you should use JPA, it’s just the current state of our product), we had to fiddle a while to set up the tests, and I want to share the learning with you.
SpringBootTest vs. standalone JPA
The main question was if we want to boot up the whole Spring container for the tests or plug the classes under test together manually. We expected the first option to be very simple to set up using @SpringBootTest, but also way slower – in the order of seconds – than setting the classes up manually.
Since we are doing test-driven development all the time and we want to run all the tests at least every minutes, we went for the manual approach first.
As expected, the set up is quite annoying: We need a persistence.xml file in the test resources to make JPA work standlone, and then we have to configure Flyway to run the database migrations before the tests. It’s not nice, but you can live with it of you don’t have to touch the set up code after writing it any more.
The more important learning was: Starting JPA standalone takes nearly as much time as booting up Spring! The performance benefit was by far not as good as expected.
If you look at the SpringBootTest, that reads pretty nicely:
It boots up the Spring application just the way it does in production, so by default we test also dependency injection, security, filters and whatever else the application includes. You may or may not want that, so far I haven’t seen drawbacks of that method. By applying a Spring profile we can disable security and filters that we want to exclude from the test; and by providing Mocks for the client classes to external systems we isolate our code from changing dependencies.
The start up time of Spring Boot is currently 5 – 8 seconds, after that all the tests run at the usual speed – the standalone JPA is simliar. Although I would love to run all the tests in under 2 seconds, I’m willing to make that trade-off for the integrated instead of isolated tests. Also, when tests fail in regression, it may be harder to track down the cause than with isolated tests – we haven’t had the issue yet, but I assume it will happen eventually.
On the upside, we are writing fewer tests, gain more confidence than from isolated unit tests and avoid the refactoring hell.
The described approach works great for the current (low) level of complexity of our latest service. When complexity increases and the number of possible cases grows exponentially, I might start to use more unit tests again for covering cases of logic-heavy classes in isolation, while using the API-based tests for the most important paths to gain confidence. Let’s see.
Find the full code on GitHub and feel welcome to send me feedback on Twitter (@ThiemoOnCoding).