Don't start with unit tests

I recently gave a presentation on #vBrownBag about testing. Preparing this talk pushed me to formalize some of my thoughts around testing.

When people ask how to get started with testing, they’re often told to start by writing unit tests. If you’re just learning about testing, focusing on unit tests is reasonable. Unit tests are small, and they’re relatively easy to write and run. When it comes to testing real-world projects, however, I don’t think unit tests are the right place to start.

If you write unit tests too early in a project, they’re more likely to break from intentional changes in the implementation of your project, rather than unintentional changes in the behavior of your project. In the early stages of a project’s lifetime, I’ve often found integration tests much more useful than unit tests.

The testing pyramid

There are many visualizations of the “testing pyramid”. My go-to version comes from Automation Panda:

We need to be clear about the different kinds of tests before we can consider which ones are most appropriate to start with. For this discussion, the main points are:

• Unit tests cover individual units of a project’s implementation: functions, classes, and other blocks of code within a project.
• Integration tests don’t deal with implementation; they verify the behavior of a project. Integration tests can cover the behavior of several units that need to work together, or they can cover the overall behavior of the project. For example an integration test can run a project, let it generate output, and then make assertions about the overall output.
• End-to-end tests exercise your project exactly as your users would, including interactions with the outside world. If your project creates external resources through third-party providers, e2e tests should do that as well. If there are multiple pathways through a typical user’s experience of working with your project, there should be tests that exercise each of these main pathways.

These represent three different kinds of tests, each of which has different benefits and different costs. Keep in mind though, that there is some overlap between the different layers.1

Why people recommend starting with unit tests

Many resources recommend that if you’re new to testing, you should start by writing unit tests. If you’re just starting to learn about testing code, without any specific project in mind, it’s natural and appropriate to start with a focus on unit tests. You can work with very small examples, such as a file containing a single function that can be tested. The tests themselves are short, require little or no configuration, and quickly generate results that are relatively easy to interpret.2

But why do people recommend unit tests when someone is looking to write their first tests for a real-world project? There are a number of reasons:

• Writing (and running) unit tests is easier than writing other kinds of tests.
• People who are experienced at testing have worked on projects where unit tests are really useful.
• Writing a single unit test feels good, and once one is written it’s even easier to add more. This is satisfying, because we can start to see test coverage increasing.

It’s natural to suggest starting with unit tests, but it’s not always the best place to start.

The problem with starting with unit tests

There’s one main reason it might not be good to start with unit tests:

Unit tests depend on a stable implementation.

If you change the implementation of your project, some of your unit tests will break, without anything being wrong with your project. Nobody experienced any bugs, but a portion of your new test suite is now broken.

What happens next? Exactly the kinds of things that make people dislike testing:

• You have to stop making progress on your project, and fix your broken tests.
• You’ll know you have to replace these tests again every time you change how your project is implemented.

This kind of experience sours people on testing in general. The good feeling of having test coverage is replaced with a feeling that the tests aren’t helping with the development process, or maintaining the integrity of the project. They’re just something that’s now getting in the way, without any clear benefit. This doesn’t happen because unit tests are bad, or testing is hard; it happens because the wrong kinds of tests were written too early in the project’s lifetime.

The right kind of tests to start with

I believe, for many projects, the best way to start testing is with integration tests. When I’m working on a new project and reach the point where it’s doing what I want to, it’s that behavior I care about, not the implementation. Integration tests verify that behavior. At this stage of a project, I know my implementation is likely to change. But the behaviors are absolutely something I want to preserve. Integration tests written at this point are much more likely to have a lasting positive impact on the project than unit tests.

Integration tests will almost certainly fail at times, but these failures indicate a problem with the project, not a problem with the tests. A failing integration test indicates a change in your project’s overall behavior, which is something you want to know about.  These are the kinds of testing experiences that make people start to value and appreciate tests. Integration tests are a little harder to write, but they’ll almost certainly have an ongoing beneficial impact on the project.

This also holds for established projects that don’t currently have any tests. When I come into an untested but heavily-used project, I don’t care too much about the implementation right away. It usually takes some time to even understand the implementation. I care much more about the overall behavior of the project, which again leads to prioritizing integration tests over unit tests.

Building a testable project

Starting with integration tests has another significant benefit that’s worth mentioning: it ensures that your project is testable.

In order to write integration tests, you need to be able to write code that runs the overall project. If the project has a CLI or an API, that’s easy. But for some projects that have only been developed with end users in mind, it’s not at all clear how to write a script that runs the project. For testing purposes, that’s an important issue to address. The earlier you face this issue in a project, the easier it is to deal with. You might choose to implement a small CLI or limited API to assist with testing. Or, you might choose to restructure your code so it’s easier to initiate the main behaviors of the project programmatically.

Addressing testability isn’t always easy, and it’s one of the reasons that writing integration tests and end-to-end tests is more challenging than writing unit tests. But these issues won’t get easier as a project grows, and putting it off only tends to compound the issue.

Here’s one testing trap that’s nice to avoid: you write a significant number of unit tests and then decide to change the implementation of your project to better support integration and e2e tests. Now your unit tests are breaking, and you have to throw them away or make time to fix them! Putting off integration testing can create significant unintentional time sinks, that end up making people want to avoid dealing with testing as much as possible.

When to start with unit tests

There are few meaningful absolutes in programming, so of course it’s appropriate to start with unit tests in some projects. There are a number of situations I can think of where starting with unit tests is justified, and unlikely to lead to frustration:

You want to write a couple quick tests to get the overall testing infrastructure in place. Writing the first tests for a project often means installing a testing framework, setting up a tests/ directory, and maybe adding some configuration files. This is quicker and easier to do in the context of unit tests, and can be a good way to start testing.

You have units that multiple people or teams depend on. If there are functions and classes already in use by multiple people or teams, it can be helpful to start writing tests for those units. This is in line with the overall spirit of this post: Write tests that will catch meaningful changes in your project’s behavior.

Write tests that will catch meaningful changes in your project’s behavior.

You have critical units. Some units play a more critical role than others in the early phases of a project. If a unit is critical to the success of your project, writing a test for that unit is perfectly appropriate.3

You have stable units. If you have some units where you’re confident the implementation won’t change, feel free to write unit tests for those parts of your project.

You’re already getting bug reports. If you get a bug report, and you can trace that bug to a specific unit, it’s perfectly appropriate to write a unit test that will catch that same bug if it reappears.

These kinds of tests will help you continue to develop your project, without making breaking changes. These tests give meaningful information when they fail or break. They’re driven by the need to verify ongoing behavior of important parts of your project, rather than a blind attempt to gain test coverage.

What about end-to-end tests?

End-to-end tests can be useful as well, but there are some issues that make them less beneficial than integration tests early in a project’s lifetime. Like unit tests, end-to-end tests are dependent on implementation. For example if your tests call out to some remote services, they’ll break if you decide to change service providers.

End-to-end tests are also less consistent than integration tests. A failing integration test almost certainly indicates a change in your project’s behavior. A failing end-to-end test may indicate a change in your project’s behavior, or it may indicate an issue with an external resource. You’ll need to deal with this, but sorting that out in the context of building a test suite may not be the best use of your time early in a project.

Conclusions

Testing is an important part of modern software development workflows, and unit tests are an important part of a mature test suite. But just because they’re at the base of the testing pyramid doesn’t mean we should start there when adding tests to an untested project. If your tests break without any benefit, then testing will be a drag, and a sunk cost as well.

Regardless of whether you’re working on a new, unreleased project or a mature, heavily-used project, consider the benefits and drawbacks of each kind of test before deciding where to start. Integration tests offer tremendous benefit for the expense of a little more work and thought in getting started. For many projects, the benefits of starting with integration tests far outweigh the costs.

Note: This post focuses on why you should consider writing integration tests before writing unit tests. A series coming up soon will focus on how to write integration tests.

Also, thank you to the people who read an early version of this post; it is much better for the feedback you all shared.