2.3 Unit Testing

A unit test is a programmer-written test for a single piece of functionality in an application. Unit tests should be fine grained, testing small numbers of closely-related methods and classes. Unit tests should not test high-level application functionality. Testing application functionality is called acceptance testing, and acceptance tests should be designed by people who understand the business problem better than the programmers.

2.3.1 Why Test?

XP cannot be done without unit testing. Unit tests build confidence that the code works correctly. Tests also provide the safety net enabling pairs of programmers to make changes to any piece of code in the system without fear. Making changes to code written by someone else takes courage, because you might not be familiar with all of the ins-and-outs of the original solution.

Imagine a scenario in which you are presented with a legacy payroll application consisting of 50,000 lines of code and zero unit tests. You have been asked to change the way that part-time employee salaries are computed, due to a recent change in the tax laws. After making the change, how can you be confident that you did not introduce a bug somewhere else in the system? In a traditional model, you hand the application to a quality assurance team that manually tests everything they can think of.^[5] Hopefully, everybody gets the correct paycheck next month.

^[5] Most companies would like to have dedicated QA teams, but few of these teams seem to exist. XP requires that programmers take on more responsibility for testing their own code.

Now imagine the XP scenario. If the original development team used XP, each class would have a suite of automated unit tests. Before you make your change, you run all of the unit tests to ensure they pass. You then write a new unit test for your new payroll calculation feature. This new test fails, because you have not written the new feature yet. You then implement the new feature and run all of the tests again.

Once all of the tests pass, you check in your code and feel confident that you did not break something else while making the improvement.^[6] This is called test-driven development, and it is how XP teams operate.

^[6] This confidence is justified because of the extensive test suite.

2.3.2 Who Writes Unit Tests?

Programmers write unit tests. Unit tests are designed to test individual methods and classes, and are too technical for nonprogrammers to write. It is assumed that programmers know their code better than anyone else, and should be able to anticipate more of the problems that might occur.

Not all programmers are good at anticipating problems. This is another example of the benefit of pair programming. While one partner writes code, the other is thinking of devious ways to break the code. These ideas turn into additional unit tests.

2.3.3 What Tests Are Written?

Unit tests should be written for any code that is hard to understand, and for any method that has a nontrivial implementation. You should write tests for anything that might break, which could mean just about everything.

So what don't you test? This comes down to a judgment call. Having pairs of people working together increases the likelihood that tests are actually written, and gives one team member time to think about more tests while the other writes the code. Some would argue that tests do not have to be written for absolutely trivial code, but keep in mind that today's trivial code has a tendency to change over time, and you will be thankful that you have tests in place when those changes occur.

There will always be scenarios where you simply cannot write tests. GUI code is notoriously difficult to test, although Chapter 4 offers recipes for testing Swing GUIs using JUnit. In these cases, your programming partner should push you to think hard and make sure you really cannot think of a way to write a test.

2.3.4 Testing New Features

XP teams write tests before each new feature is added to the system. Here is the test-driven process:

Run the suite of unit tests for the entire project, ensuring that they all pass.
Write a unit test for the new feature.
You probably have to stub out the implementation in order to get your test to compile.
Run the test and observe its failure.
Implement the new feature.
Run the test again and observe its success.

At this point, you have tested one facet of your new feature. You and your programming partner should now think of another test, and follow this process:

Write another test for some aspect of the new function that might break, such as an illegal method argument.
Run all of your tests.
Fix the code if necessary, and repeat until you cannot think of any more tests.

Once your new feature is fully tested, it is time to run the entire suite of unit tests for the entire project. Regression testing ensures that your new code did not inadvertently break someone else's code. If some other test fails, you immediately know that you just broke it. You must fix all of the tests before you can commit your changes to the repository.

2.3.5 Testing Bugs

You also write unit tests when bugs are reported. The process is simple:

Write a test that exposes the bug.
Run the test suite and observe the test failure.
Fix the bug.
Run the test suite again, observing the test succeeding.

This is simple and highly effective. Bugs commonly occur in the most complicated parts of your system, so these tests are often the most valuable tests you come up with. It is very likely that the same bug will occur later, but the next time will be covered because of the test you just wrote.

2.3.6 How Do You Write Tests?

All tests must be pass/fail style tests. This means that you should never rely on a guru to interpret the test results. Consider this test output:

Now Testing Person.java:
First Name: Tanner
Last Name: Burke
Age: 1

Did this test pass or fail? You cannot know unless you are a "guru" who knows the system inside and out, and know what to look for. Or you have to dig through source code to find out what those lines of text are supposed to be. Here is a much-improved form of test output:

Now Testing Person.java:
    Failure: Expected Age 2, but was 1 instead.

Once all of your tests are pass/fail, you can group them together into test suites. Here is some imaginary output from a test suite:

Now Testing Person.java:
    Failure: Expected Age 2, but was 1 instead
Now Testing Account.java:
    Passed!
Now Testing Deposit.java:
    Passed!
Summary: 2 tests passed, 1 failed.

This is a lot better! Now we can set up our Ant buildfile to run the entire test suite as part of our hourly build, so we have immediate feedback if any test fails. We can even instruct Ant to mail the test results to the entire team should any test fail.

Writing effective tests takes practice, just like any other skill. Here are a few tips for writing effective tests:

Test for boundary conditions. For instance, check the minimum and maximum indices for arrays and lists. Also check indices that are just out of range.
Test for illegal inputs to methods.
Test for null strings and empty strings. Also test strings containing unexpected whitespace.

2.3.7 Unit Tests Always Pass

The entire suite of unit tests must always pass at 100% before any code is checked in to the source repository. This ensures that each programming pair can develop new features with confidence. Why? Because when you change some code and a test starts to fail, you know that it was your change that caused the failure. On the other hand, if only 98% of the unit tests passed before you started making changes, how can you be confident that your changes are not causing some of the tests to fail?

2.3.8 Testing Improves Design

Writing good unit tests forces you to think more about your design. For GUIs, you must keep business logic clearly separated from GUI code if you have any hope of testing it. In this respect, the tests force you to write independent, modular code.

Writing tests also leads you to write simpler methods. Methods that perform four calculations are hard to test. But testing four methods, each of which performs a single calculation, is straightforward. Not only is the testing easier when your methods are concisethe methods become easier to read because they are short.

2.3.9 Acceptance Testing

When you need to test high-level application functionality, turn to acceptance testing. This sort of testing is driven by the customer, although they will probably need help from a programmer to implement the tests.

Unit or Acceptance Tests?

If you find that your unit tests require lots of complex initialization logic, or they have numerous dependencies that are making it hard for you to change code without rewriting your tests, you may have actually written acceptance tests, rather than unit tests.

Unit tests should test very fine-grained functionality, such as individual classes and methods. As your unit tests grow more and more complex, they start to take on the flavor of acceptance tests instead of unit tests. While these kinds of tests are valuable, it is hard to ensure that they run at 100% success because they have so many dependencies.

Like unit tests, acceptance tests should be designed to pass or fail, and they should be as automated as possible. Unlike unit tests, however, acceptance tests do not have to pass at 100%. Since programmers do not run the suite of acceptance tests with each and every change, it is likely that acceptance tests will occasionally fail. It is also likely that the acceptance tests will be created before all of the functionality is written.

The customer uses acceptance tests for quality assurance and release planning. When the customer deems that the critical acceptance tests are passing to their satisfaction, which is probably 100%, the application can be considered finished.