8.3 Creating a Timed Test

8.3.1 Problem

You need to make sure that code executes within a given amount of time.

8.3.2 Solution

Decorate an existing JUnit Test with a JUnitPerf TimedTest.

8.3.3 Discussion

A TimedTest is a JUnit test decorator that measures the total elapsed time of a JUnit test and fails if the maximum time allowed is exceeded. A timed test tests time-critical code, such as a sort or search.

A TimedTest is constructed with an instance of a JUnit test, along with the maximum allowed execution time in milliseconds. Here is an example of a timed test that fails if the elapsed time of the TestSearchModel.testAsynchronousSearch( ) method exceeds two seconds:

public static Test suite(  ) {
    Test testCase = new TestSearchModel("testAsynchronousSearch");
    Test timedTest = new TimedTest(testCase, 2000);
    TestSuite suite = new TestSuite(  );
    suite.addTest(timedTest);
    return suite;
}

In the example above, the total elapsed time is checked once the method under test completes. If the total time exceeds two seconds, the test fails. Another option is for the test to fail immediately if the maximum allowed execution time is exceeded. Here is an example of a timed test that causes immediate failure:

public static Test suite(  ) {
    Test testCase = new TestSearchModel("testAsynchronousSearch");
    Test timedTest = new TimedTest(testCase, 2000, false);
    TestSuite suite = new TestSuite(  );
    suite.addTest(timedTest);
    return suite;
}

The constructor in the previous example is overloaded to allow for a third parameter. This parameter specifies whether the timed test should wait for the method under test to complete, or fail immediately if the maximum time allowed is exceeded. A "false" value indicates that the test should fail immediately if the maximum allowed time is exceeded.

Here's an example of the output when a timed test fails.

.TimedTest (NON-WAITING): testAsynchronousSearch(com.oreilly.javaxp.junitperf.TestSearchModel): 1001 ms
F.TimedTest (WAITING): testAsynchronousSearch(com.oreilly.javaxp.junitperf.TestSearchModel): 1002 ms
F
Time: 2.023
There were 2 failures:
1) testAsynchronousSearch(com.oreilly.javaxp.junitperf.TestSearchModel)junit.
framework.AssertionFailedError: Maximum elapsed time (1000 ms) exceeded!
        at com.clarkware.junitperf.TimedTest.runUntilTimeExpires(Unknown Source)
        at com.clarkware.junitperf.TimedTest.run(Unknown Source)
        at com.oreilly.javaxp.junitperf.TestPerfSearchModel.main(TestPerfSearchModel.
java:48)
2) testAsynchronousSearch(com.oreilly.javaxp.junitperf.TestSearchModel)junit.
framework.AssertionFailedError: Maximum elapsed time exceeded! Expected 1000ms, but 
was 1002ms.
        at com.clarkware.junitperf.TimedTest.runUntilTestCompletion(Unknown Source)
        at com.clarkware.junitperf.TimedTest.run(Unknown Source)
        at com.oreilly.javaxp.junitperf.TestPerfSearchModel.main(TestPerfSearchModel.
java:48)

FAILURES!!!
Tests run: 2,  Failures: 2,  Errors: 0

The example output shows a timed test that fails immediately and another that waits until the method under test completes. The underlying results are the sameboth tests failbut the printed message is different. A nonwaiting test, or a test that fails immediately, is unable to print the actual time it took to complete the test.

Maximum elapsed time (1000 ms) exceeded!

On the other hand, a test that fails after the method under test completes provides a better message. This message shows the expected time and the actual time.

Maximum elapsed time exceeded! Expected 1000ms, but was 1002ms.

As you can see from the previous output, this test is really close to passing. An important point to make here is that when a test is repeatedly close to passing, you may wish to increase the maximum allowed time by a few milliseconds. Of course, it is important to understand that performance will vary from computer to computer and JVM to JVM. Adjusting the threshold to avoid spurious failure might break the test on another computer.

If you need to view some basic metrics about why a timed test failed, the obvious choice is to construct a timed test that waits for the completion of the method under test. This helps to determine how close or how far away you are from having the test pass. If you are more concerned about the tests executing quickly, construct a timed test that fails immediately.

Example 8-1 shows a complete JUnitPerf timed test. Notice the use of the public static Test suite( ) method. This is a typical idiom used when writing JUnit tests, and proves invaluable when integrating JUnitPerf tests into an Ant buildfile. We delve into Ant integration in Recipe 8.7.

Example 8-1. JUnitPerf TimedTest

package com.oreilly.javaxp.junitperf;

import junit.framework.Test;
import junit.framework.TestSuite;
import com.clarkware.junitperf.TimedTest;

public class TestPerfSearchModel {

    public static Test suite(  ) {
        Test testCase = new TestSearchModel("testAsynchronousSearch");
        TestSuite suite = new TestSuite(  );
        suite.addTest(new TimedTest(testCase, 2000, false));
        return suite;
    }

    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite(  ));
    }
}

JUnit's test decoration design brings about some limitations on the precision of a JUnitPerf timed test. The elapsed time recorded by a timed test that decorates a single test method includes the total time of the setUp( ), testXXX( ), and tearDown( ) methods.

If JUnitPerf decorates a TestSuite then the elapsed time recorded by a timed test includes the setUp( ), testXXX( ), and tearDown( ) methods of all Test instances in the TestSuite.

The solution is to adjust the maximum allowed time to accommodate the time spent setting up and tearing down the tests.