Skip to main content

Running Individual Test Cases from Ant

September 12, 2003

{cs.r.title}






JUnit and Ant are indispensable tools for Java development. This article discusses how to use them together more
effectively, allowing you more control over which test cases get
run. We'll start by showing how to run a specific Test
class from Ant, and then move on to selectively running individual test
case methods inside a Test.

But first, why would you want to do this? Test-first design and
Extreme Programming suggest that we should run all of the tests, all of the
time. However, not everyone has the luxury of using the test-first
approach. The techniques discussed in this article were developed to
deal with writing test cases for an existing, completely untested
system. In such a system, tests are often just used as an easier way to
make sure code doesn't blow up, rather than testing it with the UI.

The main reason I have found for wanting to run only selected test
cases is in the case of bug fixes. I'll write a test method (or three)
to exercise the bug, and then run those test cases repeatedly as I work on
the bug. In an untested system, tests rarely are completely self-verifying, so I'll need to look at the log output. Being able to look
at the output from only one test case is helpful for
debugging. Another reason for wanting to run only a few tests is that
tests often take a long time to run. Running test cases in isolation
from each other is also warranted because, as often as not, the problem
causing a test to fail is in the test itself.

I certainly recommend running all of the tests on a regular basis. But if
you are already modifying your Ant build or editing your source code
in order to selectively run your tests, you may find that the
techniques discussed in this article will help shorten your
code-compile-test cycle, resulting in better productivity for you.

For the purposes of this article, I assume you've got Ant installed and
configured correctly so you can use the optional
<junit> task. To do this, you need to make sure junit.jar and optional.jar are in the ANT_HOME/lib directory
of your Ant install. If you need more help, check out the Ant documentation (in particular, the section on library dependencies).

Running Tests from a Single Class

It's easy to run all tests from Ant (see Cooking with Java XP), but what if you only want to run a specified Test? To do this, create a target called
runtest that uses an Ant property set on the command
line. This target assumes you have a src property for
your source code, a compile target to compile the code
before running the tests, and a classpath reference set
to your classpath.

<target name="ensure-test-name" unless="test">
    <fail message="You must run this target with -Dtest=TestName"/>
</target>

<target name="runtest" description="Runs the test you specify on the command
    line with -Dtest=" depends="compile, ensure-test-name">
    <junit printsummary="withOutAndErr" fork="yes">
        <classpath refid="classpath" />
        <formatter type="plain" usefile="false"/>
        <batchtest>
            <fileset dir="${src}">
                <include name="**/${test}.java"/>
            </fileset>
        </batchtest>
    </junit>
</target>

The ensure-test-name target verifies that the
test property is set. If test is not set,
ensure-test-name fails and warns the user to set the
property.

The runtest target does the work. It depends on
compile to make sure it's run with your latest code. The
<junit> task runs the Test you specify
with the test property. The task is configured to print
all standard out and error output on the console. The
formatter is plain with
usefile="false" to print all messages to the console in
plain text format, and printsummary="withOutAndErr" will
include all messages printed to standard out and error (if you use Log4J, set the appender to ConsoleAppender to see your log messages on standard out). I also fork the JUnit JVM to avoid "Jar Hell;" this is
optional. For more configuration of the <junit>
task, see the Ant documentation for it.

The runtest target uses <batchtest>
instead of <test>, so we don't have to specify the
fully qualified class name of the Test. If no class matches the
<include> criteria, no test cases will be run and
the build will be successful (the summary report will show no test
cases were run). If a class that's not a Test matches,
the build will fail when JUnit tries to run it.

To run only the Test MyTest, you use the command:

ant runtest -Dtest=MyTest

Here, -D is the flag to pass a property to Ant. In the runtest target, you are passing in the name of the test class to run as the value of the test property.

Because runtest uses <batchtest>, you can use wildcard matching with the test property value. For example, to run all the classes ending in "Test", execute this command:

ant runtest -Dtest=*Test

Running a Test from a Single Method

After implementing running a test from a single TestCase class, I found myself wishing I could run specific test case
methods inside of a TestCase class. Unfortunately, there's
no way to do this in JUnit without changing the
TestCase's suite method and recompiling. But
by modifying the runtest target to take another parameter,
and with a little bit of Java cleverness, we can dynamically create a
TestSuite that only contains the test case methods we
want.

First, let's modify the runtest target to take an
optional argument that describes which test cases to run, as a
comma-separated list of test case names. This Ant property will be
used to create a JVM property.

<target name="runtest" description="Runs the test you specify on the command 
    line with -Dtest=" depends="compile, ensure-test-name">
    <junit printsummary="withOutAndErr" fork="yes">
        <sysproperty key="tests" value="${tests}"/>
        <classpath refid="classpath" />
        <formatter type="plain" usefile="false"/>
        <batchtest>
            <fileset dir="${src}">
                <include name="**/${test}.java"/>
            </fileset>
        </batchtest>
    </junit>
</target>

As you can see, the only modification to this version of
runtest is to set an Ant property named
tests as a JVM property using the <sysproperty>
element. The tests property is optional, so there is no
checking for it as in the case of the ensure-test-name target. Since
there is no way to conditionally change a variable in Ant, if
tests is not set on the command line, the JVM property
will have the literal value "${tests}". The Java code that reads the
JVM property will need to be aware of this.

To run all the test cases in a Test, we'll use the old command line:

ant runtest -Dtest=MyTest

To run only a few test cases, we'll use a command like this:

ant runtest -Dtest=MyTest -Dtests=testFoo,testBar

As you can see, the tests property is optional.

Supplementing JUnit

Now we need some code that can be used by Tests to
dynamically create a TestSuite from the
tests JVM property. This can be implemented as a subclass
of TestCase that your tests then extend instead of
TestCase, or it can be implemented with a few static
methods in a utility class. I favor the latter approach because it
makes it easier for developers to integrate the code into existing
Tests.

We need two public methods: one to determine if the tests property is
set and one to construct the TestSuite from the tests property. Both
of these methods need to be static so that they can be called from the
static suite method of a TestCase. The code
uses reflection to dynamically construct instances of the
TestCase subclass with each test case name from the
tests property. Reflection must be used because each
TestCase object needs to be constructed at runtime with
the name of a test case from the tests property.

public final class TestUtils {
    private static final String TEST_CASES = "tests";
    private static final String ANT_PROPERTY = "${tests}";
    private static final String DELIMITER = ",";

    /**
     * Check to see if the test cases property is set. Ignores Ant's
     * default setting for the property (or null to be on the safe side).
     **/
    public static boolean hasTestCases() {

        return
            System.getProperty( TEST_CASES ) == null ||
            System.getProperty( TEST_CASES ).equals( ANT_PROPERTY ) ?
            false :
            true;
    }

    /**
     * Create a TestSuite using the TestCase subclass and the list
     * of test cases to run specified using the TEST_CASES JVM property.
     *
     * @param testClass the TestCase subclass to instantiate as tests in
     * the suite.
     *
     * @return a TestSuite with new instances of testClass for each
     * test case specified in the JVM property.
     *
     * @throws IllegalArgumentException if testClass is not a subclass or
     * implementation of junit.framework.TestCase.
     *
     * @throws RuntimeException if testClass is written incorrectly and does
     * not have the approriate constructor (It must take one String
     * argument).
     **/
    public static TestSuite getSuite( Class testClass ) {

        if ( ! TestCase.class.isAssignableFrom( testClass ) ) {
            throw new IllegalArgumentException
        ( "Must pass in a subclass of TestCase" );
        }

        TestSuite suite = new TestSuite();

        try {

            Constructor constructor =
                testClass.getConstructor( new Class[] { String.class } );

            List testCaseNames = getTestCaseNames();

            for ( Iterator testCases = testCaseNames.iterator();
                  testCases.hasNext(); ) {

                String testCaseName = (String) testCases.next();

                suite.addTest( (TestCase) constructor.
                               newInstance( new Object[] { testCaseName } ) );
            }

        } catch ( Exception e ) {
            throw new RuntimeException
                ( testClass.getName() +
                  " doesn't have the proper constructor" );
        }

        return suite;
    }

    /**
     * Create a List of String names of test cases specified in the
     * JVM property in comma-separated format.
     *
     * @return a List of String test case names
     *
     * @throws NullPointerException if the TEST_CASES property
     * isn't set
     **/
    private static List getTestCaseNames() {

        if ( System.getProperty( TEST_CASES ) == null ) {
            throw new NullPointerException( "Test case property is not set" );
        }

        List testCaseNames = new ArrayList();
        String testCases = System.getProperty( TEST_CASES );

        StringTokenizer tokenizer = new StringTokenizer( testCases, DELIMITER );
        while ( tokenizer.hasMoreTokens() ) {
            testCaseNames.add( tokenizer.nextToken() );
        }

        return testCaseNames;
    }
}

As you can see, the hasTestCases method ignores the tests
property if it is set to the default of "${tests}", and the private
helper method getTestCaseNames uses a
StringTokenizer to parse the comma-separated list of test
case names, adding each one to a List. The
getSuite method uses this List to
reflectively construct new TestCase subclass instances
and add them to the TestSuite that it returns. If a test
case name is bogus or empty, nothing bad will happen. JUnit will not
be able to find the bogus test case and it will be skipped.

Running the Tests You Want

Now, all the pieces are in place for you to choose which test cases to
execute at run time without recompiling.

public static TestSuite suite() {

    if ( TestUtils.hasTestCases() ) {
        return TestUtils.getSuite( MyTest.class );
    }

    TestSuite suite = new TestSuite( MyTest.class );

    return suite;
}

When writing a TestCase, use
TestUtils.hasTestCases to check for the
tests property, and use TestUtils.getSuite
with the Class object for your TestCase
subclass to return the dynamically constructed
TestSuite. Otherwise, construct and return the
TestSuite as usual.

Luke Francl is a Minneapolis-based software engineer and democracy geek.
Related Topics >> Extreme Programming   |   Testing   |   

Comments

Thanks Luke, that's

Thanks Luke, that's cool.

The one thing that I found a bit troublesome with this is that you have to modify every TestCase subclass you want to be able to run this on in two ways:

1) Implement public static TestSuite suite() with the example shown above (MyTest replaced by the actual class name)
2) Implement the constructor that takes a String parameter.

It's a bit unfortunate that Java reflection won't allow you to determine the actual class at runtime from a Static method.

My tests are set up so that all descend from my own child of TestCase, FunctionalTestCase, so I could implement getSuite in FunctionalTestCase and use it from all my subclasses.
I was able to simplify 1) a bit by by putting the logic in the getSuite method instead, so all you're left with in each subclass is
public static TestSuite suite() {
return getSuite(MyTest.class);
}

I was able to get around 2) altogether by changing getSuite.

First look for a no-parameter constructor instead of the String parameter one:
Constructor<? extends FunctionalTest> constructor =
testClass.getConstructor( new Class[] { } );

Then replace
suite.addTest( (TestCase) constructor.
newInstance( new Object[] { testCaseName } ) );
with

TestCase testCase = (TestCase) constructor.newInstance();
testCase.setName(testCaseName);
suite.addTest(testCase );