Don't make a mockery of testing

If you've worked on a Java team that valued test automation, you've probably been exposed to EasyMock or JMock.

While well intended, these tools imply that teams should focus their testing efforts on granular unit tests that provide maximum code coverage. Often these teams do this at the expense of integration tests. Since time is always finite, we suggest you spend the majority of your testing time on integration tests. Only after you've exhausted all integration testing possibilities should you be tempted to use mocks.

Why your team should focus on integration testing:

1: The Golden Rule

This is the most important point. Imagine that you have just inherited a legacy web based application with an automated test suite. Imagine two parallel universes:

In universe one the test suite contains a set of Selenium or WebDriver tests along with a set of database assertions that proved that various end-user facing functionality produced the expected results in the browser/database. There are a few traditional "unit" tests for a few utility classes, but many classes have no corresponding unit tests.

In universe two the test suite contains a test class for every class in the system. Each class is tested with mocks. There are no in-container tests. The DAO unit tests talk to an embedded SQL database, but no other tests use a database.

Which universe would you prefer to inherit? Build that test suite.

2: Integration tests prove end user functionality works

End users consider the system a black box -- a von Neumann machine. Given a set of inputs, does the system produce the desired outputs. Our test suite should prove that these expectations are true.

Integration tests exercise the app from the end user perspective, and prove these expectations. Mock tests do not.

3: Integration tests can survive refactoring unchanged

This week Mike and I refactored some code. We had a SOAP service that contained the same business logic as some Spring MVC controllers. The common code was refactored into a transport-independent service class that the SOAP and controllers shared. Pretty standard exercise.

Our test suite contained unit tests that used EasyMock along with integration tests that made HTTP requests to the Spring MVC controllers and SOAP service deployed in process using an embedded Jetty server.

After we completed the refactoring, all the integration tests passed unchanged. We had not changed the external contract of our application. The SOAP service methods still had the same expected post-conditions. The only thing that changed was the internal implementation.

All the EasyMock tests of the refactored classes were broken. We had to rewrite most of these tests. Mock tests are completely dependent on the internal implementation of the class they are testing.

Fixing these tests was time consuming and felt like time poorly spent. If my methods still produce the same post-conditions given a known set of inputs, why should the test change?

4: Integration tests are more readable

The intention of an integration test is generally pretty clear -- even if the test is coarse grained.

1. Setup some seed data in the database / filesystem
2. Call your class under test
3. Run some assertions against the database, and any other external systems (e.g. did email show up in the SMTP server; did a file get written to the filesystem where I expected it to?)

The intention of a mock test is less clear. Much of the setup of a mock test involves recording expected calls on the collaborator's of the class under test. If you originally wrote the class under test, this isn't a big deal. If you aren't familiar with the class, the test looks like nonsense. Here's an example:

   public void testValidGetRequest() throws Exception {
        
        //
        // Scenario: user has been redirected to the confirm controller
        //
        
        int ridInt = 100;
        String ridCrypt = MailingRecipient.encryptRecipientId(ridInt);
        MailingRecipientDenormalized md = getMailingRecipDenormalized();
        
        expect(request.getParameter("rid")).andReturn(ridCrypt);
        expect(mailingService.getMailingRecipientDenormalized(ridInt)).andReturn(md);
        expect(request.getMethod()).andReturn("GET");
        response.setContentType("text/html");
        expect(response.getWriter()).andReturn(printWriter);
        
        replayMocks();
        controller.handleRequest(request, response);
        verifyMocks();
        
        String html = stringWriter.toString();
        assertNotNull(html);
        assertTrue(html.contains("type='hidden' name='rid' value='" + ridCrypt +  "'"));
    }

Contrast that to a WebDriver integration test that tests the same code path. Keep in mind that this test actually verifies that a real web browser can exercise the code in container:

   public void testValidGetRequest() throws Exception {
        // Create fake ExternalApp w/null unsub URL
        app = new ExternalApp("acme", "test", "john doe", "john@acme.com");
        app.setUnsubUrl(null);
        externalAppDAO.insert(app);
        
        // setup test recipients
        senderEmail = createEmailAddress("sender@acme.com");
        recipEmail  = createEmailAddress("recip@acme.com");
        
        // create test mailing
        mailing = this.createMailingAndInsert(senderEmail, MailingStatus.PENDING, app.getAppId());
        recip = this.createRecipientAndInsert(mailing, recipEmail, MailingRecipientStatus.PENDING);
        String ridCrypt = MailingRecipient.encryptRecipientId(recip.getRecipientId());

        // have firefox request page.  should be redirected to default confirm page
        driver.get(EVENT_BASE_URL + "/unsub?rid=" + URLEncoder.encode(ridCrypt, "utf-8"));
        
        // is the hidden field in the resulting page?
        String html = driver.getPageSource();
        assertNotNull(html);
        assertTrue(html.contains("type='hidden' name='rid' value='" + ridCrypt +  "'"));
   }

*5: Modern tools make integration tests easy to write

Teams often favor mock tests because they are easier to start working with. You don't have to figure out how to embed a servlet engine in your project. You don't have to worry about seeding your database. You don't have to worry about setting up all those DAO objects.

These concerns are true. Getting a system running is time consuming. But that's why we get the big bucks.

Here's a couple of tips to make writing your integration tests easier:

• If you use Spring, have your base setup method initialize your spring context. Now all your objects are immediately wired up and available to your tests.
• If you don't use Spring, consider doing so!
• Use TestNG instead of JUnit. TestNG has the notion of "groups" and you can hook behavior before any test belonging to that group runs. For example, create a "spring" group, create a @BeforeGroup(groups="spring") method that inits the spring environment, and mark all tests that need to use spring objects as: @Test(groups="spring"). This ensures that your spring context is only initialized once, and that it's initialized before any of your spring based tests run.
• Use an embedded Jetty server to test any web based services. Jetty can be checked into your project's SCM tree and used to spin up a servlet engine in-process while your tests run. Use TestNG groups to start/stop the Jetty instance.
• Use WebDriver to test your web application. WebDriver has a very simple but powerful API for controlling Firefox/MSIE/Safari/htmlunit. We've successfully used it on Mac/Windows/Linux using Firefox as the test browser.

The use of TestNG with Spring, WebDriver, and Jetty will be explored in future posts. It's a powerful combination that makes it straightforward to test web based applications.

6: IDE testing tools mitigate execution time concerns

There's no doubt that mock tests run faster than integration tests. But how important is that?

Use the TestNG or JUnit plugin in your IDE to execute a single test at a time, and use ant to run the whole test suite. Single test classes that utilize webdriver will still execute in a matter of seconds. As you do development you typically find yourself running one or two test classes repeatedly.

Run the whole test suite before you commit your changes to your SCM.

Conclusion

Your test suite is your safety net. Spend your testing time wisely on tests that prove that the system exhibits the external behavior users expect. Your tests will find more bugs, survive internal refactoring, and be easier to maintain. Those who inherit your code will thank you.