DDSteps - Data Driven Sanity

Aaarrrgh, is not only the sound of pirates. It is also the desperate cry from programmers who realize what a mess they've got themselves into. This is especially true in testing. The number of combination that you should run your code through easily outpaces you. But fear no more, here's some medicin - DDSteps.

A Project

Let us tell the tale of a project that came in from the cold. It started out as a prototype web site, quickly hacked together. Some quick fixes later it went live - slow, unstable, and leaking connections.

Refactoring the code was an inevitable next. As new people joined the project, test-driven development was introduced. The unit tests covered more and more of the code. Things were getting better and bugs were fewer. The unit tests helped the team verify that the classes were OK, but testing the application end-to-end was a different matter.

A feeble attempt was made to automate the functional tests, but it was too difficult to manage the growing amount of test data. Changing some data broke several other tests that depended on the changed data. In the end, functional tests were being done manually.

The business side came up with new feature requests and demanded that they appear in production "real soon". Therefore, a shorter release cycle was introduced, releasing every month. No test team existed, so developers had to "pre-test" the release before it went into production. Testing the site manually took two weeks. Doing this every month was not something the developers looked forward to. The site usually went into production with known, serious bugs.

It was at this point that the programmers went: "Aaarrrgh!"

The Remedy

Testing your solution for every type of input and making sure it will pass function tests is difficult at best. Usually, this is done "by hand" and requires real people. And, yes, it is boring and time consuming. So you have to automate it, just like you did with JUnit tests for unit testing.

By Function tests we mean, "Does it work like the use case says it should". We mean full, system, end-to-end tests, using a production-like environment. We mean using a browser, surfing the web application and then checking the real database for results.

Automating function tests is hard, mostly since they suffer from the same problem as the manual testing. Many of the tests are the same; just the data is slightly different in each test.
Data driven testing on the other hand separates the testing code from its input. This and the reusable test steps makes it easy to maintain and evolve the test cases, you get reuse instead of copy-paste. Need another field? Just add new column in your input and handle it in your test case and voila. There will be no explosion of test methods, no massive round of changes!

Enough of bullets, let's go graphical.

Figure: Overview of the DDSteps´ process. 1) Input is retrieved from an Excel file and populates your test cases and test step POJOs. 2) Yellow is your code. TestSomething is your test method that (re)uses different test steps. 3) The fixture F typically sets the database in the correct state for the test. The Navigators, Executors and Validators in this example use JWebUnit to perform their respective step. 4) Output - reports, console etc.

Since most tests are the same, with just different data, first we separate data (input) and the test code. The data, found in an Excel file, is inserted into your test case, each row becoming a test case run.

The second important part is the framework of reusable test steps. Test cases are broken down into steps:

• Fixture - set up the needed data in your database.
• Navigator - Navigate your web site.
• Executor - Input data into the system, like filling out a form on a page and pressing a button.
• Validator - Validate the output on a page or a row in the database etc.
• Cleaner - Clean up any mess you have made.

You implement these types of test steps using for instance JWebUnit and Spring JDBC. You can easily imagine that many test cases will use the same executor, since they pass through the same web page, so reuse is everywhere.

Finally, DDSteps is just standard JUnit, so the test outcome is reported back via Ant, CruiseControl or your preferred JUnit compatible IDE. It integrates perfectly into your existing environment.

Divide and Conquer

Let us dive deeper. Again, how do we separate data from the code? You can do this several ways. We chose to use Excel. It's not only easy to format and to enter data, it's a de facto standard and you can use formulas etc. And if you want you can always use OpenOffice.

Let us take the following use case from PetClinic from Spring - 'Add new pet'.

Figure: Use case Add New Pet. Green is data that is put into an Excel file. See next figure. The letters N, E and V are our test steps for this use case.

We have now divided our use case above into reusable test steps. Next is the Excel file. From our test case we can see that we need: Owner, pet name etc.

Figure: The Excel file that holds our input and fixture data. Note that the input data could also contain the correct result of the test case method.

Each test method is entered on separate worksheets. The fixture data we need for the database is entered as well. DDSteps find your test data in the spreadsheet using the method names in your test code, and uses JavaBeans get/set to inject data into your test case and your test steps. If we want to run another combination of test data, then we just add another row in this file. Anyone with domain knowledge can do this.

Code

Next is the test code. Let us look at just one part of it - navigating. The example covers points 1-6,7 of the use case, i.e. the first and second test step. In order for this part to work our HTML pages need to be written with JWebUnit in mind, i.e. an id is put on elements, so that we can find it and "click" etc. Let us assume this has been done. We also use Spring even though this is optional.

The test case would then look something like this.

PetclinicTestCase.java

// Our test case, see PetFTest.xls for input.
 public class PetFTest extends PetclinicTestCase {

  protected NavigateToAddPet nav;
  protected ValidatePetForm valForm;
  protected ExecuteAddPet exeAddPet;
  protected ValidatePetOnOwnerPage valOwnerPage;
  protected ValidatePetRow valRow;

  // The test method, same as tab name in Excel file.
  public void testAddPet_Ok() throws Exception {
    nav.runStep();
    valForm.runStep();
    exeAddPet.runStep();
    valOwnerPage.runStep();
    valRow.runStep();
  }

  // Used from data file to access navigator properties, e.g. "nav.name".
  public NavigateToAddPet getNav() {
    return nav;
  }
  // ...and so on for the other test steps.

In the code snippet above we declare our PetFTest test case that inherits from PetclinicTestCase. This base class only holds common things like web browser, fixture etc.
The test method testAddPet_Ok is simple and reduced to only use necessary test steps, nav (NavigateToAddPet) and valForm (ValidatePetForm).
Also note that we have getNav() which is used as the first part in the data file "nav.name". The access to properties is JavaBean based, i.e. using get/set methods.
We will only look at the first test step, the navigator NavigateToAddPet, since the other steps are similar in concept.

NavigateToAddPet.java

public class NavigateToAddPet extends JWebUnitTestStep {

  // Reuse another navigation step
  protected NavigateToOwner navigateToOwner;

  public NavigateToAddPet(WebBrowser webBrowser) {
    // The web browser is injected by Spring.
    super(webBrowser);
    navigateToOwner = new NavigateToOwner(webBrowser);
  }


  public void runStep() throws Exception {

    // Delegate
    navigateToOwner.runStep();

    // Click 'Add new pet' button,
    // which is the submit button in the form 'formAddPet'
    setWorkingForm("formAddPet");
    submit();

    // Snapshot of web page.
    writeTrail("Add New Pet Form");
  }

  // Full name is populated from "nav.name" in the data file.
  public void setName(String name) {
    navigateToOwner.setName(name);
  }
}

This test step, NavigateToAddPet, is a composite of another navigator, NavigateToOwner, and going to the "Add new Pet" page. NavigateToOwner is basically the same as the code above, with the addition of "clickLinkWithText("Find owner");". You can examine the file in detail here.

WriteTrail will write specified html page to the hard disk, a visual page debugger if you will.

So far so good, in the example code above we have surfed (navigated) to the correct page - Add new Pet - in our PetClinic web site.
Let's make sure this is the case.But first we'll introduce a superclass to handle the name, birthdate and type attributes which is used throughout the test steps (see the headers in the test data input file above).

PetWebTestStep.java

public abstract class PetWebTestStep extends JWebUnitTestStep {

    protected String name = "";
    protected String birthdate = "";
    protected String type = "";

    public PetWebTestStep(WebBrowser webBrowser) {
        // Web browser is set up in Spring config.
        super(webBrowser);
    }

    public void setName(String name) {
        this.name = name;
    }

    // ...and so on
}

Ok, now we'll assert that we're on "Add new Pet" page, i.e ValidatePetForm.

ValidatePetForm.java

public class ValidatePetForm extends PetWebTestStep {

    public ValidatePetForm(WebBrowser webBrowser) {
        super(webBrowser);
    }

    public void runStep() throws Exception {

        // There should be a form...
        assertFormPresent();

        // Check contents
        assertFormElementEquals("name", name);
        assertFormElementEquals("birthDate", birthdate);

        // There should be a type, but we don't care which is selected
        assertFormElementPresent("typeId");
    }
}

This is perhaps not the best of examples because we didn't get any data from the test data input file, meaning we have no headers such as "valForm.name".
Why? We're basically checking that the page has certain elements and that these are "blank".

Next part is the enter some data and "hit the submit button" on the page.

ExecuteAddPet.java

public class ExecuteAddPet extends PetWebTestStep {

    public ExecuteAddPet(WebBrowser webBrowser) {
        super(webBrowser);
    }

    public void runStep() throws Exception {

        // Text fields
        setFormElement("name", name);
        setFormElement("birthDate", birthdate);

        // Drop box ----
        // This will select using the SHOWN TEXT, not the value attribute of the <option> tag
        selectOption("typeId", type);

        // submit
        submit();

        writeTrail("After Execute Add Pet");
    }

}

The next step is to make sure we have successfully added a new pet and that the web site correctly displays this. This basically means doing the same type of code as in the ValidatePetForm so we'll just skip that.

However, to make the example complete we need check the database and make sure we got single record in the correct table and that the data is correct - ValidatePetRow.

public class ValidatePetRow extends SpringJdbcValidator {

    protected String name;
    protected Date birthdate;
    protected int ownerId;
    protected int typeId;


    // The data source is set up from the Spring config.
    public ValidatePetRow(DataSource ds) {
        // Here we construct the SQL query and its parameters.
        super(ds, "select * from PETS where NAME = ? and OWNER_ID = ?");

        declareParameter(new SqlParameter(Types.VARCHAR));
        declareParameter(new SqlParameter(Types.INTEGER));
        compile();
    }

    public void setName(String name) {
        this.name = name;
    }

    public void setBirthdate(Date birthdate) {
        this.birthdate = birthdate;
    }
    // ...and so on for the accessors

    // The actual validation of the record from the db and the row from the test input data.
    protected void validateRow(ResultSet rs, int rowNum) throws SQLException {

        // Superfluous, yes, but still... :-)
        Assert.assertEquals(name, rs.getString("NAME"));

        // This way we control EXACTLY which kind of date is read from the database
        DateAssert.assertDatesEqual("Wrong birthdate. ", birthdate, rs.getDate("BIRTH_DATE"));

        Assert.assertEquals("Wrong pet type", typeId, rs.getInt("TYPE_ID"));
        Assert.assertEquals(ownerId, rs.getInt("OWNER_ID"));
    }


    // Supply the actual values for this row of test data.
    protected Object[] assembleParameters() {
        return new Object[] { name, new Integer(ownerId) };
    }


    // Check the number of rows we should have.
    protected void validateRowCount(int numRows) {
        Assert.assertEquals("There should be one and only one pet row.", 1, numRows);
    }

}

We now have all the steps of the test case and, as you can see, they make a good case for reuse in future test cases.

Run

Having come this far, we can now run our test code and get the result. To do this we dive into our IDE, in this case Eclipse, and run the PetFTest.java as a standard JUnit test case.

Figure: And finally the output, in this example, in Eclipse.

Oops, better fix our code! Note that in the example above we see the errors per row, not just by test method.

What happened to the Project?

When our team employ DDSteps, they can automate their tests, and cut down the time for testing to minutes, not weeks. This means you can run all function tests every night or every time the source code changes! Suddenly, you can release to production with no bugs - because as soon as you know there is a one, you fix it during ordinary development. Function testing is now a part of development, not an afterthought.

The Project? Today it releases every third week and goes straight into production. The number of defects is zero.

Conclusion

DDSteps both tries to break new ground and to reuse solutions perfected by others. We think it is a good mix to implement function tests using data driven testing. Is it for you? Who knows? Get it from http://www.ddsteps.org to see for yourself.

Björn Granvik, Jayway