Improve the Test Suite

Owned by Former user (Deleted)
Last updated: Jan 31, 2011 by keithwoodlock (Unlicensed)Version comment

(TODO: MERGE THIS PAGE WITH Evolving Test Suite

Current Implementation

Currently most tests in the Mifos test suite are integration or functional tests that exercise end to end functionality including the database where an independent unit test could be used. A number of business objects include transaction management code in their methods which makes it difficult or impossible to test them independently of the database. There are also order dependencies between tests such that test A must run before test B or test C must not be run after test D. Each test does not clean up properly in case of a failure so that when one test fails, most or all of the tests which follow it will also fail. Finally, the dependency on Mayfly often blocks development since Mayfly's SQL parser is an incomplete implementation of the SQL standard.

Goals

  • completion of "ant run_test" in under 5 minutes on "average" hardware: 1.8Ghz single-proc x86, 1GB RAM, 7200RPM HD
  • memory consumption less than 512M for "ant run_test"

Proposed Improvements

  • Remove database transaction code from business objects and move it to a service layer
  • Create Repository (DAO) interfaces for accessing objects of a given type and when needed use these interfaces in the business objects. By using these interfaces, the interface could be implemented by an in memory test stub for unit testing rather than hitting the database)
  • Separate out functionality that can be tested by a unit test rather than an integration test and try to rewrite as many test cases as possible as unit tests that operate on POJO business objects and don't touch the database.
  • Eliminate test order dependencies
  • Make it possible to enumerate suites to run in a single place. We shouldn't have to maintain ApplicationTestSuite (or contained suites) while also maintaining ApplicationTestSet1, 2, 3, and 4. Perhaps we can even eliminate the need to maintain any list of tests to run, and instead just search for existing test classes and run them all
  • Make each test self contained, such that a failure in one test will not affect other tests
  • Consider an alternative to Mayfly, or improve Mayfly itself
  • Consider structural "SQL diffs" rather than text-based
  • Consider reducing scope of LatestTest (SQL upgrade tests) Once we have no users on older database revisions, we should no longer need to test them.
  • Standardize unit test class names so all have the word Test as the suffix. For example, FooTest is preferred over TestFoo. This emphasizes the class being tested and requires less typing while searching for unit test classes.
  • Allow for autodiscovery of test cases rather than using hard-coded suites. (previous bullet would make this easier, too)
  • Consider doing away with parallel builds (these are only really useful for very beefy multiproc boxes).
  • Consider using TestNG
    • benefit: test groups can be used for running only slow/fast/work-in-progress tests during ant builds

Anticipated Benefits

  1. Increase developer productivity
    • A fast test suite leaves more time for the developer to write code
  2. Reduce the chance of introducing bugs
    • A fast test suite will be run more often so bugs will be caught sooner and the chance of an untested change being committed and breaking the build will be reduced
  3. Make identifying and fixing problems more efficient
    • Eliminating test dependencies will mean that only those tests actually affected by a problem will fail and can be immediately compared to help identify a problem (rather than only the first failure yielding useful information)
  4. Ease of maintenance
    1. #* Autodiscovery will alleviate problem of forgetting to add a particular test