Pentaho Report Testing

Vision:

Test new Mifos Pentaho reports in an automated fashion, using a separate testing framework from Mifos. Create test cases at multiple levels, validating data at each layer of our design. All standard reports are under test so they are tested continually, reducing risk in regressions of ETL jobs and Pentaho reports.

Testing Requirements

1. Reporting system generates correct results. Numbers from reports match queries to database, expected results.
2. The reporting system performs well under load when extracting data.
3. The reporting system performs well under load generating analyzed/computed results.
4. Reporting system is usable for Mifos users. Data warehouse is intuitive and documented.
5. Summary results are accurate
6. Can drill down from summary results to accurate information? (Question: are cubes a requirement?)
7. Test error situations, verify accurate notification happens
8. Audit, error logging occurs
9. Secure access to reports

Testing Stages

BI testing should test data integrity at different stages.

Pretest: run queries against source data to generate expected data for ETL tests. Open Question: Run once and save off expected results or run query every time ETL is tested/ Trade offs with both solutions.

Stage:

1. ETL
2. Report Generation - file level test result
3. Report Generation - UI level

The database test should be a SQL query to confirm the direct table query gives expected results, e.g. selecting loans in arrears amount for a given branch. Steps include determining a finite test case, developing the proper query, then using DBUnit or a similar tool to validate query results against the Mifos database.

The ETL test would execute a test case to extract data that's been generated from the ETL job, e.g. selecting loans in arrears amount for a given branch. In this test, check for accuracy and completeness of data. Compare expected results using simple comparison of output file or DB query results from Pretest step. ETL tests includes actions that are invoked for error conditions, etc.

Report Generation file testing will test a defined set of reports, comparing expected results to generated results.. Test will execute Pentaho report, but validate results below the Report UI, comparing csv, excel, or html file results using TestNG or JUnit.

UI level testing will test the layout and structure of the standard report. Ensuring report spacing, font, etc. is according to report specification. Layout validation by automated test method is tbd. Also need to spike on whether Pentaho report can be driven with Selenium.

Test data considerations:

1. Will sample data suffice? Should include irregular data as part of our test data to test for input errors.
2. Use same test data through each stage of testing to ensure consistent results are being returned at each stage.
3. Test data size. Functional tests can use minimal or typical MFI sized data. Performance test requires large MFI data set to stress the data extraction

Types of data to validate

1. loan amounts and totals
2. client/group/center info
3. transaction dates
4. Accounting details
5. transaction dates
6. calculated values - e.g. PAR, # of active clients, total savings, etc.

Tasks:

1. Spike on writing a ETL level automated test - capture result of ETL job and compare with expected results. Result is a single working, simple example (POC)
2. Spike on writing a Report Generation test executes report, and compares report result via file comparison (e.g. csv files) Investigate how to run Pentaho, preferably as a java application that runs the java engine. Result is a single working, simple example (POC)
3. Write ETL acceptance/integration test for ETL job used in new Pentaho standard report.
4. Write acceptance/integration test to validate results returned in the standard report.
5. Spike on validating Pentaho report for proper UI format. Can Selenium work with Pentaho.(P2)
6. Acceptance test to run Pentaho report and UI level and validate UI results.(P2)