mifos_dw_etl_test_dataset.zip
Table of Contents |
---|
The Requirements
...
/Test Summary
This is (probably) a temporary wiki page showing details of manual testing of Data Warehouse 1.0. Attached is the mifos 1.6.1 database backup (mifos_dw_etl_test_dataset.zip) which has the test data used during ETL testing. This page gives some idea of what's involved in the ETL job that moves data from mifos to the data warehouse. The actual Pentaho Data Integration job (CreateDataWarehouse.kjb) is in the business intelligence git repository (git://mifos.git.sourceforge.net / mifos/bi). Manual testing was by query inspection and running against some standard reports (Loan Officer Detailed Report, MFI Progress Report, Due vs Collect Branch Report and Due vs Collected Loan Officer Report).
Note: The tests on this page have been automated.
The Data Warehouse ETL job should...
- copy over the simple 'dimensions' such as products, currencies, offices, personnels correctly
- handle being run against an initialised mifos database
- handle simple hierarchy (loan officer, center, group, client, loan/savings account) creation.
Involves handling new entries and catering for status changes when making groups, clients and accounts active. - handle when a client without a group membership is created and has its status changed and has a loan and a savings account.
- handle when a group without center hierarchy is created and has its status changed and has a client with a loan and a savings account and has its own savings and loan account as well.
- handle when a center has a savings account.
- handle moving a client from one group to another in the same center.
- handle moving a client from one group to another in the same branch
- handle moving a client from one group to another in a different branch
- handle removing group membership from a client.
- handle moving client with no group membership to another branch.
- handle a client with no group membership becoming a group member.
- handle a Loan Officer change for a client with no group membership.
- handle a Loan Officer change for a group with no hierarchy.
- handle moving group with no hierarchy from one branch to another.
- handle center change for a group to center in same branch
- handle center change for a group to center in different branch
- handle adding a group hierarchy back to a center
- should handle a Loan Officer change for a center
- All savings transactions should link to the latest savings account entry that is 'open' (active or inactive but not closed) for the date matching the transaction action_date.
- All repayment transactions should link to the latest loan account entry that is 'active' (good or bad standing)for the date matching the transaction action_date
- All write off transactions should link to the loan account entry having a loan writeoff status.
Write-off account_action_id is 15 - 'AccountAction-LoanWrittenOff' - All disbursal transactions should link to the latest loan account entry that is pre-active - 'AccountState-ApplicationApproved'standing or 'AccountState-DisbursedToLo
(which is probably not used and deprecated) for the date matching the transaction action_date - All Customer Miscellaneous charges and penalties (that are setup on customer_schedule) should appear on the customer charges and fees table linked to an active or on-hold customer
- All Customer Fees (that are setup on customer_fee_schedule) should appear on the customer charges and fees table linked to an active or on-hold custome
Issues Outstanding
MIFOS-3871 - Data Warehouse doesnt handle removing group membership from a client. (fixed)
MIFOS-3872 - Data Warehouse - mifos doesnt currently audit group transfer (no center hierarchy) to another branch.(fixed in 1.7)
MIFOS-3873 - Data Warehouse - mifos doesnt currently audit status change for group transfer (within a center hierarchy) to another branch (fixed in 1.7)
MIFOS-3479 - account_state_change_history and account tables not correctly updated when loan adjustments are made (history fixed for 1.7 release, closed date fix outstanding)
Basic Copy Data and Status Tests
Requirement | Test | Result |
---|---|---|
The ETL job should copy Copy over the simple 'dimensions' such as products, | Setup up an office hierarchy, some loan and savings products, some system users (loan officer and non loan officers) and run ETL | Pass. |
...
Handle being run against an initialised mifos database |
...
|
...
bi server |
...
(for running reports) | run ETL against an initialised mifos 1.6.1 database |
...
Pass |
...
Handle simple hierarchy (loan officer, center, group, client, loan/savings account) creation |
...
. |
...
Setup a center hierarchy down to clients having loan and savings accounts | Pass. All new customers and accounts are added to the datawarehouse as expected |
...
Handle when a client without a group membership is created and has its status changed and has a loan and a savings account. |
...
Client: BO 1 / client w/o group mem xxx 0003-000000028 |
...
000100000000052 |
...
000100000000053 |
...
Pass. |
...
New client and accounts are added to the datawarehouse as expected |
...
Handle when a group without center hierarchy is created and has its status changed and has a client with a loan and a savings account and has its own savings and loan account as well. |
...
Group: group w/o center hierarchy 0004-000000029 |
...
000100000000058 | Pass. |
...
New group, client and all accounts are added to the datawarehouse as expected |
...
Handle when a center has a savings account. |
...
Center: center with savings a/c 0004-000000031 |
...
000100000000061 |
...
Pass. Center and savings account added as expected. |
...
Requirement
The ETL job should copy over the simple 'dimensions' such as products, currencies, offices, personnels correctly
Notes
It is possible to create queries to test this more exactly. I didn't this time because I've used this simple area a lot and chose to only do inspection so as to spend more time on the riskier areas (customers, accounts and transactions).
Test
Setup up an office hierarchy, some loan and savings products, some system users (loan officer and non loan officers) and run ETL
Result
Pass. Looked good on simple inspection using queries and also current reports.
Requirement
The ETL job should handle being run against an initialised mifos database
Notes
Just in case this edge case produced unexpected results. When testing ensure the job points to the correct source and destination database names by inspecting and editing the initialise dataware house 'bat' file and the jndi settings for kettle and the jndi setting for the report designer and for the bi server.
Test - run ETL against an initialised mifos 1.6.1 database
Result
Pass.
Requirement
The ETL should handle simple hierarchy (loan officer, center, group, client, loan/savings account) creation which involves new entries and catering for status changes when making groups, clients and accounts active. Also clients without group membership and group without a center hierarchy should be
Test - setup a center hierarchy down to clients having loan and savings accounts
Result
Pass. All new customers and accounts are added to the datawarehouse as expected
Requirement
The ETL should handle when a client without a group membership is created and has its status changed and has a loan and a savings account.
Test
Client: BO 1 / client w/o group mem xxx 0003-000000028
Savings: savings product 1 # 000100000000052
Loan: loan prod 1 # 000100000000053
Result
Pass. new client and accounts are added to the datawarehouse as expected
Requirement
The ETL should handle when a group without center hierarchy is created and has its status changed and has a client with a loan and a savings account and has its own savings and loan account as well.
Test
Group: group w/o center hierarchy 0004-000000029
Group Loan: group load # 000100000000055
...
Customer Hierarchy Change Tests
Note: There are 3 types of change. New, Hierarchy and Status and if they occur on the same day they are processed in the order New, Hierarchy then Status. This means it can be technically wrong. For example, if you create a client, make her active then move the client to another group all on the same day it will show as new client, move to new group, make active. Couldn't distinguish which event was first because auditing in mifos is at a day level.
Moving clients/groups can only be done if no active accounts exists for that client (all accounts must be closed). Except for clients with no group membership... moving to another branch puts them on-hold and the loan officer is removed and has to be reassigned.
Requirement | Test | Result |
---|---|---|
Handle moving a client from one group to another in the same center. | Group: group dw1 1 - 0003-000000002 | Pass. Client entry added to reflect new group. and savings entry added to link to new client entry. |
Handle moving a client from one group to another in the same branch | Group: group dw1 1 - 0003-000000002 | Pass. Client entry added to reflect new group. and loan entry added to link to new client entry. |
Handle moving a client from one group to another in a different branch | Group: group dw1 1 - 0003-000000002 | Pass. Client entry added to reflect new group. and savings entry added to link to new client entry. |
Handle removing group membership from a client. | client: client remove group membership xxx 0003-000000050 | Fail. The data is ETL's but the outcome looks as if the client had not group membership from the start. |
Handle moving client with no group membership to another branch. | Client: client w/o gr memb xfer to branch xxx 0003-000000045 | Pass. Client and savings account show application of 'Loan Officer Assigned', 'Branch membership' |
Handle a client with no group membership becoming a group member. | Client: client add back to group hierarchy after xxx 0003-000000051 | Pass. |
Handle a Loan Officer change for a client with no group membership. | Client: client w/o grp memb hier LO change xxx 0003-000000044 | Pass. Client and Savings entries as expected. As LO change was done on same day as client and |
Handle a Loan Officer change for a group with no hierarchy. | Group: group w/o center hier LO change 0003-000000042 | Pass. Group, client and account entries as expected. As LO change was done on same day as |
Handle moving group with no hierarchy from one branch to another. | Group: group w/o center hier xfer branch 0003-000000046 |
...
hier xfer branch xxx 0003-000000047 |
...
000100000000086 | Fail. Mifos audit bug - should audit like moving a client to another branch but doesnt. |
Handle center change for a group to center in same branch | Group: group for change to center in same branch 0003-000000056 |
...
Result
Pass.new group, client and all accounts are added to the datawarehouse as expected
Requirement
The ETL should handle when a center has a savings account.
...
# 000100000000100 | Pass. | |
Handle center change for a group to center in different branch | Group: group for change to center in diff branch 0003-000000057 | Pass (enough). However, there is a minor mifos audit BUG where the changing of status to on-hold is not audited. |
Handle adding a group hierarchy back to a center | Group: group going back to a center hier 0003-000000052 | Fail. Connected to MIFOS-3871 |
Handle a Loan Officer change for a center | Center: center with |
...
a |
...
savings: center savings # 000100000000061
Result
...
LO change 0003-000000032 | Pass. center, group & client customer entries good and associated accounts good. |
Transaction Tests
Notes:
Transactions are linked into account entries that are relevant to them at the action (or disbursal) date of the transaction - 'open' (savings), active (repayments) or pre-active (disbursals).
Requirement | Test | Result |
---|---|---|
All savings transactions should link to the latest savings account entry that is 'open' (active or inactive but not closed) for the date matching the transaction action_date. | 1. Deposit 100 into savings account. | Pass. |
| 2. On one day - deposit 100 into a savings account, make inactive, deposit another 100 (which makes it active) | Pass. All deposits and withdraws link to the latest open (inactive in this case) entry for the savings account. |
All repayment transactions should link to the latest loan account entry that is 'active' (good or bad standing)for the date matching the transaction action_date | 1. Disburse a loan and make a repayment (or two) | Pass. |
| 2. Disburse a loan and make a repayment (or two) and then repay early (which closes the account). All on same day. | Pass. |
| 3. Disburse a loan and make a repayment (or two) and then repay early (which closes the account). Then adjust the account which reopens the loan. All on same day. | Fail. No account_status_change_history entry was added by mifos so change from closed to active in good standing not identified. |
| 4. Disburse a loan and make a repayment (or two). Adjust the loan to remove the last payment. | Pass. |
| 5. Disburse a loan and make a repayment (or two). Reverse the loan. On the same day. | Pass. |
All write off transactions should link to the loan account entry having a loan writeoff status. | Disburse a loan and make a repayment (or two). Then write-off the loan. On the same day. | Pass. |
All reschedule transactions should link to the loan account entry having a loan rescheduled status. | Disburse a loan and make a repayment (or two). Then reschedule the loan. On the same day. | Pass. |
All disbursal transactions should link to the latest loan account entry that is pre-active - | 1. Disburse a loan and make a repayment (or two) on the same day. | Pass. |
| 2. Disburse a loan and make a repayment (or two). Reverse the loan. On the same day. | Pass. |
All Customer Miscellaneous charges and penalties (that are setup on customer_schedule) should appear on the customer charges and fees table linked to an active or on-hold customer | Group: Group with lots of fees 0006-000000062 | Pass. 2 entries created on table fact_customer_fees_and_charges for the customer. The misc. fee entry had an account_charge_type_id of 2 (misc fee), and the misc. penalty had an account_charge_type_id of 3 (misc penalty). |
All Customer Fees (that are setup on customer_fee_schedule) should appear on the customer charges and fees table linked to an active or on-hold customer | Group: Group with lots of fees 0006-000000062 | Pass. Entries created with an account_charge_type_id of 1 (fees) |
|
|
|