Data Migration Strategy

This page highlights the findings of the research conducted to ascertain the impact of having a data migration tool kit that will enable smoother migration to mifos. In addition, this page will allow the reader to understand the reasoning upon which the strategy is formed. It provides some good examples of previous data migration strategies to date. 

Overall Data Migration Process and Approach

The data migration tool itself can be looked at as a 3 stage process

1. Having separate adapters for individual MIS systems that interface with a standard format of a Mifos Data Migration tool. 
2. Having a standard format for data extraction 
3. Scripts/code that will read from this standard format and push into Mifos db.

Stage 2 and 3 will have to be generic and be the same regardless of which MIS data is being migrated.

Stage 1 unfortunately cannot be generalized as each MIS system will operate on its own customized schema. This will get further complicated with customers using excel.

Data Migration Experiences

Conflux Technologies Data Migration Experiences

The following are notes recapping the data migration tool that Conflux Technologies developed to migrate data from the FIMO MIS used in India. 

Having spoken with Conflux, it appears that most of the pain and effort resides in stage 1.
For Conflux, once the standard format (MS Access) and the scripts to push data into Mifos (VB scripts) are available, running the scripts is no more than 5-10% of the total effort.
Also once the scripts and the standard format are ready, it is unlikely to change for different MIS systems (Read: one time effort in getting this in shape, and then no effort in this space)
Conflux took about a week or two to write the adapter to move FIMO data to standard format. Once written, the same adapter can be used for any FIMO data migration into Mifos. (Read: Writing adapters is also a one time effort per MIS system). 
To summarize, the maximum effort in the entire process is consumed in data clean up and moving it into a standard format. From there on, if one has a generalized set of scripts to read from the standard format, it is probably a day or two worth of effort to move the data by running the required scripts, although this is usually an iterative action due to inconsistency in the source data.
Both Binny and I are a bit wary of basing our Mifos tool on microsoft technology (MS Access, VB etc) or for that matter, any proprietary software for obvious reasons.

SunGard Data Migration Tool

At present (13 jan 2011) Sunguard's version of a generic data migration tool is quite some time away from completion. On hearing the conceptual approach of theirs, it does seem right, however what materializes at the end of it is yet to be seen.

Kredits Experiences

Discussion with Vikas Jhunjhunwala, who has a good knowledge of another MIS system, Kredits, led us to the same inference that the pain continues to be in stage 1 of the process which includes data cleansing.

Kredits' team has a dedicated data migration specialist who, over time, has built expertise in migration to Kredits. Re-using of existing scripts to push data from an intermediate standard format has brought efficiency. 

Mifos migration for Associacao Progresso: Natu Lauchande

Their approach with selenium was very simple. They simulated a user introducing clients and loans with a csv file and automated it.

They used a csv file and C# to automate the selenium tests (actually data entry) with firefox.

On error handling, they pre-verified the excel files before turning them to sellenium, and monitored the logs of selenium.

The source format was CSV which was taken from a legacy system.

The script (C#) that worked for selenium automation and data validation was the easiest part and was done in one weeks time by a single individual. 

They needed +/- two weeks to clean and make sure that all data was correct with the excel files.

The validation of all the migration process took another week. In total, about a month for a client base of about 1000.

The most painful part, no surprise here, was the data cleansing. While getting the selenium scripts together was very straightforward.

It took two weeks because the MFI personnel actively participated in the cleansing activity.

They did manual testing of the migrated data with the loan officer.

In case of error, rollback was manual. They had some intermediary databases after migration of each file.

He also mentioned that if a data migration tool existed (what we plan to have), this will enable the 'migrators' to focus all the available resource on cleansing part of the entire process.

Interim Conclusion:

For stage 2 and 3, Mifos should provide a solution that other software vendors/MFI's IT personnel can easily write adapters to migrate data from their source data.

2 options are available at this point in time:

1. Build from scratch scripts that will insert data from a standard intermediate platform into Mifos
2. Complete re-use of the scripts that Sunguard's team is working on (or explore other selenium/vb scripts already present), and if the need be, put in minimal effort towards enhancement.