Firsthand The following consists of firsthand tips and experiences from our program manager on the data migration process to process from our program manager to help evaluate if it's worth the effort.
...
This is not meant to scare you off, but rather provide you with a reality check. Take for instance a data migration we are currently working on. We have three full-time people working on it, two developers and an Analyst. In addition to that, there is also the client and I and we're each spending quite a few hours each week attempting to resolve the issues that come up. Development time will be somewhere between two to two and a half months, but beforehand, both the development team and I spend quite a few weeks doing an initial analysis . And and the analysis is still going on, the Analyst . The Analyst has been working full-time since researching every possible issue and figuring out the best solution, trying to stay ahead of the developers. Added all together, there is probably somewhere between three to four months of work, spread across five people.
And the source database was simple, out of some 60+ tables we’re only working with about 25 actual tables and out . Out of those, only six really hold relevant data, most every other table is a reference table or holds only a few pieces of information. What all are we migrating? Just Clients, Groups, Loans, and Transactions. Everything else we setup else is set up manually (Office Hierarchy, all users (400+), config, etc).
Gets It gets worse. We’re also working with a moving target. We have to constantly update the mapping files (between the source and Mifos system) for users and offices everytime we make changes. We are creating a set of queries to run before doing migrations to try and suss out every possible type of dirty data that may have appeared between development and the migration. It can't really helped, the MFI has to make forward progress even as we put these scripts together.
...
- Missing client data in the source database: Things things like spouse information, poverty status, and meeting schedule datedates
- In Mifos, a client is linked directly to a loan officer . Not however, this is not so in the source, the client . In the source the client is linked to a loan and loans are linked to a loan officer, so we had to backtrack to figure this out. Seems simple correct? What But what if the client does not have a loan? Who do they get assigned to? What if the client has loans under different loan officers? How do we determine which loan to use for determining who is the appropriate loan officer?
...
The thing with data migrations is you simply cannot figure everything out before handbeforehand, or rather you’d probably spend twice as long as if you were to just do it. It’s an iterative process, its extremely explorative and half the time you won’t find something until you start working with it directly. That, unfortunately, makes it hard to plan for, but there is not much that can be done. Fortunately, we’re collecting more and more information about moving data into Mifos, mapping what is required for creating clients, groups, loans, etc. That should help reduce some of the time required.
In the end, data migrations are completely do-able doable and there is nothing to be scared of, just make sure you plan very, very conservatively.
...