Data Migration Tools Project

Owned by Former user (Deleted)
Last updated: Mar 28, 2011 by Ed CableVersion comment

Data Migration Tools Project Summary

The goal of this project is to propose a framework for data migration (importing legacy data into Mifos) that includes a well defined data import format and to implement the import of one business object type (the Mifos Center type) as a proof of concept. This project is a first step towards the Desired Feature item Improved Tools For Data Migration.

Project Artifacts

Final Project Status - 10 April 2007

Work on this project (implementing a proof of concept) has been completed. A project proposal for a full implementation has been submitted.

Work completed

Week ending 13 April 2007

  • prepared project plan for a complete Mifos Data Exchange XML import implementation
  • wrote a sketch of a data migration framework using data import-- the Data Migration Roadmap
  • performed a quick survey of open source ETL tools
  • final checkin of project code

Week ending 6 April 2007

  • completed proof of concept import functionality for Mifos Center objects
  • worked on project planning and estimation for a complete Mifos Data Exchange XML import implementation
  • studied Mifos codebase to understand potential issues with a complete Mifos Data Exchange XML import implementation
  • generalized import code to handle custom field entries
  • revised schemas to handle custom field entries
  • investigated XML comparison libraries to handle validating equality of unordered elements
  • had a discussion with Terry Wong regarding prior work on a Mifos API and its relation to data migration

Week ending 30 Mar 2007

  • initial check-in of schemas and code for reading and validating Mifos Data Exchange XML
  • completed a first pass at persisting imported Mifos Center objects into the database and did a check-in
  • continued refining the XML schema
  • had a discussion with Arun Kanabar from MFTech about their experience as the Mifos specialist doing data migration for Grameen Koota (see [MIFOS:Data Migration MFTech

Discussion])

Week ending 23 Mar 2007

  • preliminary decision taken to use RELAX NG (RNG) as the primary XML schema language for data migration and also support W3C XML Schema (WXS) (see Data Migration Schema And Tool Choices)
  • preliminary decision taken to use JAXB to generate java classes for reading,writing and validating XML
  • defined and iterated on an RNG schema for the Mifos Center business object
  • created an ant build file that includes targets for: _ RNG -> WXS conversion _ RNG -> RNC (RELAX NG compact syntax format) _ WXS -> java class generation using JAXB _ running tests to exercise XML validation, reading & writing
  • successfully converted the RNG schema to a WXS schema
  • successfully generated and compiled java files from the WXS schema using JAXB
  • created a sample Mifos Data Exchange XML file for Center objects
  • wrote and successfully ran initial tests to validate, read and write Mifos Data Exchange XML for Center objects

Week ending 16 Mar 2007