Developers' FAQ on Liquibase in Mifos

Why liquibase in Mifos?

1. Rely on a feature rich db version control tool instead of custom framework
2. Give an option to decouple db upgrade and application server startup in production
3. Bring in db rollback facility after an upgrade attempt, this is a risk mitigation step for upgrades
4. Allow customers to upgrade to latest version of Mifos skipping any in between releases

I am a developer, how do I apply liquibase change sets?
For the convenience of developers and small deployments, db upgrade during application server start up is retained.

How do developers create a new Mifos db?
Run mifos/db/src/main/resources/sql/base-schema.sql, mifos/db/src/main/resources/sql/base-data.sql and mifos/db/src/test/resources/sql/init_mifos_password.sql against your empty schema. Start your tomcat server for the liquibase upgrade to happen.

Should we create DB checkpoints after a release?
Creating checkpoints per release is discouraged as it breaks customers ability to skip in between releases while upgrading to latest version of Mifos. Over a longer period of time, say an year or two, we may have to consider creating a checkpoint depending on the situation at the time.

Where is the seed data for integration and acceptance tests stored?
Integration test data is available in mifos/db/src/test/resources/sql/integration_test_data.sql. Acceptance test data is in projects/mifos/db/src/test/resources/sql/acceptance_test_dump.sql . Use either one of them depending on context. Use integration_test_data.sql if you just want to have some sample data in local application server.

Why do we classify liquibase changes sets as "expansion" and "contraction"?
Expansions represent the minimum db change required to run new version of application. These are essentially new additions to the database like add column/table/index/key/insert etc. Contractions are deletions from existing db like delete column/table/index/key/data etc which were required by older version of application and not mandatory for new version. Once expansion of db is done, it can support both old and new versions of application. After doing contraction the old version of application is not supported by the db. This gives more flexibility for Mifos administrators to plan db expansion, application server upgrade and db contraction staggered across multiple days.

Do all sql operations naturally fit under "expansion" and "contraction"?
Some operations like rename, data update etc needs to be split into expansion and contraction. Developers have to take care of this considering the bigger goal of expansions and contractions.

What happens if a change set fails to apply on db?
If the failed change set is an "expansion", the application upgrade is not possible, so new version of application shows and error screen listing the unapplied expansions.

Do change set failures leave DB in unstable state?
No. Liquibase applies change sets within a transaction and rollback on failure. For the transactional boundaries to work for a change sets, do not use both DDL and DML in one change set.

Where can I get help for liquibase syntax?
See http://www.liquibase.org/manual/home. Configure you favorite IDE for auto completion using http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd

How do rollbacks work, how can I rollback?
Each change set has a rollback section to be filled by developers. Both expansions and contractions can be rolled back individually. See production db upgrade help for details. Unlike expansions, rollbacks are never applied during application server start up. The shell scripts and batch files used in production are used in CI deployments.

How do I test my change set?
Do mvn clean install on mifos/db module. The integration-test phase of the mifos-db module does an upgrade->rollback->upgrade cycle for verifying the change sets. Syntax errors and rollback errors are caught here. For application level testing, you should still write integration tests for your story.

Where do I see the list of applied upgrades?
This information is available in DATABASECHANGELOG table. You can also use, Admin->View System Information->View applied DB upgrades link to view this information. If there are unapplied change-sets, i.e. contractions, they are also listed. The information displayed on the screen shows the state during server startup, do a server restart for display of latest information.

I do not see a Java based db upgrade mechanism, how do I manage logic involved?
SQL and stored procs are proven mechanisms for relational databases. Liquibase supports stored procedures. Also, Java based db upgrades are hard to use when a deployment has to skip releases.

Can I use liquibase for data migration?
Depends on the kind of data migration. Some can be achieved via plain SQL and others using stored procedures from within Liquibase. The complex ones can use application APIs or ETL tools. Liqibase is not a silver bullet for all DB development.

From when do we allow customers to upgrade to latest version of Mifos skipping any in between releases?
Starting with release G

How do we know the contract and expansion scripts are correctly classified?
Developers should classify every change set they introduce into either the expansion or the contraction categories. As described earlier, this encourages a safe & flexible DB migration for the end-users.

If in doubt, how a given change set should be classified, please use our developer mailing lists or IRC forum. You can also reach out to any our developers to review your change sets before including them to the code base.

After moving to newer version of Mifos, can I wait a week or longer to run db contraction?
Yes. DB contraction needs to happen anytime before next Mifos upgrade. It is a mandatory step in production and must be planned for.

How do we upgrade DB for QA?
For most of the testing purposes like story sign offs, we can rely on tomcat start up for the DB upgrade. The shell scripts used in production is used in CI for DB expansion. Rollback scripts and windows batch files are not part of any automated testing. It is recommended to do one round of manual test with these scripts before a release.

I have created a change set, can I see the SQL equivalent generated by Liquibase?
Before applying change set on the db, run "mvn liquibase:updateSQL" from db module. This step generates the SQL for un-applied upgrades and saves it to a file without applying it to the database.