Mifos Business Intelligence Deployment

As a Mifos systems administrator I want an easy way to deploy Mifos Business Intelligence code so I can effectively manage my Mifos reporting solution.

As a Mifos BI developer I want a fast and reliable way to develop Mifos Business Intelligence code so I can quickly deliver new BI features and bugfixes.

The Mifos BI software doesn't currently have a "build" per-se (no code needs to be compiled), the reports and ETL jobs simply need to be installed on a pentaho server. But we need some amount of automation (scripts, installer, whatever) to make sure that the Mifos BI software can be reliably deployed and updated.

Terminology

• BI = Business Intelligence
• "Install" may be considered a synonym of "deploy" in this spec.

Mifos BI code

includes:

• Reports
  • Tally Integration (reports that output XML for tally) - .jar file, xaction, etc.
  • MixXRBL (outputs XML)
  • Mifos
    • PPI, loan officers, etc.
• Database schema, data
• Kettle Jobs

Goals

• allows automated, simple installation of Mifos BI code
• initializes the data warehouse tables (if they don't already exist)
  • updates data warehouse tables, if necessary
• loads initial data necessary for Mifos BI code to function
• uses only FLOSS tools
• provides Mifos BI code "releases" with a version number
• installs reports into a pentaho server
  • updates existing reports, if necessary
• installs Kettle jobs into a pentaho server
  • updates existing Kettle jobs, if necessary
• user may specify pentaho installation location (default is /opt/pentaho)
• works within a continuous integration system
• runs on Windows, Mac OS X and GNU/Linux (at least Ubuntu)
  • Ubuntu is the primary target
• assumes java, MySQL, pentaho, etc. are already installed
  • pentaho shall be installed separately by sysadmin
    • but hopefully to a standard location, like /opt/pentaho
• useful in Mifos hosted/cloud deployments as well as community/external deployments

Non-Goals

• i18n/L10n. The deploy procedure will be English only.
• install prerequisites
  • install java
  • install MySQL
  • install pentaho
• provide a GUI (although this may be useful)

Development

Install/Deploy steps for a development machine

• How to set up a Pentaho for Mifos

Upgrade steps for a development machine

• ...
• ...

Production

Install/Deploy steps for a production server

• install prereqs
  • machine instance, OS, java, mysql, pentaho
• deploy Mifos BI software
  • copy ETL jobs, reports, into appropriate places on server
  • create data warehouse database
  • schedule ETL jobs to run at regular intervals
• configure
  • set up JNDI db connections
  • startup/shutdown scripts

Upgrade steps for a production server

• update configuration, if necessary
• copy in new versions of BI software

Validate Environment

• for dev: run automated test suite
• Can connect to Pentaho console
• Can execute report
  • report is pulling the right MFI data
  • we can view the right versions (q: how to we verify report version?)

Questions

Testing

How shall we test the installation? Would an idea be to install a vanilla pentaho environment, run the installation/deployment and then confirm everything runs?
It seems obvious but I think there are already a few 'add-ons' to be taken into account... e.g. (1) Tally export... Udai did a java class to use freemarker and (2) Graymatter did some config for report parameters. Would these add-ons be part of installation or follow on from a customised Pentaho installation.

Recent discussion on testing

ETL tests on our ci server

Priority

We need to prioritize having this procedure for our hosted customers (cloud) and then the procedure for those in our community who wish to deploy these reports themselves.  Is this possible?  Is the cloud process basically a subset of larger deployer that we can knock out and break down stories/cards for first?

Configuration

In Mifos, we have a custom properties file where you say how many decimals you want to be displayed for example.

How will this be handled in Pentaho when we deploy? Does each customer choose certain configurations, or do we have a standard file we apply to everyone?

Or will there be some things we set as default, while others such as decimals can be configured by the MFI.

Eventually, we want to allow consistency across Pentaho and Mifos.

Ideas

Designed for the Mifos cloud reporting environment

• Ubuntu GNU/Linux 10.04

When porting the installer to other platforms, the install code would (ideally) be mostly reusable.

Potential execution targets

• Java 6 version 1.6.x (Java or Groovy source only, please)
• Bash shell
• Python 2.6.x

May make use of environment or configuration variables

• JAVA_HOME - path to root of Java runtime installation
• BI_HOME - path to root of clone of Mifos "bi" git repository
• path to pentaho installation
• mysql connection information

Ideas from Graymatter

• create a Pentaho plugin to help with installing Mifos BI software
  • update installed Mifos Intelligence Suite by asking a central server if updates are available
  • assumes build script, source control for plugin
  • what about kettle/PDI? (ie: ETL jobs)
• script that copies stuff into the right folders, for example, reports, ETL jobs
• buildscript (aka, install/deploy script, like an Ant build.xml file)
  • see pentaho-sso - biserver-manual
  • http://wiki.pentaho.com/display/ServerDoc2x/Running+the+Single+Sign-On+Enable+Script
• has anything special been done in graymatter reports which requires doing anything special during the install/deploy of BI software?

Issues

• initial spike: MIFOSBI-56