Mifos Technology Plan

The Mifos software is made by a team lead by Grameen Foundation's Technology for Microfinance (T4MF) division. T4MF's vision is "three billion Maries" where Marie means women who raise themselves and their families out of poverty and become leaders in their community.

This document lays out the vision for the Mifos software over the next 18 months.

We want Mifos to become the primary platform for poverty alleviation.

Three things prevent Mifos from being this right now:

Development of new features is too slow.
Mifos doesn't deliver high quality business intelligence to its users.
Mifos Cloud can't handle millions of clients.

To that end, we have three main goals for the next two years:

Make feature development 10 times faster.
Transform Mifos into a best-of-breed business intelligence system for microfinance.
Make Mifos Cloud scalable to 10 million clients hosted in cloud datacenters.

We need to make progress on all the goals simultaneously, so each release will have some features or stories on each. Different releases may be weighted to one release or another. For more information see the Technology Plan spreadsheet.

1. Make feature development 10 times faster.

The reason for this goal, and the reason that it is first, is that if we can make software development 10x faster than it is now, we can deliver any feature or group of features fast enough so that the business can respond to learning or market changes. It drives the other goals.

By 10x faster, we not only mean in "developer hours", the time that a developer spends working on a feature – but also in clock time, the amount of time between the point that it takes to think up a feature and the time it gets into the hands of customers. The reason that decreasing clock time is important is that it gives us the ability to adapt our products to the needs of customers faster, enabling us to iterate on our search for a profitable, effective social business model.

We are trying to have quarterly releases now – every 90 days – but in reality, we release every 4 to 5 months. Being 10x faster means that we will be able to think up features, design them, test them, build them, and ship them in 9 to 15 days from start to finish. Epic features might take longer.

There are two parts to this problem, one in the team, and one in the software.

Team

Our focus should be to limit work-in-progress to a minimum, and maximizing shipped value to customers. It should be on getting learning back to our team so we can use what we learn to innovate, ship, learn, and do it again and again. This is the essence of Lean.

To get here, we will be implementing Lean Thinking techniques, such as very frequent feature shipment. This means one-month release cycles, or pushing software to our Cloud application servers as soon as the feature is done – for example, every week.

To do this, we will have to experiment with different techniques and tools. These include building a "cluster immune system" – automated tests that can verify our software is working as it passes through hosting stages such as test, staging, and production.

Software

The other part of this problem is that Mifos is hard to understand and change. Our task is to make a significant leap in making Mifos comprehensible and flexible. We need to make Mifos into a modern, best-practice-oriented web application. These applications share several characteristics:

A. They are modular.

Code for each main functional area is broken down into understandable, simple chunks, or modules. You can thoroughly understand a module without understanding the whole application. Most code you develop for a module only has to be tested within that module. Modules can be 10x easier to understand when compared to the entire application.

B. They use plugins.

Plugins allow software to be developed separately from the "core" or central code of the application. This has two advantages. The first is that software in a plugin doesn't affect other plugins or the core code, so different versions and technologies can be used independently. The second is that third-party developers don't have to be part of the core development process to develop add-ons, making development faster.

C. They use modern programming languages and frameworks.

Modern programming languages, like Groovy, Python, or Ruby are more concise than Java, allowing developers to express themselves more clearly. Modern frameworks, like Spring Framework and Grails, provide well-tested standard "plumbing" for the things that all applications need – ways to make plugins, separately tested modules, modular user interfaces, good security, reliable data storage, and more.

Areas of Improvement

Here are some areas of the software that related to increased feature development speed. See Appendix A for more info on the technical details.

New architecture transition complete
Modular development
Groovy/Grails
OSGi plugins
APIs
QA automation complete
Production database simplification
Data warehouse completion

We believe that these improvements will result in 10x faster feature development clock time and developer time.

Testing Our Hypotheses About Feature Development Speed

As we make these improvements, we need to make measurements to test our hypotheses, so we know whether we are on track to achieve our goals. To do so, we will compare estimates and actual results for stories programmed at various stages of Mifos' evolution to stories programmed using the current paradigm. Engineering and Program Management will work together to make that happen.

2. Transform Mifos into a best-of-breed business intelligence system for microfinance.

This is a business goal, and is vitally important for our customers, because the way MFIs succeed is by having timely and accurate social and financial information about their business. This is on several different time scales – daily, weekly, yearly, and multi-year. MFI executives, managers, and investors are all interested in this information.

This information is not limited to just Mifos' current loan portfolio management and savings, but extends to information in accounting systems and other systems.

This means having a good data warehouse is important, with an emphasis on having a data warehouse that can both incorporate data from other systems, evolve to be better over time, without frequently breaking existing users' reports.

The way we will accomplish this is by integrating a best-of-breed Open Source business intelligence suite tightly with Mifos, the Pentaho Business Intelligence Suite. This product is also based on Spring Framework and Java, so is compatible at a basic level with Mifos' technology choices, and brings good solutions for report and dashboard creation, reporting server, ETL system, and OLAP analysis.

For our data warehouse, we will use the data warehouse tables as APIs, creating new tables, reports, and associated ETL with every release, and deprecating the ones that are more than 12 months old. This will enable MFIs with custom reports to have reports that don't break with every Mifos release.

New features and plugins from us or third parties will ship with Business Intelligence functionality built-in: reports, dashboards, ETL, production and data warehouse schema changes and updates.

Having a separate data warehouse means that we are free to change the production database however we want. We will simplify the production schema and optimize it for use with our application, leaving the data warehouse with a simpler schema that is optimized for reporting.

3. Make Mifos scalable to 10 million clients hosted in cloud datacenters.

There are two parts to this goal:

Scalable to 10 million clients
In hosted cloud data centers

This has several implications.

Performance and reliability are first-class features

Performance and reliability are first-class features of Mifos, not something that is added in later. All work needs to take performance and reliability into account – in the web interface, workflows, business logic, batch jobs, ETL, reports.

We will need to set up systems for continuous performance testing, and also for reliability – no crashing or data corruption during long periods. Software must pass performance tests before being pushed to production.

A team-wise implication of this is that all developers should understand the Mifos architecture and be able to explain it to others. This is so they can take advantage of the architecture's support for performance and reliability, and know when it is not working so that they can change it.

In particular, we need Mifos to scale to be one year ahead of our largest customer. This is currently GK, at 500k clients, and they are doubling every year. This means we need to be capable of scaling to 1M clients now, and 2M clients in one year.

Performance and reliability includes hosted systems

We need to think holistically about the systems we run ourselves, as well as the software we produce. This will mean learning about these systems and putting creativity and energy and time into making them run well.

Hosted systems means we have an operations team, and since we are a small, but global company, we will all need to wear the hat of "cloud systems engineer" some of the time. We are going to build a 24x7x365 virtual network operations center with people in our many different time zones.

We will become experts at scaling software in the cloud, building reliable software in the cloud, and securing our customer data in the cloud.

This does not mean a single instance of Mifos scales to 10M clients; it could mean 10 instances each with 1M, 20 with 500k, 200 with 50k, and so on.

Appendix A: Technical Details for Developers

Here are technical details for developers.

1. Make feature development 10 times faster.

New architecture

Complete conversion of the core application to Spring Framework:

Spring Transactions
Spring Security
Freemarker
Spring Batch and Quartz
Audit logging via Hibernate
Removal of current audit logging system
Removal of current batch system
Removal of all Struts code
Removal of all JSPs

The completion of this effort will be a huge simplification in the Mifos application, making Mifos into a modern Spring-based application with a modern, Maven-based fully-automated build system. This will increase our approachability to developers, since many things about our project will be "standard" and also make programming easier, since all these are good tools and are well-documented.

Modular development

All new major features will be developed in their own modules, like the Question Group module. This will enable us to keep the complexity low inside a module, enabling fast development.

We will also start breaking the Application module into smaller modules, further reducing complexity and increasing speed.

Modules should have their own user interface, business logic, ETL, reports, and database tables and updates.

Application module has been broken into at least three separate modules to reduce size and complexity.

OSGi plugins

OSGi plugins can add business logic, UI, ETL, database schema, and reports, as well as changing workflow to existing flows.
OSGi plugins can be written in 100% Groovy/Grails
Automated tests can be written in Groovy
Mifos runs inside of Eclipse Virgo OSGi application server

Groovy and Grails offer extremely rapid application development, similar to the successes that Ruby and Rails had, but with all of Java software available to Groovy/Grails code. We expect this will give us another large speed-of-development boost, due to Groovy's conciseness, and Grails' great framework code that minimizes boilerplate. Grails also has tight Spring integration.

OSGi is necessary to allow plugins (modules) to use incompatible library versions. This will allow module development to proceed independently from each other, further increasing our speed. Grails, in particular, needs this feature, since Grails is several versions behind the Spring Framework tip of development, and Mifos uses the most recent Spring version.

Eclipse Virgo is the leading open source application server for OSGi-based applications. We need this server to act as a scalable container for OSGi plugins.

Together, this system will allow the Mifos team and third party developers to write extensions to Mifos entire in Groovy/Grails, without needing to check code into our central source code repository and integrate it as part of our release process. All plugin development can proceed independently, uncoupling application developers from our core team, increasing the speed of Mifos development as a whole.

APIs

Needed for effective testing and for effective modularization and plugins. No business logic should be embedded in the user interface – all business logic should be accessible via APIs.

This will make it easy to acceptance-test functionality programmatically, improving developer speed and quality.

We will need policies and conventions for API design, maintenance, and test.

Incoming and outgoing web services interfaces allow for easy integration to other Internet-connected systems.

10 minute build

This will be a large accelerator for Mifos developers. We will only do acceptance smoke tests on developer boxes; a full acceptance regression suite will be run on the continuous integration server.

All new modules will emphasize unit tests, instead of integration tests. We will follow the testing pyramid: few manual exploratory testing, some automated acceptance tests at the UI level, more automated acceptance tests at the API level, then more integration tests, then mostly unit tests.

Plugin configuration

Webapps can by delivered with all plugins, some plugins, or just one plugin while offering all Mifos security and business intelligence features.

This will allow for stripped-down versions of Mifos that only offer PPI, for instance, or access to another plugin, but will include access to Mifos' security and business intelligence systems.

Database simplification

With the data warehouse in place we can radically simplify the production database. This will make it much easier to understand, and hence make software development faster.

2. Transform Mifos into a best-of-breed business intelligence system for microfinance.

Integrate Pentaho BI suite deeply
- Modules or plugins deliver ETL, reports, and dashboards.
- Single sign on to Pentaho Report Server with Mifos username and password.
- Skinning Pentaho server so it looks like Mifos.
- Mifos Admin reports delivered by Pentaho.
Full data warehouse for customers
- Reports only access the data warehouse, and do not use production database or schema.
- Data warehouse is updated by the Pentaho ETL system
- Use data warehouse as APIs for reports – incorporate versioning system, policies for deprecating old APIs.
- Automated pushing of information feeds to approved clients like MIX, Synoptics, etc.
- Grameen Foundation Central Data Warehouse
- Consolidated data warehouse that includes all Cloud customer data.
- ETL systems that move data from customer databases to the central database.
- Reporting server, OLAP cubes, reports and dashboards that allow analysis of this data.
- Uses same database-table-as-API scheme for deprecating data warehouse tables.

3. Make Mifos scalable to 10 million clients hosted in cloud datacenters.

24x7x365 virtual Network Operation Center
At least 3 offices, 8 hours (time zones) apart
Each with 3 technical people, to allow for time off and sick days
At first, we cross-train technical people in each time zone. Eventually, we specialize.
Cluster Immune System
- This is key to decreasing clock time to get features to customers.
- Each service stage – testing, staging, production – has automated smoke and regression tests to verify they are working properly. Automatic rollback if the stage or fleet subset fails verification.
- Ability to push software to system at any time – customers do not notice.
- Continuous performance testing – periodically (daily), builds that pass regression tests have performance tests run on them. Hudson spins up Amazon EC2 instances for tests, shuts them down when done.
- Statistical analysis is done on consolidated logs to catch failures in production.
Scaling
- Multi tier app with linear scaling – just add servers. TCP or HTTP load balancers are used. No state on the web server session. No single point of failure.
- Batch job cluster using Spring Batch, with parallelized batch jobs, so that nightly batch jobs run overnight. Batch jobs access the read slaves, and only write to the master. Possible introduction of Hadoop.
- Users can still use Mifos while batch jobs are running.
Multi tenant
- Vision: one piece of software runs all customers.
- We will get there in stages:
  - Can run Mifos in different tomcat servers, different JVMs on single host (fix timezone bug). (ASAP!)
  - Can run many Mifos copies (mifos.war) on one tomcat, one JVM, on one server (get configuration location from server.xml) (6 – 12 months away)
  - Can run one copy of Mifos for all customers, but using different databases. (18 months away)
  - Can run one copy of Mifos for all customers with one production database. (not in next 12 months)
Data Warehouse features
- Consolidated data warehouse – Pentaho ETL cluster ETLs data from individual Mifos Cloud customers' data warehouses and consolidates to a single MySQL Cluster or Oracle Cluster.
- Batch and ETL jobs can run with users logged in and using the system
- Hadoop or custom "big data" cluster batch jobs can do analysis on the entire data set.

Technology Improvement Release Plan

Current state:

Spring starts Mifos
Configuration Locator reads configuration location only from environment variable
Spring MVC and Freemarker working in the app, Admin pages partly converted
Batch system is now Spring Batch and Quartz, configured using Spring Batch configs.
Spring Transactions working in the app, but only Question Groups module fully implements it.
Pentaho Reporting Server, Pentaho Data Integration Server work with Mifos
Data Warehouse initial version
OSGi spike complete
BIRT reporting is deprecated, but not removed
Continuous performance testing deployed

Release E:

Release F (quarterly):

Q1 2011

Entire user interface converted to Spring MVC and Freemarker; all Struts and JSP code removed from the app
Spring Transactions used throughout the app. Legacy transaction management code removed.
Question Group module converted to OSGi
BIRT reporting must be turned on via a configuration option
Configuration Locator reads configuration from environment variable and server.xml
Groovy/Grails/OSGi spike complete
First API acceptance tests
Acceptance test pass 75% automated
All stored procedures removed from the data warehouse
Ping page implemented
Continuous performance testing used extensively during release
Load balancers deployed for high availability (HA).
Eclipse Virgo used as application container; Apache tomcat is removed.

Release G, H, I (monthly):

Q2 2011

Acceptance test pass 100% automated; only exploratory testing is manual now. Acceptance tests 50% via APIs.
First major feature (Epic Story) done with Groovy/Grails/OSGi. 3x developer time speedup achieved.
Smoke and regression tests can be run against production stages.
Batch system broken into separate application; batch cluster deployed.
No state in HTTP session. First load balancers deployed for scaling.
Application module split into 3 separate OSGi modules
Batch jobs can run while users are logged into the system

Release J, K, L (monthly):

Q3 2011

BIRT removed from the app
Acceptance tests now 75% via API
3 major features implemented in Groovy/Grails. 10x developer time speedup achieved.
First spike of continuous pushes to Mifos Cloud
First major features pushed to Mifos Cloud via continuous release system. 3x clock time speedup achieved
Single-sign-on spike completed
User management module spike completed
Pentaho Mifos skin deployed

Release M, N, O (continuous):

Q4 2011

3 major features pushed to Mifos Cloud via continuous release system. 10x clock time speedup achieved.
Single-sign-on for Mifos and Pentaho via LDAP server
User management feature completed – uses LDAP server