Mifos Technology Plan

The Mifos software is made by a team lead by Grameen Foundation's Technology for Microfinance (T4MF) division. T4MF's vision is "three billion Maries" where Marie means women who raise themselves and their families out of poverty and become leaders in their community.

This document lays out the vision for the Mifos software over the next 18 months.

We want Mifos to become the primary platform for poverty alleviation.

Three things prevent Mifos from being this right now:

  1. Development of new features is too slow.
  2. Mifos doesn't deliver high quality business intelligence to its users.
  3. Mifos Cloud can't handle millions of clients.

To that end, we have three main goals for the next two years:

  1. Make feature development 10 times faster.
  2. Transform Mifos into a best-of-breed business intelligence system for microfinance.
  3. Make Mifos Cloud scalable to 10 million clients hosted in cloud datacenters.

We need to make progress on all the goals simultaneously, so each release will have some features or stories on each. Different releases may be weighted to one release or another. For more information see the Technology Plan spreadsheet.

1. Make feature development 10 times faster.

The reason for this goal, and the reason that it is first, is that if we can make software development 10x faster than it is now, we can deliver any feature or group of features fast enough so that the business can respond to learning or market changes. It drives the other goals.

By 10x faster, we not only mean in "developer hours", the time that a developer spends working on a feature – but also in clock time, the amount of time between the point that it takes to think up a feature and the time it gets into the hands of customers. The reason that decreasing clock time is important is that it gives us the ability to adapt our products to the needs of customers faster, enabling us to iterate on our search for a profitable, effective social business model.

We are trying to have quarterly releases now – every 90 days – but in reality, we release every 4 to 5 months. Being 10x faster means that we will be able to think up features, design them, test them, build them, and ship them in 9 to 15 days from start to finish. Epic features might take longer.

There are two parts to this problem, one in the team, and one in the software.

Team

Our focus should be to limit work-in-progress to a minimum, and maximizing shipped value to customers. It should be on getting learning back to our team so we can use what we learn to innovate, ship, learn, and do it again and again. This is the essence of Lean.

To get here, we will be implementing Lean Thinking techniques, such as very frequent feature shipment. This means one-month release cycles, or pushing software to our Cloud application servers as soon as the feature is done – for example, every week.

To do this, we will have to experiment with different techniques and tools. These include building a "cluster immune system" – automated tests that can verify our software is working as it passes through hosting stages such as test, staging, and production.

Software

The other part of this problem is that Mifos is hard to understand and change. Our task is to make a significant leap in making Mifos comprehensible and flexible. We need to make Mifos into a modern, best-practice-oriented web application. These applications share several characteristics:

A. They are modular.

Code for each main functional area is broken down into understandable, simple chunks, or modules. You can thoroughly understand a module without understanding the whole application. Most code you develop for a module only has to be tested within that module. Modules can be 10x easier to understand when compared to the entire application.

B. They use plugins.

Plugins allow software to be developed separately from the "core" or central code of the application. This has two advantages. The first is that software in a plugin doesn't affect other plugins or the core code, so different versions and technologies can be used independently. The second is that third-party developers don't have to be part of the core development process to develop add-ons, making development faster.

C. They use modern programming languages and frameworks.

Modern programming languages, like Groovy, Python, or Ruby are more concise than Java, allowing developers to express themselves more clearly. Modern frameworks, like Spring Framework and Grails, provide well-tested standard "plumbing" for the things that all applications need – ways to make plugins, separately tested modules, modular user interfaces, good security, reliable data storage, and more.

Areas of Improvement

Here are some areas of the software that related to increased feature development speed. See Appendix A for more info on the technical details.

We believe that these improvements will result in 10x faster feature development clock time and developer time.

Testing Our Hypotheses About Feature Development Speed

As we make these improvements, we need to make measurements to test our hypotheses, so we know whether we are on track to achieve our goals. To do so, we will compare estimates and actual results for stories programmed at various stages of Mifos' evolution to stories programmed using the current paradigm.  Engineering and Program Management will work together to make that happen.

2. Transform Mifos into a best-of-breed business intelligence system for microfinance.

This is a business goal, and is vitally important for our customers, because the way MFIs succeed is by having timely and accurate social and financial information about their business. This is on several different time scales – daily, weekly, yearly, and multi-year. MFI executives, managers, and investors are all interested in this information.

This information is not limited to just Mifos' current loan portfolio management and savings, but extends to information in accounting systems and other systems.

This means having a good data warehouse is important, with an emphasis on having a data warehouse that can both incorporate data from other systems, evolve to be better over time, without frequently breaking existing users' reports.

The way we will accomplish this is by integrating a best-of-breed Open Source business intelligence suite tightly with Mifos, the Pentaho Business Intelligence Suite. This product is also based on Spring Framework and Java, so is compatible at a basic level with Mifos' technology choices, and brings good solutions for report and dashboard creation, reporting server, ETL system, and OLAP analysis.

For our data warehouse, we will use the data warehouse tables as APIs, creating new tables, reports, and associated ETL with every release, and deprecating the ones that are more than 12 months old. This will enable MFIs with custom reports to have reports that don't break with every Mifos release.

New features and plugins from us or third parties will ship with Business Intelligence functionality built-in: reports, dashboards, ETL, production and data warehouse schema changes and updates.

Having a separate data warehouse means that we are free to change the production database however we want. We will simplify the production schema and optimize it for use with our application, leaving the data warehouse with a simpler schema that is optimized for reporting.

3. Make Mifos scalable to 10 million clients hosted in cloud datacenters.

There are two parts to this goal:

This has several implications.

Performance and reliability are first-class features

Performance and reliability are first-class features of Mifos, not something that is added in later. All work needs to take performance and reliability into account – in the web interface, workflows, business logic, batch jobs, ETL, reports.

We will need to set up systems for continuous performance testing, and also for reliability – no crashing or data corruption during long periods. Software must pass performance tests before being pushed to production.

A team-wise implication of this is that all developers should understand the Mifos architecture and be able to explain it to others. This is so they can take advantage of the architecture's support for performance and reliability, and know when it is not working so that they can change it.

In particular, we need Mifos to scale to be one year ahead of our largest customer. This is currently GK, at 500k clients, and they are doubling every year. This means we need to be capable of scaling to 1M clients now, and 2M clients in one year.

Performance and reliability includes hosted systems

We need to think holistically about the systems we run ourselves, as well as the software we produce. This will mean learning about these systems and putting creativity and energy and time into making them run well.

Hosted systems means we have an operations team, and since we are a small, but global company, we will all need to wear the hat of "cloud systems engineer" some of the time. We are going to build a 24x7x365 virtual network operations center with people in our many different time zones.

We will become experts at scaling software in the cloud, building reliable software in the cloud, and securing our customer data in the cloud.

This does not mean a single instance of Mifos scales to 10M clients; it could mean 10 instances each with 1M, 20 with 500k, 200 with 50k, and so on.

Appendix A: Technical Details for Developers

Here are technical details for developers.

1. Make feature development 10 times faster.

Complete conversion of the core application to Spring Framework:

The completion of this effort will be a huge simplification in the Mifos application, making Mifos into a modern Spring-based application with a modern, Maven-based fully-automated build system. This will increase our approachability to developers, since many things about our project will be "standard" and also make programming easier, since all these are good tools and are well-documented.

All new major features will be developed in their own modules, like the Question Group module. This will enable us to keep the complexity low inside a module, enabling fast development.

We will also start breaking the Application module into smaller modules, further reducing complexity and increasing speed.  

Modules should have their own user interface, business logic, ETL, reports, and database tables and updates.

Application module has been broken into at least three separate modules to reduce size and complexity.

Groovy and Grails offer extremely rapid application development, similar to the successes that Ruby and Rails had, but with all of Java software available to Groovy/Grails code. We expect this will give us another large speed-of-development boost, due to Groovy's conciseness, and Grails' great framework code that minimizes boilerplate. Grails also has tight Spring integration.

OSGi is necessary to allow plugins (modules) to use incompatible library versions. This will allow module development to proceed independently from each other, further increasing our speed. Grails, in particular, needs this feature, since Grails is several versions behind the Spring Framework tip of development, and Mifos uses the most recent Spring version.

Eclipse Virgo is the leading open source application server for OSGi-based applications.  We need this server to act as a scalable container for OSGi plugins.

Together, this system will allow the Mifos team and third party developers to write extensions to Mifos entire in Groovy/Grails, without  needing to check code into our central source code repository and integrate it as part of our release process. All plugin development can proceed independently, uncoupling application developers from our core team, increasing the speed of Mifos development as a whole.

Needed for effective testing and for effective modularization and plugins. No business logic should be embedded in the user interface – all business logic should be accessible via APIs.

This will make it easy to acceptance-test functionality programmatically, improving developer speed and quality.

We will need policies and conventions for API design, maintenance, and test.

Incoming and outgoing web services interfaces allow for easy integration to other Internet-connected systems.

This will be a large accelerator for Mifos developers. We will only do acceptance smoke tests on developer boxes; a full acceptance regression suite will be run on the continuous integration server.

All new modules will emphasize unit tests, instead of integration tests. We will follow the testing pyramid: few manual exploratory testing, some automated acceptance tests at the UI level, more automated acceptance tests at the API level, then more integration tests, then mostly unit tests.

Webapps can by delivered with all plugins, some plugins, or just one plugin while offering all Mifos security and business intelligence features.

This will allow for stripped-down versions of Mifos that only offer PPI, for instance, or access to another plugin, but will include access to Mifos' security and business intelligence systems.

With the data warehouse in place we can radically simplify the production database. This will make it much easier to understand, and hence make software development faster.

2. Transform Mifos into a best-of-breed business intelligence system for microfinance.

3. Make Mifos scalable to 10 million clients hosted in cloud datacenters.

Technology Improvement Release Plan

Current state:

Release E:

Release F (quarterly):

Q1 2011

Release G, H, I (monthly):

Q2 2011

Release J, K, L (monthly):

Q3 2011

Release M, N, O (continuous):

Q4 2011