Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0

...

  • Tomcat running on its own server (application server)
  • Batch jobs running on (application server)
  • MySQL 5 running on its own server (database server)

How do we proceed?

We need to identify just where the scaling and design problems exist.

Metrics:

Simple metrics need to be put in place on the GK production server(s) that show transaction and reporting response times throughout the day. Batch figures are already available. These figures can be used to confirm the impact of changes.

Indicators:

Although many areas of Mifos would and will benefit from scalability improvements, the performance lab assessment will center on the collection sheet process which is very heavily used at GK.

It would be helpful to get accurate detail on the most used functionality. It is more than likely 'Collection Sheet' but it would be useful to see what other product features are heavily used. We could then inspect and test these areas to check for any possible problems.

What we are currently thinking:

  1. Turn on Access Log on tomcat at GK to help give us an idea of the way the application is used at GK (John also created filter to refine these results but this will have to be deployed on GK's app server)
  2. Based on results gathered from Access Logs and customer feedback, identify areas of application to focus on. Then performance test the version of mifos that is currently in GK: #. Create test enviroment that replicates GK's hardware/software configuration #. Create Dataset (or use existing one) that is similar to GK's in size and data distribution. #. Consider using Glassbox within performance lab #. Create JMeter tests scripts for identified problem areas to load test application.

Use data returned from JMeter, Access Logs and Glassbox to generate metrics and benchmarks and to identify areas of the application to approach.

  • Repeat all of the above steps for the latest version of trunk. This will help us assess changes currently incorporated into 'trunk'. These changes are expected to increase throughput by between 33-66% to bring Mifos capacity somewhere within the 400k-500k client range). see Note below.

NOTE: changes currently incorporated into 'trunk'

These are changes made by John around collection sheet save plus some prefetching into hibernate session cache. There's been a number of performance related changes in the collection sheet save but two are significant.

  1. The use of 'bag' instead of 'set' in hibernate configuration for a few tables that are only added to during the process. The account_payment table is an example of this. Previously, using 'set', when adding an account_payment, hibernate retrieved all account_payments for an account... now it doesn't.
  2. Secondly, the prefetching tries to get as much of the customer, account, schedule, fees and other info upfront in a couple of queries to avoid lots of database requests.]

Quick Wins

Metrics and Indicators

  1. Turn on Access Logs
  2. Use Glassbox

Hardware

Sungard have already run tests against some hardware changes that make sense to apply right away as a 'buffer'.

In load testing collection sheets, results showed a 2-3 fold throughput improvement by using a dual processor database server and doubling memory to 8G. If this type of hardware upgrade was introduced to GK now it should alleviate some of the GK performance problems until the number of clients reaches around 500-700K clients.

Reporting

Putting back on the 'mirror' machine for reporting makes obvious sense (if reporting is using up alot of resources)

Compression

Turn on compression using tomcats proprietary approach or using a Serlvet Filter (more portable but that doesn't really matter for mifos).

This will result in better end user experience especially for mifos users with poor internet connectivity (which is probably most of the user base)

Batch Jobs

  1. ApplyCustomerFeeTask
  2. GenerateMeetingsForCustomerAndSavingsTask

GenerateMeetingsForCustomerAndSavingsTask in praticular is taking a long to finish (sometimes up to 7hours) and that as GK scales it may no longer be able to finish within out of business hours. A volunteer has already offered a quick solution to this batch recognising that it spent a large amount of its time in Holidays retrieval. We propose implementing this as a quick win by refactor code in question that is responsible for calling HolidayUtils. see MeetingBO.getAllDates(int o)

Proposed Work

Phase 1

Time: Feb 1 - April 30Business Goal: Cater for 500K Clients (Is this correct?)* Begin investigating the proposal to RedesignAccountsSchedulesAndMeetings. The investigation is very likely to result in a list of incremental changes (probably looking at customer_schedule removal early on). Before taking on the risk of change, we would simulate and verify the effect of these changes in the performance testing lab (set up by Jeff and Sungard).

  • Begin investigating the proposal to Decouple General Ledger (GL) from Mifos. Again, if there are some incremental changes that would achieve quick wins we would do them first in this phase and we would simulate in the performance testing lab prior to development.

Phase 2

Time: May 1 - Jul 31Business Goal: Cater for 1 to 1.5M Clients

Phase 3

Time: Aug 1 - Oct 31Business Goal: Cater for 1.5M+ Clients