...
- 24x7x365 virtual Network Operation Center
- At least 3 offices, 8 hours (time zones) apart
- Each with 3 technical people, to allow for time off and sick days
- At first, we cross-train technical people in each time zone. Eventually, we specialize.
- Cluster Immune System
- This is key to decreasing clock time to get features to customers.
- Each service stage – testing, staging, production – has automated smoke and regression tests to verify they are working properly. Automatic rollback if the stage or fleet subset fails verification.
- Ability to push software to system at any time – customers do not notice.
- Continuous performance testing – periodically (daily), builds that pass regression tests have performance tests run on them. Hudson spins up Amazon EC2 instances for tests, shuts them down when done.
- Statistical analysis is done on consolidated logs to catch failures in production.
- Scaling
- Multi tier app with linear scaling – just add servers. TCP or HTTP load balancers are used. No state on the web server session. No single point of failure.
- Batch job cluster using Spring Batch, with parallelized batch jobs, so that nightly batch jobs run overnight. Batch jobs access the read slaves, and only write to the master. Possible introduction of Hadoop.
- Users can still use Mifos while batch jobs are running.
- Multi tenant
- Vision: one piece of software runs all customers.
- We will get there in stages:
- Can run Mifos in different tomcat servers, different JVMs on single host (fix timezone bug). (ASAP!)
- Can run many Mifos copies (mifos.war) on one tomcat, one JVM, on one server (get configuration location from server.xml) (6 – 12 months away)
- Can run one copy of Mifos for all customers, but using different databases. (18 months away)
- Can run one copy of Mifos for all customers with one production database. (not in next 12 months)
- Data Warehouse features
- Consolidated data warehouse – Pentaho ETL cluster ETLs data from individual Mifos Cloud customers' data warehouses and consolidates to a single MySQL Cluster or Oracle Cluster.
- Hadoop or custom "big data" cluster batch jobs can do analysis on the entire data set.
...