...

Disaster recovery

Database

TODOMost of persistent data is stored in RDS.  This implies the data is highly available as it is replicated synchronously in two availability zones, however it is certainly not impossible to lose an entire region e.g. due to natural disaster etc.  In addition to relying on multi-AZ functionality we also save and encrypt daily full mysqldump to cloudboss (in us-east-1b AZ) here: https://cloudboss.mifos.org/cloud   In the event of disaster you would need to download and decrypt the snapshot and create a new RDS instance and follow the instructions that apply when migrating an MFI from the old infrastructure.

Front-end

Application server (Tomcat/Jetty). What clients hit.

In the event of AZ being unavailable or hardware failure.

...

(Tomcat/Jetty). What clients hit.

In the event of AZ being unavailable or hardware failure.

• Identify which situation, by checking if other nodes in the same AZ are available or not.
• If hardware failure, simply launch a new instance with the appropriate AMI, add it to chef config, remap elastic ip, mount volumes, etc.
• If an entire AZ is down then
  • create new volumes in alternate AZ, and retrieve from backuppc the uploads, custom reports etc.
  • relaunch each frontend into an alternate AZ, add to chef config, remap elastic ips, mount new volumes etc.

Statefiles

Statefiles are lists of specific versions of packages to be included in images. They are kept in the statefiles/ dir in the "cloud" git repo.

...