...

One time setup to start working with cloud. There are two major tools that need to be setup, the amazon ec2/rds tools and the chef command line client. This guide assumes Ubuntu desktop, lucid release (10.04). Later versions of Ubuntu might work , tooas well. 

EC2/RDS command line tools

...

• • verify your connectivity:

knife node list

you should see a list of nodes that are currently managed by chef

Knife hints

Cookbooks

List cookbooks that chef server knows about

...

$ mkdir -p ~/git/mifos-cloud
$ git clone git://mifos.git.sourceforge.net/gitroot/mifos/cloud ~/git/mifos-cloud
$ cd ~/git/mifos-cloud
# update version before doing anything: vi chef/cookbooks/<cookbook>/metadata.rb
$ make changes
$ knife cookbook upload -o ~/cloud/chef/cookbooks <cookbook you changed>
$ git add/commit/push

...

You will need to install amazon-ec2 gem e.g. sudo gem install amazon-ec2 and knife should be working. Save the following script as Invoke maint/state.rb and then invoke it knife exec state.rb like so:

#!/usr/bin/env ruby

# invoke with knife:
# knife exec state.rb
#
# also make sure ruby gem for amazon ec2 is installed e.g.: sudo gem install amazon-ec2

require 'rubygems'
require 'AWS'

ACCESS_KEY_ID = ENV['AMAZON_ACCESS_KEY_ID']
SECRET_ACCESS_KEY = ENV['AMAZON_SECRET_ACCESS_KEY']

ec2 = AWS::EC2::Base.new(:access_key_id => ACCESS_KEY_ID, :secret_access_key => SECRET_ACCESS_KEY)

amis = {}

ec2.describe_images(:owner_id => "self").imagesSet.item.each do |image|
    name = image["name"]
    if !name.nil?
        m = name.match('CI\/EC2\/AMI\/(?:[^\/]+)\/(?:[^\/]+)\/TAG_hudson-cloud-(.+)(?:-)(.+)')
        if !m.nil?
            tag = m[1]
            if tag.eql? "mifos-image"
                tag = "mifos_2_1-bi_1_1-image"
            end
            amis[image["imageId"]] = [tag, m[2]]
        end
    end
end

mfis = {}

nodes.all { |n|
    if n.name =~ /^i-/
        mfi = n.override.tomcat.vhost
        env = (mfi.start_with? "testing-") ? "test" : "prod"
        mfi = mfi.match('(?:testing-){0}reporting-([^\/.]+).mifos.org')[1]
        mfis[mfi + " " + env] = [mfi, env, amis[n.ec2.ami_id][0], amis[n.ec2.ami_id][1]]
    end
}

printf("==============================================================\n")
printf("%-20s\t%-5s\t%-20s\t%s\n" % ["MFI", "ENV", "AMI", "BUILD"])
printf("==============================================================\n")
mfis.keys.sort.each { |mfi|
    printf("%-20s\t%-5s\t%-20s\t%s\n" % [mfis[mfi][0],mfis[mfi][1],mfis[mfi][2],mfis[mfi][3]])
}

How to change Pentaho to run reports and ETL against an RDS replica

If you wish to use the RDS instance for Mifos and Pentaho, ignore this section.

1. Set up RDS replica.

2.

knife role edit mifos_MFI

3. Edit override_attributes.pentaho.mifos_database_replica_host (optionally, adding this setting). "null" means fall back to override_attributes.mifos.database_host, and is the same as omitting override_attributes.pentaho.mifos_database_replica_host (see cookbooks/pentaho/recipes/default.rb in the cloud repo for details).

NOTE: nothing maintains Pentaho's database (ex: "MFISHORTNAME_prod_hib"), so "SourceDB" must be changed manually here.

NOTE: data sources in BIRT reports must be maintained manually, separately.

NOTE: data sources in Jasper Servers must also be maintained manually, separately.

Starting a new mifos/pentaho instance

• if converting an mfi from old infrastruture
  • Stop mifos instance
  • Dump database
  • Copy uploads/config in MIFOS_CONF
• Create security group in EC2 (AWS UI)
• SSH "gateways" setup/info
  • allows us to limit points of entry for our hosted machines
  • note hosts below in ec2-authorize commands... the gateways are currently birch.mifos.org (the whole Seattle GTC, actually) and cloudboss.mifos.org.
  • add to your .ssh/config (substituting MFINAME for something meaningful):

    No Format
    Host *MFINAME.mifos.org ProxyCommand ssh birch.mifos.org exec /bin/nc %h %p

• one-time setup for EC2 physical firewall
  • manually change EC2_ACCOUNT_NUMBER with the 12-or-so digit number fetched from the AWS console
  • SSH via gateways only - also images will constantly change so you WILL see the "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED" often. You can verify the key if you want to with ec2-get-console-output or via the ec2 web ui.
  • 18980-18981 is for monitoring JMX over RMI via OpenNMS
  • Create security group first via web ui or cli
     

    No Format

    #!/bin/bash # login through web console and create new security group, e.g. green-grameen set -ex SEC_GROUPS="digamber light-microfinance rise keef" EC2_ACCOUNT_NUMBER=000000000000 for SEC_GROUP in ${SEC_GROUPS} do ec2-authorize -P tcp -p 22-22 -s 75.149.167.24/32 ${SEC_GROUP} ec2-authorize -P tcp -p 22-22 -s 10.252.50.116/32 ${SEC_GROUP} ec2-authorize -P tcp -p 22-22 -s 184.72.240.48/32 ${SEC_GROUP} ec2-authorize -P tcp -p 443-443 -s 0.0.0.0/0 ${SEC_GROUP} ec2-authorize -P tcp -p 80-80 -s 0.0.0.0/0 ${SEC_GROUP} ec2-authorize -P tcp -p 18980-18981 -s 10.252.50.116/32 ${SEC_GROUP} ec2-authorize -P tcp -p 18980-18981 -s 184.72.240.48/32 ${SEC_GROUP} ec2-authorize -P udp -p 161-161 -s 10.252.50.116/32 ${SEC_GROUP} ec2-authorize -P udp -p 161-161 -s 184.72.240.48/32 ${SEC_GROUP} ec2-authorize -P tcp -p 161-161 -s 10.252.50.116/32 ${SEC_GROUP} ec2-authorize -P tcp -p 161-161 -s 184.72.240.48/32 ${SEC_GROUP} ec2-authorize -P icmp -t -1:-1 -s 10.252.50.116/32 ${SEC_GROUP} ec2-authorize -P icmp -t -1:-1 -s 184.72.240.48/32 ${SEC_GROUP} ec2-authorize -o ${SEC_GROUP} -u ${EC2_ACCOUNT_NUMBER} ldap done

• Create RDS security group
  • Authorize EC2 security group for MFI
  • Authorize default EC2 security group temporarily to make importing existing database more straightforward, remove after importing
• Create RDS instances as m1.small initially
  • v5.1.50
  • enable auto minor version upgrade
  • allocate 10GB
  • use MFI long name for MySQL instance (ie: "rise", "secdep")
  • initial user/pass can be anything simple, this will be changed later
  • leave Database Name blank
  • Db Parameter Group: "mifoscloud"
  • backup retention period: 8 days (best for PITR/binlogs)
  • backup window: 1600-1700 UTC (good for India/Philippines/Africa)
  • maintenance windows Saturday 1700-1800 UTC
  • example:

    No Format

    Engine: mysql Engine Version: 5.1.50 Auto Minor Ver. Upgrade: Yes DB Instance Class: db.m1.small Multi-AZ Deployment: Yes Allocated Storage: 10 DB Instance Identifier: rise Master User Name: mifos Master User Password: mifos Database Name: Database Port: 3306 Availability Zone: Using a Multi-AZ Deployment disables this preference. DB Parameter Group: mifoscloud DB Security Group(s):rise Backup Retention Period: 8 Backup Window: 16:00-17:00 Maintenance Window: Saturday 17:00-Saturday 18:00

• Create chef roles, base + test + prod + optional MFI specific recipe
  • look at an existing role
    • knife role show mifos_digamber
  • create a new role
    • knife role create mifos_rise
    • knife role create mifos_rise_test
      (look at mifos_digamber_prod, mifos_digamber_test for examples)
• Create 2 EBS volumes 1 for test and prod each (storing uploads)
  for testing:

  No Format
  ec2-create-volume --snapshot snap-5abd2f36 -s 1 -z us-east-1d ec2-create-tags -t Name=testing-digamber <vol-id>

  for prod:

  No Format
  ec2-create-volume --snapshot snap-5abd2f36 -s 1 -z us-east-1a ec2-create-tags -t Name=digamber <vol-id>

• Update DNS if required, this is typically required if this is a new mfi.
  • Allocate elastic ip
  • nslookup ip and get hostname
  • create CNAME in network solutions management console (use CNAME instead of A record so inter/intra-AZ data transfer will be charged at lower rate)
• Create EC2 instances
  • get ami-id from hudson job: https://ci.mifos.org/hudson/view/cloud/job/cloud-mifos-image
  • latest ami-id for 2.0.2:ami-8a8d7fe3 (see at end: https://ci.mifos.org/hudson/view/cloud/job/cloud-mifos-image/44/console)

    No Format
    ec2-run-instances ami-8a8d7fe3 --instance-type m1.small -z us-east-1d -d '{ "run_list": ["role[ldapclient]", "role[base]" ] }' --disable-api-termination -g rise ec2-create-tags -t Name=testing-rise.mifos.org -t Service=Mifos INSTANCE_ID

  • on boot, the node will add itself to the chef server (see rc.local, imaging/create_image.py, cloud source code)
  • make sure you can login via SSH. If not, fetch console output (this is something you may have to do from time to time):

    No Format
    ec2-get-console-output INSTANCE_ID

  • attach ec2 volume

    No Format
    ec2-attach-volume -i INSTANCE_ID -d /dev/sdc1 vol-ee7e2386

  • knife node edit INSTANCE_ID.mifos.org, add "role[mifos_rise_test]" to run list section or do through http://manage.opscode.com
  • log into box and run sudo chef-client to see change immediately or wait 30 minutes or so
• set up Mifos and Pentaho databases

  No Format
  sudo /etc/pentaho/system/mifos_pentaho_init.sql -u mifos -pmifos

  • change mifos password via AWS Web UI (Modify RDS instance, put a password generated with, for example, apg, in the "Master User Password" field)
• add backup jobs to BackupPC
  • when adding new backup host, use the NEWHOST=COPYHOST syntax mentioned on the "edit hosts" pageedit ~backuppc/.ssh/config , disabling strict host checkingedit hosts" page
• add monitoring of box to OpenNMS

...

Most of persistent data is stored in RDS.  This implies the data is highly available as it is replicated synchronously in two availability zones.  However, however it is certainly not impossible to lose an entire region e.g. due to natural disaster etc.  In addition to relying on multi-AZ functionality we also save and encrypt daily full mysqldump to cloudboss (in us-east-1b AZ) here: https://cloudboss.mifos.org/cloud   In the event of disaster you would need to download and decrypt the snapshot and create a new RDS instance and follow the instructions that apply when migrating an MFI from the old infrastructure.

...

Application server (Tomcat/Jetty). What clients hit.

In the event of AZ being unavailable or hardware failure.

...

.

In the event of AZ being unavailable or hardware failure.

• • • • • •  Identify which situation by checking if other nodes in the same AZ are available or not.
• If hardware failure, simply launch a new instance with the appropriate AMI, add it to chef config, remap elastic ip, mount volumes, etc.

• If an entire AZ is down:
• create new volumes in alternate AZ, retrieve from backuppc the uploads, custom reports etc.
  • only use "tar download" restore method, and only of /etc/mifos/uploads dir
  • download tar to local machine, then copy to remote host and untar as user "tomcat6"
• relaunch each frontend into an alternate AZ, add to chef config, remap elastic ipips, mount new volumes , etc.If an entire AZ is down then
• create new volumes in alternate AZ, and retrieve from backuppc the uploads, custom reports etc.
relaunch each frontend into an alternate AZ, add to chef config, remap elastic ips, mount new volumes etc

Statefiles

Statefiles are lists of specific versions of packages to be included in images. They are kept in the statefiles/ dir in the "cloud" git repo.

• updated from ci periodically (right now * */3 * * *)
• commited/tagged/pushed to "cloud" git repo at sf.net if there is a change (can monitor commit logs) to be notified of a change
• tag has build number and job name

Image maintenance

When upgrading machines, be sure to schedule outages.

Upgrades for new Mifos versions

...

• modify the AMI generation script for that Mifos release to use the new point release version. (We would modify imaging/mifos_2_1_bi_1_2.sh to get the 2.1.x Mifos version along with BI 1.2).
• update the "mifosversion" variable in the script to be 2.1.9 (commit and push)
• rerun re-run the hudson job "cloud-mifos_2_1-bi_1_2-image" to create a new AMI with the updated Mifos war (the name of the new AMI will be in the console log output on of the hudson job).
• follow the groovy script usage below using the new AMI generated in the previous step.

...

• Statefiles with lists of latest packages are created periodically (see above).
• An administrator must keep track of security releases in upstream Ubuntu packages.
• A groovy script is available to move a customer from one image to the next (it could be an upgrade or just a security update), this script should only be used if the mfi deployment for the environment already exists, volumes created, elastic ips associated etc. It can be invoked like so:

groovy upgrade.groovy <mfi e.g. rise> <environment testing|prod> <ami id>

...

When using the upgrade script use the "Long Name" from below for the MFI argument

Image production

Image production ci jobs are manually kicked off since there is a cost associated with storing Amazon EC2 images. Fire off the cloud-mifos-image job on the ci server to create a new image.

...

ldapdelete -x -W -D cn=admin,dc=mifos,dc=org -h ldap.mifos.org -ZZ 'uid=johndoe,ou=people,dc=mifos,dc=org'

Also, you should fill in their data bag with an invalid key e.g.:

...

ldapsearch -LLL -x -W -D cn=admin,dc=mifos,dc=org -h ldap.mifos.org -ZZ 'cn=*Jeff*'

The last argument 'cn=Jeff' can be adjusted accordingly based on which field you want to search on.

...

#!/bin/sh

# ./reset.sh johndoe <secret>

cat << EOF

dn: uid=$1,ou=people,dc=mifos,dc=org
changetype: modify
replace: userPassword
userPassword: $2
EOF

then you can invoke it like so:

./reset.sh johndoe <THE NEW PASSWORD> | sudo ldapmodify -x -W -D cn=admin,dc=mifos,dc=org -h ldap.mifos.org -ZZ

...

• Update role (usually) mifos_<MFI>
• run chef-client on each host to see changes immediately
• update uploaded reports in /etc/mifos/uploads/report with something like: sudo find /etc/mifos/uploads/reports -type f -exec sed -i -e 's/secdep-db.mifos.org/secdep.cz2a1vveusgo.us-east-1.rds.amazonaws.com/g' {} \;
• update pentaho datasources in <mfi_shortname>_<environment>_hib e.g. sec_prod_hib with a query similar to:

...