GSOC 2013 - Distributed Caching support

Content

Identify

Project summary 

Mifosx is built with the multitenancy concept in which a single Mifosx deployment can server several organizations. It is very important to consider performance optimization in this multitenant environment. The main focus of this project would be to implement a distributed caching system to cache most frequently accessed data.

Since MifosX is a stateless application, adding caching can contribute in a greater degree to improve the API request processing performance. However the tricky part of this project is addressing  the multitenant nature.

Spring framework caching abstraction will used on top of the data-service methods which needs to be cached. And JDK concurrent has map will used as the initial caching storage, this can be easily replaced by a distributed caching storage system in future.

Project implementation 

Git hub development branch : Link

 

  1. Test the caching functionality by implementing it on few selected methods.

       We can easily enable tenant aware caching by annotating the target method with @cachable

@Cacheable(value = "users", key = "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().gettenantIdentifier().concat(#username)")
    public UserDetails loadUserByUsername(final String username) throws UsernameNotFoundException, DataAccessException {
		//method body
	}

I have subdivide the cache based on the tenant name by adding tenant name to the parameters of the target method.

As stated in the summary, I have used a concurrent hash map to store the cached results.

<cache:annotation-driven />
	<!-- generic cache manager -->
	<bean id="cacheManager" class="org.springframework.cache.support.SimpleCacheManager">
		<property name="caches">
			<set>
				<bean
					class="org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean"
					p:name="clients" />
				<bean
					class="org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean"
					p:name="users" />
				<bean
					class="org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean"
					p:name="tenants" />
			</set>
		</property>
	</bean>

 

  1. Identify most suitable data-service methods to be cached.
  2. implement caching on the identified methods.
  3. Decide and plugin a suitable distributed cache storage.

Monitoring MifosX platform with Spring Insight

Spring insight is a great tool to monitor java web applications. I've used this tool to measure the performance of MifosX platform before and after adding caching.

Below are the steps to deploy MifosX on a spring insight instance.

 

  1. Download tc Server Developer edition from the VMware Download Center.

  2. Create a directory for tc Server, for example:

    /home/tcserver

  3. Unpack the tc Server archive into the directory created in the previous step, such that the tc Server home directory is a sub- directory of it, for example:

    /home/tcserver/vfabric-tc-developer-2.x.x.x.
  4. Create a tc Server instance where Insight will run:

    1. In a terminal window, change to the tc Server home directory.

    2. Create a tc Server instance with tcruntime-instance.sh or tcruntime-instance.bat, using the insight template.

       command:  ./tcruntime-instance.sh create Insight -t insigh


  5. Start the new tc Server instance using the tcruntime-ctl.sh or tcruntime-ctl.bat command.(This command should execute from .../insight/bin/)

    • On Unix, to start an instance named "Insight": 

      ./tcruntime-ctl.sh Insight start
    • On Windows, install a Windows service for the instance before you start it the first time (After that, you can control the service from the Windows services control panel.) To install an instance named "Insight" as a Windows service, and then start it :

      1. tcruntime-ctl.bat Insight install
      2. tcruntime-ctl.bat Insight start

     

  6. Deploy MifosX application to tc Server by copying the WAR file to the webapps directory of the tc runtime instance. update conf/server.xml file to support MifosX application.

    (How to create Amazon public AIM from scratch). Below is a sample server.xml file.

    <?xml version="1.0"?>
    <Server port="${base.shutdown.port}"
            shutdown="SHUTDOWN">
        <Listener className="org.apache.catalina.core.JasperListener"/>
        <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
        <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"/>
        <Listener className="com.springsource.tcserver.serviceability.deploy.TcContainerDeployer"/>
        <Listener accessFile="${catalina.base}/conf/jmxremote.access"
                  authenticate="true"
                  bind="127.0.0.1"
                  className="com.springsource.tcserver.serviceability.rmi.JmxSocketListener"
                  passwordFile="${catalina.base}/conf/jmxremote.password"
                  port="${base.jmx.port}"
                  useSSL="false"/>
    			  
        <GlobalNamingResources>
            <Resource auth="Container"
                      description="User database that can be updated and saved"
                      factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
                      name="UserDatabase"
                      pathname="conf/tomcat-users.xml"
                      type="org.apache.catalina.UserDatabase"/>
    				  
    		<Resource type="javax.sql.DataSource"
                name="jdbc/mifosplatform-tenants"
                factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
                driverClassName="com.mysql.jdbc.Driver"
                url="jdbc:mysql://localhost:3306/mifosplatform-tenants"
                username="root"
                password="mysql"
                initialSize="3"
                maxActive="10"
                maxIdle="6"
                minIdle="3"
                validationQuery="SELECT 1"
                testOnBorrow="true"
                testOnReturn="true"
                testWhileIdle="true"
                timeBetweenEvictionRunsMillis="30000"
                minEvictableIdleTimeMillis="60000"
                logAbandoned="true"
                suspectTimeout="60"
           />
        </GlobalNamingResources>
    	
        <Service name="Catalina">
            <Executor maxThreads="300"
                      minSpareThreads="50"
                      name="tomcatThreadPool"
                      namePrefix="tomcat-http--"/>
            <Engine defaultHost="localhost"
                    name="Catalina">
                <Realm className="org.apache.catalina.realm.LockOutRealm">
                    <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
                           resourceName="UserDatabase"/>
                </Realm>
                <Host appBase="webapps"
                      autoDeploy="true"
                      deployOnStartup="true"
                      deployXML="true"
                      name="localhost"
                      unpackWARs="true">
                    <Valve className="org.apache.catalina.valves.AccessLogValve"
                           directory="logs"
                           pattern="%h %l %u %t &quot;%r&quot; %s %b"
                           prefix="localhost_access_log."
                           suffix=".txt"/>
                </Host>
            </Engine>
    		<Connector protocol="org.apache.coyote.http11.Http11Protocol"
               port="8443" maxThreads="200" scheme="https"
               secure="true" SSLEnabled="true"
    		   keystoreFile="C:\res\mykey"
               keystorePass="mifostest"
               clientAuth="false" sslProtocol="TLS"
               URIEncoding="UTF-8"
               compression="force"
               compressableMimeType="text/html,text/xml,text/plain,text/javascript,text/css"/>
    		   
            <Connector URIEncoding="UTF-8"
                       connectionTimeout="20000"
                       port="${insight.http.port}"
                       protocol="HTTP/1.1"/>
    				   
    				   
        </Service>
        <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"/>
    </Server>
  7. Access the Spring Insight Dashboard by opening http://localhost:8080/insight in your browser.

Troubleshooting 

  • MifosX application context initialization might fail sometimes (guess it's due to version incompatibilities of spring jars with spring insight instance), this can be fixed by copying spring jars from insight web application lib folder to mifosx lib folder.Make sure to restart the insight instance after copying lib folders.

Identification of suitable methods to be cached

 

There were total of 27 API end point classes which were observed in-order to choose the best candidates for caching.

  • closure api
  • staff api
  • glaccount api
  • calendar api
  • journalentry api
  • charge api
  • rule api
  • client api
  • Makercheckers api
  • collateral api
  • codes api
  • fund api
  • configuration api
  • group api
  • dataqueries api
  • guarantor api
  • document management api
  • loanproduct api
  • security api 
  • note api
  • holiday api
  • savings api
  • monetary api
  • search api
  • office api
  • user administration api
  • loanaccount api
 

 

Important results gathered from spring insight

  • Every API request is filtered through the TenantAwarebasicAuthentication filter. And this filter performs 4 database read operations on M_APPUSER, M_OFFICE, M_APPUSER_ROLES and M_ROLE_PREMISSIONS. So this is the very first candidate to be chached in mifosx platform.
    • Target method : TenantAwareJpaPlatformUserDetailsService.loadUserByUsername

  • JDBCTenantDetailsService.LoadTenantById , this method is also invoked before the execution of every API request.
    • Blocker : Although this method is compatible with all the requirements to be eligible for caching, it doesn't get cached.


Service End pointCached methodT1 

T2

T3Notes
Organization related
GET/mifosng-provider/api/v1/offices

OfficeReadPlatformServiceImpl.retrieveAllOffices [OfficeAPI]

40.821.812.1Most frequently used end point. (Used in 6 places of the reference UI application)
GET/mifosng-provider/api/v1/offices/templateOfficeReadPlatformServiceImpl.retrieveAllOfficesForDropDown [OfficeAPI]29.416.39.4This method is used in 13 places of the MifosX platform
GET/mifosng-provider/api/v1/clients/template

OfficeReadPlatformServiceImpl.retrieveAllOfficesForDropDown [OfficeAPI] is the only data base call (which is already cached) invoked in ClientReadPlatformServiceImpl.retrieveTemplate [Clients API] no need of caching Clients API retrieveTemplate()

32.425.814.9Creation of a new client is one of the most frequently used feature of MifosX
GET/mifosng-provider/api/v1/clients/{clientId}ClientReadPlatformServiceImpl.retrieveOne [Clients API] *(Suffering with issue 3 mentioned in challenges section)   This method is used in 8 places of the MifosX platform
GET/mifosng-provider/api/v1/groups/{groupId}GroupReadPlatformServiceImpl.retrieveOne [Groups API] *(Unable to cache - issue 3 )   Used in 8 places of MifosX
GET/mifosng-provider/api/v1/loans/template

LoanProductReadPlatformService.retrieveAllLoanProductsForLookup [Loans API]

LoanReadPlatformService.retrieveLoanProductDetailsTemplate [Loans API]

LoanReadPlatformService.retrieveClientDetailsTemplate [Loans API]

LoanReadPlatformService.retrieveCalendars [Loans API]

GroupReadPlatformService.retrieveOne [Groups] * (issue 3)

LoanReadPlatformService.retrieveGroupDetailsTemplate [Loans API]

LoanReadPlatformService.retrieveGroupAndMembersDetailsTemplate

StaffReadPlatformService.retrieveAllLoanOfficersInOfficeById [Staff API]

StaffReadPlatformService.retrieveAllLoanOfficersInOfficeAndItsParentOfficeHierarchy [Staff API]

43.825.615.5Most frequently used feature
GET/mifosng-provider/api/v1/savingsaccounts/templateSavingsAccountReadPlatformService.retrievetemplate [Savings accounts API] *(issue 1 ,3)25.8   
GET/mifosng-provider/api/v1/accountingrulesAccountingRuleReadPlatformService.retrieveAllAccountingRules [Accounting rules API] *(self invocation methods - issue 2)    
GET/mifosng-provider/api/v1/chargesChargeReadPlatformService.retrieveAllCharges [Charges API]27.419.610.2 
GET/mifosng-provider/api/v1/fundsFundReadPlatformService.retrieveAllFunds [Funds API]31.621.710.8This method is used in 4 places of the MifosX platform
GET/mifosng-provider/api/v1/staffStaffReadPlatformService.retrieveAll [Staff API] *(issue 2)    
System related
GET/mifosng-provider/api/v1/codesCodeReadPlatformService.retrieveAll [Codes API]34.226.412.7 
GET/mifosng-provider/api/v1/codes/{codeId}/codevaluesCodeValueReadPlatformService.retrieveAllCodeValues [Codes API]30.225.914.1 
GET/mifosng-provider/api/v1/datatablesReadWriteNonCoreDataService.retrieveDatatableNames [Datatables API]45.225.513.2 
GET/mifosng-provider/api/v1/reports

ReadReportingService.retrieveReportList [Reports API] *(issue 2)

    

 

T1 - Avg. response time before caching (ms)

T2 - Avg. response time after caching only the end point (ms)

T3 - Avg. response time after caching + caching the security filter (ms)


CPU Used

Intel(R) Core(TM) Duo CPU P8700 @ 2.53
GHz

CPU cores2
Cache size3MB

 

Method : Above T1,T2,T3 values were obtained by averaging 5 API request processing times which were obtained using spring insight made via Firefox REST client.

Most frequently used functions on MifosX platform


ActionAssociated API requests
Add new clientGET/mifosng-provider/api/v1/clients/template
Add new loan applicationGET/mifosng-provider/api/v1/loans/template
Enter repayment 

 

Spring insight results for retrieve client without caching

Spring insight results for retrieve client

Cache eviction

Cache eviction is really an important consideration, it is not enough to cache results but it is also necessary  to evict the stale or unused data from the cache. Spring abstraction allows us to use another annotation to achieve this goal.

  • What are the methods that should cause cache eviction?

All the update and delete methods of the above identified cacheable methods of the data service layer must be cached. It is obvious that the delete or the update operation on a particular entity must trigger cache eviction.

 Enabling cache eviction on updateOffice :

@Transactional
@Override
@Caching(evict = {
            @CacheEvict(value = "offices", key = "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().gettenantIdentifier().concat(#officeId)"),
            @CacheEvict(value = "office_template", key = "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().gettenantIdentifier().concat(#officeId)") })
public CommandProcessingResult updateOffice(final Long officeId, final JsonCommand command) {
//method body
}

 

The target cache eviction methods are on XXXWritePlatformServiceJpaRepositoryImpl classes.

Challenges

  1. Spring does not provide an option to make use of a custom key generator class. It becomes bit tricky in situations where we have multiple parameters in the target cache-able method ,since we are adding  the tenant name to each and every method parameter to split the cache.
  2.  'Caching' skipped when method that is using @Cacheable is called through self-invovation
        - by default spring uses 'proxy' mode to support annotation based caching - as a result, it cant pick up on 'self-invocation' calls to cached method. This would impact the benefits of caching on methods identified, so its seems like we would have to use the 'aspectJ' mode instead of 'proxy' mode. On the above identified cacheable end points there are two classes (StaffReadPlatformServiceImpl, ReadReportingServiceImpl)  that causes to use aspectJ mode

Returned values depends not only with the method arguments but also with the logged in user.

    /*
     * Although this method is eligible for caching we cannot add caching for this method
     * ,as the return value also depends on the logged in user.
     * */    
    // @Cacheable(value = "clients", key =
    // "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().getName().concat(#clientId)")
    public ClientData retrieveOne(final Long clientId) {
        try {
            AppUser currentUser = context.authenticatedUser();
            String hierarchy = currentUser.getOffice().getHierarchy();
            String hierarchySearchString = hierarchy + "%";
            String sql = "select " + this.clientMapper.schema() + " where o.hierarchy like ? and c.id = ?";
            ClientData clientData = this.jdbcTemplate.queryForObject(sql, this.clientMapper,
                    new Object[] { hierarchySearchString, clientId });
            String clientGroupsSql = "select " + this.clientGroupsMapper.parentGroupsSchema();
            Collection<GroupGeneralData> parentGroups = this.jdbcTemplate.query(clientGroupsSql, this.clientGroupsMapper,
                    new Object[] { clientId });
            return ClientData.setParentGroups(clientData, parentGroups);
        } catch (EmptyResultDataAccessException e) {
            throw new ClientNotFoundException(clientId);
        }
    }

Plugging in different cache managers

So far I've been using the default cachemanager provided by spring cache module. It is most suitable for small non distributed environments. When the systems grows JDK inmemory hash map will not be sufficient to server the demand. There are couple of alternatives available off the shelf for example Ehcahce, Memcached, Spring Gemfire. I've tried both ehcache and memcached they all works pretty well with spring cache abstraction thus there are couple of challenges remain with integration. More details about these framework integration can be found here.

Links to Stack Overflow posts.

Additional Information