Content

Identify

Project summary 

Mifosx is built with the multitenancy concept in which a single Mifosx deployment can server several organizations. It is very important to consider performance optimization in this multitenant environment. The main focus of this project would be to implement a distributed caching system to cache most frequently accessed data.

Since MifosX is a stateless application, adding caching can contribute in a greater degree to improve the API request processing performance. However the tricky part of this project is addressing  the multitenant nature.

Spring framework caching abstraction will used on top of the data-service methods which needs to be cached. And JDK concurrent has map will used as the initial caching storage, this can be easily replaced by a distributed caching storage system in future.

Project implementation 

1. Test the caching functionality by implementing it on few selected methods.

       We can easily enable tenant aware caching by annotating the target method with @cachable

@Cacheable(value = "users", key = "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().gettenantIdentifier().concat(#username)")
    public UserDetails loadUserByUsername(final String username) throws UsernameNotFoundException, DataAccessException {
		//method body
	}

I have subdivide the cache based on the tenant name by adding tenant name to the parameters of the target method.

As stated in the summary, I have used a concurrent hash map to store the cached results.

<cache:annotation-driven />
	<!-- generic cache manager -->
	<bean id="cacheManager" class="org.springframework.cache.support.SimpleCacheManager">
		<property name="caches">
			<set>
				<bean
					class="org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean"
					p:name="clients" />
				<bean
					class="org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean"
					p:name="users" />
				<bean
					class="org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean"
					p:name="tenants" />
			</set>
		</property>
	</bean>

1. Identify most suitable data-service methods to be cached.
2. implement caching on the identified methods.
3. Decide and plugin a suitable distributed cache storage.

Monitoring MifosX platform with Spring Insight

Spring insight is a great tool to monitor java web applications. I've used this tool to measure the performance of MifosX platform before and after adding caching.

Below are the steps to deploy MifosX on a spring insight instance.

Troubleshooting 

• MifosX application context initialization might fail sometimes (guess it's due to version incompatibilities of spring jars with spring insight instance), this can be fixed by copying spring jars from insight web application lib folder to mifosx lib folder.Make sure to restart the insight instance after copying lib folders.

Identification of suitable methods to be cached

There were total of 27 API end point classes which were observed in-order to choose the best candidates for caching.

Important results gathered from spring insight

• Every API request is filtered through the TenantAwarebasicAuthentication filter. And this filter performs 4 database read operations on M_APPUSER, M_OFFICE, M_APPUSER_ROLES and M_ROLE_PREMISSIONS. So this is the very first candidate to be chached in mifosx platform.
  • Target method : TenantAwareJpaPlatformUserDetailsService.loadUserByUsername
    
    
• JDBCTenantDetailsService.LoadTenantById , this method is also invoked before the execution of every API request.
  • Blocker : Although this method is compatible with all the requirements to be eligible for caching, it doesn't get cached.
    
    
    

T1 - Avg. response time before caching (ms)

T2 - Avg. response time after caching only the end point (ms)

T3 - Avg. response time after caching + caching the security filter (ms)

Method : Above T1,T2,T3 values were obtained by averaging 5 API request processing times which were obtained using spring insight made via Firefox REST client.

Most frequently used functions on MifosX platform

Spring insight results for retrieve client without caching

Cache eviction

Cache eviction is really an important consideration, it is not enough to cache results but it is also necessary  to evict the stale or unused data from the cache. Spring abstraction allows us to use another annotation to achieve this goal.

• What are the methods that should cause cache eviction?

All the update and delete methods of the above identified cacheable methods of the data service layer must be cached. It is obvious that the delete or the update operation on a particular entity must trigger cache eviction.

 Enabling cache eviction on updateOffice :

@Transactional
@Override
@Caching(evict = {
            @CacheEvict(value = "offices", key = "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().gettenantIdentifier().concat(#officeId)"),
            @CacheEvict(value = "office_template", key = "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().gettenantIdentifier().concat(#officeId)") })
public CommandProcessingResult updateOffice(final Long officeId, final JsonCommand command) {
//method body
}

The target cache eviction methods are on XXXWritePlatformServiceJpaRepositoryImpl classes.

Challenges

1. Spring does not provide an option to make use of a custom key generator class. It becomes bit tricky in situations where we have multiple parameters in the target cache-able method ,since we are adding  the tenant name to each and every method parameter to split the cache.
2.  'Caching' skipped when method that is using @Cacheable is called through self-invovation
       - by default spring uses 'proxy' mode to support annotation based caching - as a result, it cant pick up on 'self-invocation' calls to cached method. This would impact the benefits of caching on methods identified, so its seems like we would have to use the 'aspectJ' mode instead of 'proxy' mode. On the above identified cacheable end points there are two classes (StaffReadPlatformServiceImpl, ReadReportingServiceImpl)  that causes to use aspectJ mode

Returned values depends not only with the method arguments but also with the logged in user.

    /*
     * Although this method is eligible for caching we cannot add caching for this method
     * ,as the return value also depends on the logged in user.
     * */    
    // @Cacheable(value = "clients", key =
    // "T(org.mifosplatform.infrastructure.core.service.ThreadLocalContextUtil).getTenant().getName().concat(#clientId)")
    public ClientData retrieveOne(final Long clientId) {
        try {
            AppUser currentUser = context.authenticatedUser();
            String hierarchy = currentUser.getOffice().getHierarchy();
            String hierarchySearchString = hierarchy + "%";
            String sql = "select " + this.clientMapper.schema() + " where o.hierarchy like ? and c.id = ?";
            ClientData clientData = this.jdbcTemplate.queryForObject(sql, this.clientMapper,
                    new Object[] { hierarchySearchString, clientId });
            String clientGroupsSql = "select " + this.clientGroupsMapper.parentGroupsSchema();
            Collection<GroupGeneralData> parentGroups = this.jdbcTemplate.query(clientGroupsSql, this.clientGroupsMapper,
                    new Object[] { clientId });
            return ClientData.setParentGroups(clientData, parentGroups);
        } catch (EmptyResultDataAccessException e) {
            throw new ClientNotFoundException(clientId);
        }
    }

Plugging in different cache managers

So far I've been using the default cachemanager provided by spring cache module. It is most suitable for small non distributed environments. When the systems grows JDK inmemory hash map will not be sufficient to server the demand. There are couple of alternatives available off the shelf for example Ehcahce, Memcached, Spring Gemfire. I've tried both ehcache and memcached they all works pretty well with spring cache abstraction thus there are couple of challenges remain with integration. More details about these framework integration can be found here.

Links to Stack Overflow posts.

• Caching support for multitenant enviornment.
• Spring custom annotations for caching.
• Spring cache support for pure JDBC read operations.
• How can I set a custom key generator for Spring cache

Additional Information

• MIFOSX-259 - Getting issue details... STATUS
• Spring cache documentation
• Spring source blog
• Proxy vs AspectJ caching with spring