The Mifos API and Web Services

This tech note describes work begun on implementing a prototype API and Web Services infrastructure for Mifos. It summarizes the work, lessons learned, and additional considerations. It also provides recommendations for follow-on activities.

Background

Work to develop an API and Web Services for Mifos was suggested in this section from Evolving Mifos:

The services of the system should be exposed as a set of web services. A web services API would provide a significant level of flexibility in the composition of services into applications for the range of delivery vehicles described earlier. Some system architects and developers object to the introduction of such an API given the early stage of system development. The objections relate to the need to support multiple versions of an evolving API. While the concern is valid, it is perhaps mitigated by the fact that there are any number of useful financial services APIs that can be used as models; the project would not be inventing in a vacuum.

An API, combined with a strong scheduler framework, provides the basis for integration with third party systems, such as cash management and accounting systems. It should be noted that an API would ease the pressure on the system to be "all things to all people", and instead allow it to focus on being the central element in what may sometimes be a collection of systems that best fit the needs of a given MFI: Choosing Mifos ceases to be an all-or-nothing proposition, and the system may gain in competitiveness as a result.

Some Considerations and Concepts

Services vs. API

One might be asked to describe the difference between a Service and an API. The term is loosely defined, but one might refer to WikiPedia for one version: http://en.wikipedia.org/wiki/Systems_Architecture_Service

For our purposes, one might suggest that we have, in the existing Mifos code, sets of functions related by business concept corresponding roughly to the "modules" specified in the beta iterations of Mifos. These include Customer Account, Loan Account, Fee, etc. Each of these business functions/modules have defined within them certain actions, which correspond to what we will define as services, i.e. specific actions, either discrete or sequenced as a discrete unit of work, performed on objects in the business domain.

The existing code succeeds in varying degrees to be expressive of the business domain. However, integrators (Mifos specialists) and new developers approaching Mifos for the first time find the overall complexity of code remains high, even after substantial improvements from refactoring efforts such as the M1 to M2 migration. Those approaching Mifos with the intention of customizing and deploying it in the field, and not necessarily interested in Mifos internals, may still find approaching the Mifos code a daunting task, and could benefit from a simplified mental model and programming support.

It is this more approachable, platform- and toolkit-oriented presentation of the underlying Mifos services that is the motivation for Mifos API development. The API will provide the Mifos specialists and developers with access to those services as we have just defined them, and to supporting structures such as operating context (underlying communications structures such as a "Session") and object metadata. It will do so in a way that reduces overall complexity, as well as providing an approachable mapping between business concepts and the underlying implementation of solutions with Mifos.

Web Services we will define as Remote Facades to those underlying services, layered on top of the API.

So there is a layer cake of items in the services-enabled, fully API-based mifos:

+-------------------------------+-------------------------------+
|                               |  mifos custom utils and apps  |
+           mifos web           +-------------------------------+
|                               |       mifos web services      |
+-------------------------------+-------------------------------+
|                     mifos services (API)                      |
+-------------------------------+-------------------------------+
|                        mifos lib (M2)                         |
+---------------------------------------------------------------+
|              mifos server and DAO (Hibernate, M2)             |
+---------------------------------------------------------------+
|                        mifos Database                         |
+---------------------------------------------------------------+

Note that this is not the current implementation, but rather a direction toward which the project may evolve.

Objects vs "Modules" -------------------- As we have suggested in our definition of services, there is a correspondence between the modules as specified in the work done in the beta iterations of Mifos and business objects and the services that interact with those objects:

Beta "Modules":

Center
CheckList
Client
Cron Jobs
Customer Account
Fee
Fund
Group
Loan Account
Loan Product Definition
Login
Meeting
Notes
Office
Personnel
Product Category
Roles & Permissions
Savings Product Definition
Scheduler
Security

Here is a non-comprehensive attempt at a list of Mifos Objects (with corresponding "Modules"):

Account (Customer Account)
Center (Center)
Checklist (Checklist)
Fee (Fee)
Group (Group)
LoanAccount (Loan Account)
LoanProduct (Loan Product Definition)
LoanTransaction
Office (Office)
Note (Notes)
Product (Loan Product Definition, Savings Product Definition)
ProductCategory (Product Category)
RepaymentSchedule (Scheduler)
Role (Roles & Permissions)
SavingsAccount (Savings Account Definition)
ScheduleTask (Cron Jobs)
User (Login, Security)

On each of these objects, one can then define a standard set of methods. Instead of working bottom up, one can work top down, since there are already many objects in the existing Mifos code base, exposing a substantial set of methods, not all of them consistent. By working top-down, we can define the API in terms of the Web Services facades that will eventually be exposed, work down another layer into the services layer objects and methods--the business objects as well as operating context for the API--and then have a basis on which to define activities, if necessary, to modify the base Mifos code to support the API.

REST vs. SOAP

It is possible to present two styles of Web Services bindings, both REST and SOAP. There are precedents illustrating the usefulness of exposing both styles, for example at Amazon.com --> http://www.oreillynet.com/pub/wlg/3005

REST Methods usually include (by definition):

GET
POST
PUT
DELETE

SOAP Methods might include:

create
retrieve
update
delete
find
getDeleted
getUpdated
describe

The tradeoffs here are twofold:

Simplicity in interface. REST relies on the standard methods already built into HTTP, whereas SOAP supports richer specifications of interaction, but with more complexity in not only the wire protocol, but, some would argue, in programming.
Tools availability. One would argue that simplicity in web services may come with support for programming tools and standard practice. There are no toolkits for REST, and client programs often hand-code the processing logic. SOAP clients in Java are often built using the Apache Axis toolkit, with its mature wsdl2java and java2wsdl tools. These tools essentially hide the Web Services layer from the programmer, allowing interactions with pure Java interfaces. Some would argue that in the context of mifos, given the existing investment in Java, SOAP support using Axis might represent the more desirable implementation path.

API Development

The development of a basic Mifos API involves defining a layer of simplified service objects that will improve the apparent complexity of the system for new developers and integrators. The layer will serve as a more or less fixed point of reference for development on both sides of the API. Those developing Mifos internals will be free to change the underlying implementation with minimum impact on developers using the API, provided a contract (in the form of interfaces) and a sufficiently complete set of tests are shipped with the published API specification.

Existing Business Objects and logic would be exposed via a simple set of objects that new developers could manipulate in a straightforward manner.

An implementation of facades that unify the underlying Business Objects/Value Objects/DAOs/Services/Processors scheme could provide the necessary simplification that could then be continued in a straightforward way into a Web Services implementation using a common Web Services infrastructure like Apache Axis.

The facades (defined via Java interfaces) represent the contract between system developers and integrators working with the system, and will allow system developers to restructure the underlying implementations over time, minimizing disruption to the larger community of users and integrators.

Prerequisites for an API

(partial) Rationalization of domain model
Create mifos .jar (library) without dependencies on Struts or Hibernate

Requirements

The following are characteristics the finished API should exhibit:

Transport neutral
Channel security via SSL
Implements methods in existing business layer
Follows financial services industry standards as applicable (e.g. IFX)
Backwards compatibility: extensible API
Deployable independent of the Struts-based web application

Approach

Prototyping Activities (Completed)

Remove dependencies on container such that API can be deployed independently of the
Struts Web Front End
Identify key object(s) to perform a simple query and, time permitting, a create operation
Code Sample of Service Layer, to show reduction of complexity and collecting of concerns
Define Sample contract as WSDL
Generate Server Skeletons and Client Stubs
Implement Server code and test client(s)
Demonstrate working API and Web Services stack through simple query operation
Time permitting, Demonstrate working API and Web Services stack through simple create operation

Activities for Full API Implementation (Future)

Define key business cases
Obtain real, applicable sample data
Identify/Reconcile applicable MifosObjects and methods
Write tests
Enhance existing codebase where necessary
Write JavaDoc and Code APIs/Sample Clients
Test
Iterate until done

Activities for Web Services (Future)

Create/Modify Web Services Contract (WSDL)
Generate/Regenerate Service Skeletons and Sample/Test Client Stubs
Write JavaDoc and Code Skeletons and Sample/Test Clients
Test
Iterate until done

Notes on Work to Date

Prototyping Activities (Completed)

During August and September 2006, the following activities were undertake to build a prototype of the API, to understand better the issues and complexity the project will face in building and exposing support for a full Mifos API and Web Services stack.

Remove dependencies. Dependencies on JNDI and on the JBoss container were removed, such that a program driven from the command line could start and drive a mifos application context, independent of a web container. The result, built through the lib and lib-x ant targets, was usable from a simple commmand line program. However, complete independence from Struts was not achieved, because of a number of Mifos framework dependencies on EntityMaster. At the time, it was deemed more expedient to bundle the required org.mifos.framework.struts classes, rather than struggle with code that was slated to be refactored as part of the M1 to M2 migration.
Identify key object(s). Based on discussions related to likely requirements for API-based applications, the project decided that the Loan object would be a useful focus. An attempt was made to isolate the Loan class from as many object dependencies as possible, to allow quick implementation of a Service Layer. We were able to do this for a query operation on a Loan, narrowing the required dependencies to: _ org.mifos.application.accounts.loan.business.LoanBO _ org.mifos.application.accounts.loan.business.service.LoanBusinessService _ org.mifos.framework.business.service.ServiceFactor _ org.mifos.framework.exceptions.ServiceException * org.mifos.framework.util.helpers.BusinessServiceName
Code Sample of Service Layer. A Loan object, representing a simplified interface to Mifos's LoanBO and other objects, was developed. It served as the fundamental building block for the prototype Service Layer. In addition, MifosService, which serves as a single collection point for application context items, such as the Hibernate Session, was created. org.mifos.api.LoanProductDumper was implemented as a short test of the MifosService, and to ensure that the bootstrapping of the API independent of the web application was successful, within the constraints imposed by the continued presence of EntityMaster and the M1 to M2 migration. org.mifos.api.TestClient was also implemented, to demonstrate the working approach with arguments and to prepare for use of the Web Services stack.
Define WSDL. The next step was to define the contract that the Web Services built on the API and exposed to the world outside mifos would support and that clients, remote or otherwise, would expect. Since we fully expected to use SOAP, a WSDL definition was appropriate. MifosService.wsdl found in src/org/mifos/api/resources is the result. It contains the definition of a highly simplified Loan and a single method, findLoan. One might consider this level of definition a grossly insufficient oversimplification, until the following items become apparent: _ Almost all CRUD operations will require at least one argument _ Some methods will return a result * All operations require a remote invocation

Since the mechanisms for marshalling of data and message routing is the same in SOAP RPC style calls regardless of which of the CRUD operations is invoked, one can see that the definition is sufficient.

Generate Server Skeletons and Client Stubs. Using Apache Axis2's wsdl2java tool, both Server Skeletons and Client Stubs were generated. The following command line was issued from the Axis2 bin directory:

This command generated a number of key artifacts, modified during the prototyping process. It should be expected that these artifacts will need to be regenerated and potentially merged with older versions as the production version of the API is implemented and undergoes maintenance and enhancement. The key classes included:

The server skeleton code itself:

org.mifos.api.MifosServerSkeleton
org.mifos.api.MifosServerSkeletonInterface

Helper objects for marshalling:

org.mifos.www.services.FindLoanRequest
org.mifos.www.services.FindLoanResponse
org.mifos.www.services.Loan

The client stub, instantiated in a client program:

org.mifos.api.MifosServiceStub

Implement Server code and test client(s). MifosServerSkeleton's findLoan was implemented, using classes from org.mifos.api and org.mifos.ww.services for support.
Demonstrate working API and Web Services stack. Using the generated MifosServiceStub, a test client was implemented. That client takes a command line argument subsequently passed to the findLoan Web Services endpoint (http://localhost:8080/mifos/services/MifosService) to retrieve a number of basic Loan information items from a live Mifos database, using the Mifos system, and not raw database queries. A similar client was also implemented using Ruby and its built-in SOAP support. A similar C# client could also have been implemented as well. Both the implemented clients were used with success to retrieve a Loan information from a running Mifos system, with the same results.

Code

The code for the prototype API and Web Services stack have been integrated into the main mifos source distribution.

The code for the sample clients can found in the project Subversion repository at https://mifos.dev.java.net/svn/mifos/trunk/mifosSOAP.

Lessons and Decisions from the Prototyping Activities

Mifos as a Library

It has been demonstrated that it is possible to deploy a Mifos platform nearly independent of the Struts-based web application. In order to do so with success, the project should examine the M2 code to ensure that dependencies on EntityMaster are removed, and that no additional dependencies such as on a new Struts plugin are introduced.

Web Services by Contract

As discussed earlier, an API specification represents a contract, by which underlying development on Mifos internals and client program code on a variety of platforms and delivery vehicles may vary independently of one another. This characteristic generally will afford greater freedom in implementation choice and may in some cases afford a greater range of solutions for a given problem space.

As a WSDL defines a contract, the emerging WSDL2 standard defines support paths for both REST and SOAP styles of Web Services. During the course of the prototype, the question of which style of web service to support disappeared, given that the two major candidate toolkits both provided support. The more important question emerged: how, in fact, does one enable support for in .

Both the Java Web Services Developer Pack (JWSDP) and Apache Axis2 were considered. Neither represented an ideal solution, and either choice involved resolving a number of tradeoffs.

JWSDP offers strong support for existing and emerging web services standards, including with WS-* specifications. It also provides a mechanism for exposing web services through the use of Java5 annotations. While the use of annotations provides a simple mechanism for exposing key methods as web services, and could easily have been applied to Service Layer objects to expose Mifos web services, further consideration suggested that the use of annotations for web services was not in fact attractive.

Annotations in the Service Layer code are more mechanism than a clear specification, and while it may have been possible to retrieve an accurate WSDL from the server once the service were deployed, that WSDL would not represent a contract upon which client developers could rely to remain fixed: errors in the Service Layer code or annotations would change the WSDL to represent the new implementation as correct.

Axis2 offers code generation for both server and client code, producing server code skeletons and client stubs from a WSDL. There was also past experience with Axis1 on the project, and support sufficiently similar to JWSDP's for WS-* specifications through the use of modules. For these reasons, the prototype was implemented with Axis2. It is also for these reasons that the project should continue use if Axis2 in its full Web Services implementation.

No performance measures were taken during the prototyping process. It should be noted, however, that both packages have wide community support, and implement different versions of StAX (Streaming API for XML): SJSXP (JWSDP) and AXIOM (Axis2). Axis2 offers additional approaches to XML pull-through processing and tree caching that might be considered, should the need arise.

New Jars in lib

The following jars were added to provide Web Services support with Axis2:

XmlSchema-1.0.2.jar
annogen-0.1.0.jar
axiom-api-1.0.jar
axiom-dom-1.0.jar
axiom-impl-1.0.jar
axis2-adb-1.0.jar
axis2-codegen-1.0.jar
axis2-kernel-1.0.jar
backport-util-concurrent-2.1.jar
commons-codec-1.3.jar
commons-httpclient-3.0.jar
geronimo-spec-activation-1.0.2-rc4.jar
geronimo-spec-javamail-1.3.1-rc5.jar
geronimo-spec-jms-1.1-rc4.jar
jaxen-1.1-beta-8.jar
neethi-1.0.1.jar
stax-api-1.0.jar
wsdl4j-1.5.2.jar
wstx-asl-2.9.3.jar

New Java Packages

The Service Layer and key API and Web Services structures are in org.mifos.api.

Marshalling code are in org.mifos.www.services. The project may choose to reconsider this decision, but should be warned that the Axis2 generator code may pose difficulties that preclude a completely arbitrary selection of package name.

Unit Tests for API

Testing of the API was done with test clients invoked from the command line. The approach was to test end-to-end using command line clients that used the Service Layer objects alone, and then with another client that invoked the API through the Web Services stack.

Sufficient loan data and member data was required in order to test the services layer and the web services stack. As it was not clear how to input test data without the web front end, and because the relationships between tables was complex, no attempt was made to create a JUnit test suite for the single object and method call implemented as part of the prototype.

When the In Memory Database for testing is integrated with the project, these tests should be added, provided the test data is avilable. Again, the complexity of the data relationships made it impossible to create usable JUnit-based test fixtures and unit tests in the short time allotted for the prototype.

Changes to web.xml

References to the Axis2 servlet were added to web.xml, in order to embed SOAP support into the Mifos war. One should note that the Axis web administration utility application was not installed. The distribution used was the Axis2 minimal implementation, recommended by the Axis2 development team for embedding in standalone deployments --> http://apache.hoxt.com/ws/axis2/1_0/axis2-min-1.0-bin.zip

Key changes to build.xml

Three targets were added to the ant build.xml file:

lib: Create a standalone mifos library, deployable separate from the Struts web app.
lib-x: This should disappear. This is only to satisfy remaining mifos dependencies on Struts that could not be removed during the prototype, as M1 to M2 migration was in-process.
api: Run compile, copy_files, lib, and lib-x, so that we can test the API from the command line.

The existing .war target bundles the Axis2 support we embedded with the production application without additional work.

During the prototype, the use of the WSDL2Java code generation capability included with Axis2 was invoked from the command line, as described earlier in this document. Although it is not strictly necessary, the project might consider including a build target to invoke WSDL2Java when upgrades to the API and/or Web Services specification occur. Though ant support for wsdl2java is on the surface desirable, it should be noted that any invocation of the code generator will only generate blank skeletons, and will require developer time to merge existing code into the newly generated skeletons.

Security

There were three issues related to security that were considered briefly and deferred in the prototype:

Socket-level encryption (SSL)

Message-level encryption (WS-Security)

Identification and Authentication (UserNameToken and Rampart)

SSL

It is common practice to use HTTPS to ensure encrypted communications between client and server. Because this falls in the domain of web server configuration, no attempt was made to enable communication using SSL/HTTPS for the prototype, although it should be recommended for any production installation.

WS-Security

The Axis2 documentation indicates that Apache Rampart, implementing WS-Security, was bundled with the Axis2 distribution. This page provides pointers for how to engage the module in an Axis2 distribution

In fact, Rampart is not bundled with the existing Axis2 distribution. It is available from the Apache Axis2 Modules page

The WS-Security specification provides for encryption of message sections and sub-sections, and defines rules for processing. The project should review this specification, and consider it in light of real requirements, trading off implementation complexity (potentially high) with less complex options, like simple SSL encryption. The project may wish to undertake these activities with the help of a qualified security consultant.

Authentication

With the Apache Rampart module engaged, one can implement the server-side callbacks required for identification and authentication, using a UserNameToken

This step was not taken during the prototype, and should be considered a high priority next step.

Open Web Service

The team sould take steps to ensure that in a production deployment of mifos, the current entries for the Axis2 servlet are commented out, as they are not yet secured at the application level. Administrators may also consider using an HTTP proxy configuration, for example with Apache2's mod_proxy, configured such that calls to the mifosservice are ignored.

Server-side logging

No attempt was made, during the prototype, to integrate with the Mifos Logger. This activity will need to occur. Also important will be the definition of the logger hierarchy for the service layer and the web services messages.

It should be noted that although the service layer classes and the web services skeleton and support were placed in the same package for simplicity, it might be appropriate to separate them in the future. The consideration of the requirement to separate logging messages may be a factor in deciding the final packaging of the two groups of classes.

Error Handling/Exceptions ------------------------- Simple exception passthroughs were implemented in the protoype, such that exceptions thrown at any level in the ServiceLayer rise up to the top of the stack for analysis. A full implementation will require careful consideration of the exception handling appropriate at the server level and development of appropriate error messages to be returned in the SOAP envelopes, tailored according to the audience/application. This handling and communication is sufficiently different from that of the Struts front end that it should be called out as a separate item.

i18n and l10n

The Services Layer and Web Services support demonstrated in the prototype are not implemented with internationalization or localization support. If the full implementation requires localized messages, those messages should be identified up-front.

One might argue that since the SOAP API might be used in end-user tools, like for reporting, it may be appropriate to provide a set of "localized method calls". This point should be explored, if such requirements arise in discussions, in light of well-known web services implementations already deployed, such as the Google and Amazon APIs for various applications, as well as the standards for financial information exchange, such as IFX.

Next Steps

Based on the work done to date on implementing a prototype API and Web Services infrastructure for Mifos, the following next steps are recommended:

Define Data Migration APIs. Work on the API was undertaken with the clear intention that its products would eventually serve to satisfy critical user needs. Data Migration was an area identified as most appropriate to take advantage of an API. Therefore, a suitable next step would be for the project to articulate clear use cases for data migration, from which appropriate migration-oriented APIs can be identified. To the extent possible, these APIs should satisfy the requirements for data migration, while remaining as general in purpose as possible.
Iterate on Data Migration APIs. Development of the APIs and the data migration clients using those APIs should be done in short iterations, to ensure clear communication of requirements and possible solution paths, show frequent delivery of real results, and mitigate overall risk.
Unit test with real data. Changes in underlying database structure, as well as the inherent complexity of the database and object structures limited the possibility of testing with extensive amounts of "real" data in the context of automated unit and integration tests. As progress continues with the incorporation of an in-memory database for testing, the project should make every effort to provide automated unit tests with extensive data for testing.
Complete separation of mifos-lib pieces. As noted earlier in this document, it was not possible to ship the mifos libraries without Struts dependencies, because of ongoing changes related to the M1 to M2 migration occurring at the same time as the prototyping effort. Assuming not further dependencies have been introduced, the project should remove the dependency on EntityMaster, thus completing the work required to ship Mifos as a standalone library.
Clean up ant build. The lib-x target would no longer be required after completion of 4. The project should also consider creating API support as an optional, separately shippable build target, for cases where the Struts front end is not appropriate, while the library, its API, and Web Services may be, thus truly delivering Mifos as a platform for MFI applications.
Error Handling. The cases for error handling in the context of an API should be defined and implemented, both for the native Java bindings, and for the Web Services Infrastructure.
Security Requirements. The considerations identified in the notes on Security earlier in this document, with respect to the trade-offs between complexity (SSL only vs encrypted message blocks), as well as the appropriate implementation of authentication in the context of Mifos, should be explored as a high priority item, with the help of a qualified security consultant. Furthermore, if it chooses to use the existing Mifos authentication structures in its Web Services implementation, the project should consider creating a roles for Web Services users and administrators, or at least for administrators, so that the various features made possible through the API and Web Services infrasturcture may be exposed appropriately to the appropriate parties.
i18n, l10n. As discussed earlier in this document, the project should consider the requirements for message-level i18n and l10n, with respect to API verbs and error messages and feedback. The appropriate actions to take in this area are highly application and client dependent.

Longer Term

In the longer term, the project may also consider the following activities

Make web app use API (but NOT web services). The Struts-based web application may, in time, evolve to become a first-class client of the API, alongside other applications built using the Mifos API. At this point, the goal of Mifos as the platform for MFI applications will have been realized.
Create desktop and mobile apps for specific profiles, as suggested in Evolving Mifos. Given appropriate requirements and real needs from the field, there may be opportunities here to deliver significant gains in reach and operational value.
Develop support for IFX and other applicable standards, tailored for Microfinance. Attachments support in the Web Services specifications may be appropriate for exploring this path. Work in this area should be driven with real requirements from real MFI customers who will realize significant advantages from standards compliance. The project should identify a larger MFI in a position to provide vision and leadership in this area of standardization.

References

Apache Axis2. http://ws.apache.org/axis2
IFX Forum. http://www.ifxforum.org/standards/
Java Web Services Developer Pack. http://java.sun.com/webservices/jwsdp/index.jsp
OASIS Web Services Security. http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wss#technical
UserNameToken Authentication with Rampart. http://www.wso2.net/articles/rampart/java/2006/08/15/usernametoken-auth
XML Data Standards for Microfinance. http://cs.washington.edu/homes/tapan/papers/datastandards.pdf
Fielding, R. Architectural Styles and the Design of Network-based Software Architectures. http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm