Synchronous Message Processing – Request Response Pattern

This is the simple and most frequently used design pattern for application integration.

Source application invoke web service and wait for reply. The integration layer post the messages to the destination and send back reply to the source application.  Until response is received source application thread is blocked.

Validation:

Incoming messages should be validated against schema before forwarding it to destination. Schema can be stored in the database or in filesystem. If validation fails error message should be sent to requester.

Enrichment:

Message enrichment may be required before posting the message to the destination application.  External web services, database, filesystem or other source can be used to enrich the messages.  For complex business logic, you might have to make multiple web service call or you may have to do service orchestration. But this should be avoided in synchronous design pattern. No complex data enrichment or complex business rule should not be implemented in synchronous design pattern.

Integration process should be very light weight and should be able to send response within few seconds as requester is waiting for reply and connection timeout would happen if response not sent in shorter duration.

Transformation:

Ideally transformation should happen after data enrichment is completed. We should do field mapping from source to destination. Normally middleware tools provide drag and drop facility for field mapping.

Auditing:

Best practice is you have to keep audit logs of the service invocation.

Like when request has been sent by source application and when response has been sent back. You can post audit messages to the queue or database.  For faster response time you can write audit log in asynchronous way like sending the audit messages in a queue. You can write the audit messages directly to the database in case database response time is faster.

Audit information can contain source application, destination application, timestamp, and payload. If payload is big you can extract id /key values or some significant fields from the payload and store it.

Error Handling:

In case of error, you should notify requester application. You can populate HTTP Error code or you can provide error messages in the body as agreed upon. For business logic error you should normally pass message in the message body along with HTTP error code.

HTTP error code may be set to   401 (unauthorized) 500 (internal server error) or any others depending on the cause as per standard.  Also you can use any custom error code. But this should be agreed upon with respective project team of source application.

You should not retry to connect destination multiple times because source system thread is blocked and waiting for response.

Duplicate checking:

Duplicate checking is very important for non-idempotent provider web services. Normally duplicate messages comes mostly due to retry operation from the service requester application. Sometimes due to network issues TCP connection might be broken. Though messages are successfully delivered, requester application could not get response. So it retries and it results in duplicate messages.

If duplicates messages are passed to provider application and it may change the state of the field values. In case of idempotent web services duplicate messages are not a challenge but for non-idempotent services, duplicate messages should be rejected.

We can do this simply by keeping messages id or any key values in the look up table or in server cache. Server cache should be the most preferred option as it would be faster to retrieve value from the cache.

For single instance of server keeping the last processed message id in the cache will solve the issues. But for server cluster distributed caching should be used. If requester application is single threaded then keeping the last message id in the distributed cache and compare it with incoming message id will resolve the issues.

For multithreaded clients or cluster clients you need to store   messages ids for last few minutes or hours. For most of the integration requirements, it might be enough.  For zero duplicate toleration it is good to use look up table, where you can store all the previous message id in the database. Incoming messages should be looked up and validated against this table.

But it will have performance overhead. Remember to set cache expiry to reasonable time limit so that cache does not grow big.

Usages:

This pattern can be used when provider application can response back to requester within few seconds.    When requester needs to send messages to provider in real time, this pattern should be applied. Requester application should take the responsibility of error handling, like   communication fault, system faults or business logic faults.

We need to have   a single process to implement this design pattern in ESB.

 Requester will invoke the Integration application web service and wait for response.

1. Integration application will validate the request and if messages are ok, enrich and transform the messages  and invoke provider service.
2. Integration application will send the response in real time to the requester
3. The other reusable sub process  for audit, error handling to be developed and called from main process accordingly.
4. In case of error , this process will call common error handling sub  process