Synchronous Message Processing – Request Response Pattern

This is the simple and most frequently used design pattern for application integration.

Source application invoke web service and wait for reply. The integration layer post the messages to the destination and send back reply to the source application.  Until response is received source application thread is blocked.

Validation:

Incoming messages should be validated against schema before forwarding it to destination. Schema can be stored in the database or in filesystem. If validation fails error message should be sent to requester.

Enrichment:

Message enrichment may be required before posting the message to the destination application.  External web services, database, filesystem or other source can be used to enrich the messages.  For complex business logic, you might have to make multiple web service call or you may have to do service orchestration. But this should be avoided in synchronous design pattern. No complex data enrichment or complex business rule should not be implemented in synchronous design pattern.

Integration process should be very light weight and should be able to send response within few seconds as requester is waiting for reply and connection timeout would happen if response not sent in shorter duration.

Transformation:

Ideally transformation should happen after data enrichment is completed. We should do field mapping from source to destination. Normally middleware tools provide drag and drop facility for field mapping.

Auditing:

Best practice is you have to keep audit logs of the service invocation.

Like when request has been sent by source application and when response has been sent back. You can post audit messages to the queue or database.  For faster response time you can write audit log in asynchronous way like sending the audit messages in a queue. You can write the audit messages directly to the database in case database response time is faster.

Audit information can contain source application, destination application, timestamp, and payload. If payload is big you can extract id /key values or some significant fields from the payload and store it.

Error Handling:

In case of error, you should notify requester application. You can populate HTTP Error code or you can provide error messages in the body as agreed upon. For business logic error you should normally pass message in the message body along with HTTP error code.

HTTP error code may be set to   401 (unauthorized) 500 (internal server error) or any others depending on the cause as per standard.  Also you can use any custom error code. But this should be agreed upon with respective project team of source application.

You should not retry to connect destination multiple times because source system thread is blocked and waiting for response.

Duplicate checking:

Duplicate checking is very important for non-idempotent provider web services. Normally duplicate messages comes mostly due to retry operation from the service requester application. Sometimes due to network issues TCP connection might be broken. Though messages are successfully delivered, requester application could not get response. So it retries and it results in duplicate messages.

If duplicates messages are passed to provider application and it may change the state of the field values. In case of idempotent web services duplicate messages are not a challenge but for non-idempotent services, duplicate messages should be rejected.

We can do this simply by keeping messages id or any key values in the look up table or in server cache. Server cache should be the most preferred option as it would be faster to retrieve value from the cache.

For single instance of server keeping the last processed message id in the cache will solve the issues. But for server cluster distributed caching should be used. If requester application is single threaded then keeping the last message id in the distributed cache and compare it with incoming message id will resolve the issues.

For multithreaded clients or cluster clients you need to store   messages ids for last few minutes or hours. For most of the integration requirements, it might be enough.  For zero duplicate toleration it is good to use look up table, where you can store all the previous message id in the database. Incoming messages should be looked up and validated against this table.

But it will have performance overhead. Remember to set cache expiry to reasonable time limit so that cache does not grow big.

Usages:

This pattern can be used when provider application can response back to requester within few seconds.    When requester needs to send messages to provider in real time, this pattern should be applied. Requester application should take the responsibility of error handling, like   communication fault, system faults or business logic faults.

We need to have   a single process to implement this design pattern in ESB.

 Requester will invoke the Integration application web service and wait for response.

1. Integration application will validate the request and if messages are ok, enrich and transform the messages  and invoke provider service.
2. Integration application will send the response in real time to the requester
3. The other reusable sub process  for audit, error handling to be developed and called from main process accordingly.
4. In case of error , this process will call common error handling sub  process

 

I would like to provide you a practical guide on application integration. I have come up with the patterns that we commonly use in integrating application.

Success of the integration project depends on selection of suitable design pattern.

The design patterns described below may be used as a reference for the developers and architects for building new integration process.  Suitable design pattern should be selected for implementation as per customer requirement. Complex design pattern should be avoided to reduce cost of development and maintenance.

Find below the list of the standard patterns. This list is commonly followed approach in EAI.

 

Type	Pattern
Message Processing	Synchronous Message Processing – Request Response Pattern
	Synchronous Message Processing – Request-Response pattern; Handle failed request asynchronously
	Synchronous Message Processing – Request-Response pattern, handle both failed request and response asynchronously
	Asynchronous Fire and Forget Pattern
	Asynchronous Message Processing – Delayed response
	Asynchronous Message Processing –Asynchronous Response.
	Asynchronous Message Processing – Queue as staging area for requester messages
	Asynchronous Message Processing – Queue as staging area for response messages
	Asynchronous Message Processing – Queue as staging area  for request message and response messages
	Asynchronous Message Processing -Publish and Subscribe pattern
	Asynchronous Message Processing-Requester read the status of messages  from a database
FTP  Processes	Standard FTP Process – One Source to One destination
	Standard FTP Process – One Source to Multiple (N) Destinations.
	Standard FTP Process – Multiple Sources (N) to Multiple (N) Destinations.
Bulk Data Processing	Bulk data processing– Read database and call web services to process each records individually.
	Bulk data processing– Read the file and call web services to processes each record individually.
	Bulk data processing– Read File from share directory, process each record individually and Merge and write back to the shared directory/FTP server.
	Bulk data processing– Read File from FTP server and process each record individually and FTP the file.
	Bulk data processing– Read multiple Files from FTP server and processes each record individually and FTP the files.
	Bulk data processing – Single message  containing  multiple records
	Sending reports in the mail – Read the database or file and produce a report in a scheduled process
Database Polling	Poll databases for new records
Special  Messaging Need	Asynchronous Message Processing – Message to be processed in order of receiving.
	Asynchronous Message Processing – Asynchronous Request, – Message set (co-related) to be processed in order irrespective of receiving order.
Message Coordination	Message aggregation
	Message Split
	Message Orchestration
Messages Security	Authentication
	Encryption
	Threat prevention in messages content.
	Transport layer data protection
Message Storage	Database vs Message Queue
Other Topics	Will Come up later

 

Follow the next post for details of each design pattern.