Publish Subscribe using Topic

Publish  Subscribe  using Topic  

 

This is a variant of publish/subscribe design pattern.  In this design pattern there are multiple subscribers for a single message. We need JMS Topic to implement the same. Each subscriber is given a single copy of the messages. In case of durable subscription, messages will be retained for the subscriber which is not normally  available at particular point of time.

 

When considering zero message loss policy, it is convenient to use virtual topic concept provided by some messaging provider. Messages are published to the topic. But messages are forwarded to the different queue as per predefined policies. Subscriber process consumes the messages from the corresponding queue. In this way message tracking are easy. If messages has persistence storage then, there will be zero message loss.

 

Publish Subscribe Design Pattern

publish subscribe design pattern

This design pattern can be achieved by using JMS QUEUE. In this exchange pattern publishers send the messages in the Queue. Subscribers get the messages from the queue.

We need three processes to implement this design pattern; a publisher process, a subscriber process and a error handler batch process.

Publisher process.

  1. Source application invoke this process through web service
  2. Publisher send the messages of the source application to the queue after successful validation
  3. Publisher sends the error messages to source application if validation fails.
  4. Publisher commit the messages to the Queue and send success response to the source application or send failure response if commit to the queue fails.

Subscriber process.

  1. Subscriber process listen to the Queue.
  2. Messaging system deliver the messages to the subscriber process
  3. Subscriber enrich and transform the messages and deliver to the target application
  4. The messages which could not be posted, are sent to the Error Queue or Error table.(You may use error table, instead of queue to achieve fine grained control)

 

Batch Process.

  1. This process handles the failed messages which could not be posted to the target application
  2. This is a scheduled process which read the messages from the message Queue and try to post the messages to the destination.
  3. The messages will be written back to the error queue which could not be posted.
  4. In case if the process support transaction, messages will be there in the error queue, if the messages cannot be posted to the destination.

Use Case

This is asynchronous design pattern variant. You should use this pattern if your integration layer has standard messaging support. It is better to use Queue instead of database to store the messages especially in cluster environment.

Also where there are single publisher and multiple subscriber, you should use this pattern. To handle this scenario with database is not recommended.

 

 

 

 

Asynchronous Design Pattern

Asynchronous Design Pattern

In this message exchange pattern, messages from source application are processed asynchronously.

Source application send the messages to the integration process. Integration process commit the messages in a database of integration layer and send the response back (HTTP OK) to the source application. Another integration process will pickup the messages from the database and send the message eventually to the target application.

The response message to the source application is normally HTTP 200 OK for success response or HTTP 500 for failure (or any other error code)   It is the responsibility of integration layer to send the messages to the target application.

We can implement this pattern with two processes.

First Process

  1. Source application will invoke the integration process
  2. The process validates the messages and after successful validation commit the messages in the database.
  3. If validation is unsuccessful, it sends HTTP error code to the source application.
  4. Send HTTP OK /HTTP Error according to the successful commit or failure.

 

 

Second Process

  1. This is batch process which is scheduled or it can be a listener process which will poll the database.
  2. This process will pick up messages from the database and then validate, enrich and transform the messages and invoke provider web service.
  3. This process will delete the record from the database after successful processing.

  Use Case

    Use this pattern when messages delivery from source to target takes significant amount of time.  Also for cloud to cloud communication, sometimes connection failure occurs vary frequently. In that situation it is better to choose asynchronous design pattern. Also when middle-ware needs to have lots of complicated business logic for messages enrichment and transformation, you may choose this design pattern.

Fire and Forget Design Pattern

Fire and Forget Design Pattern

This is simple message exchange pattern where requester does not wait from any specific response/acknowledgement from provider. The requester just need HTTP OK or Error message from integration layer as soon as message is received. The integration layer   continues to process the messages further after sending HTTP OK response.

The integration processes ensure messages delivery are guaranteed to the target system.

Asynchronous design pattern can make use of fire and forget design pattern.

  Process

  1. Requester will invoke the integration Process.
  2. The integration process will send HTTP OK response to the requester immediately without waiting for completion of the processes.
  3. The integration process validate, enrich and transform the request and invoke provider web service.
  4. The integration processes may store the messages in a database or queue.

 Exception Handling

  1. Process should be resubmitted by admin in case of error. Or target application can invoke the process again.

Use Case

When requester application does not need to know that request is reached to the final destination, this pattern can be used.  Also requester may provide callback service for getting the response from long running process.

Hybrid Design Pattern Variant

Synchronous Message Processing –Hybrid Design pattern; Handle request and response message  Asynchronously in case of failure.

This pattern is a variant of hybrid design pattern. The difference is response messages also needs to be queued in case of failure.    The pattern is rarely used. To implement this pattern you should have 3 processes. 1 Main process and additional 2 batch processes.

Main Process.

 

  1. The main  process will validate the request and if messages are ok, enrich and transform the messages and invoke provider service.
  2. The process  will send the response in real time to the requester
  3. The process will send the  failed messages( which could not be posted)  to the request message queue.
  4. The process will send the  response messages  which could not be posted back to requester to response message queue.

Process 2.

This process  should be a scheduled processes which will read the messages from the queue or database as similar to earlier hybrid design pattern.

Process 3.

This process should be scheduled process which will read the file from Response messages queue and will try to push the messages to the source  or requester application. This is similar to process 2.

Use Case

This is similar to hybrid pattern use case. But  here additionally we are saving provider response messages so that you don’t need to invoke provider we services again in case of connection failure with requester.

 

Sandbox Server Topology

This server   is uncontrolled server. Normally access is given to every developers to play with menu, interfaces, configuration etc. Also this server can be used for patch testing and version upgrade.  You don’t need to have this environment if you don’t want to invest more on infrastructure. But we recommend to have this server.

 

 

preprod server topology

Preproduction environment should be exact replica of production environment. Though you don’t need to have DR setup for preprod and corresponding WAF and LB setup. It should have same infrastructure; same hardware in terms of machine, CPU, memory and peripherals.  This environment can be used as staging area before codes are moved to production environment. Also this environment can be used for performance testing. Because this   is replica of production environment and by doing performance testing in preprod, you can ensure that system will react similarly in production with the same load. So you can determine how interfaces will work in different load condition in production.

It is not mandatory to have preprod environment.  Many of server setup in reputed organization does not have prepod. But it is good to have specially when performance is paramount important and one should do performance testing before production migration of the interfaces.

You can simulate normal load situation in production using preprod .You can also simulate Disaster situation load by switching servers on and off.

In two nodes preprod cluster you can have 2 nodes in primary site. You don’t need DR site servers.

 

In four node clusters topology , you should have 4 nodes ;two node in primary and 2 nodes in DR site.

QA Dev Server Topology

QA servers are used for user acceptance testing. It can be used for integration testing before actual user acceptance testing.

Dev server is used for development and unit testing.  Also it can be used for dry run.

 

It is better to have similar configuration for Prod, Preprod, QA and Dev server. If prod has server cluster, other lower environment also should have server cluster. Very often we avoid this aspect to save money on infrastructure which create inconvenience in later stage.

Sometime we see that lower environment like QA and DEV has single server installation. They don’t have server clusters. It creates problem.  You may get correct data in those environment .But you may get incorrect data in production. Even process can fails.  This is especially true when message sequencing is required.  You will not notice this issue in single server environment. So multi server and multi-threading processing should be taken care of beforehand.

Also QA, Dev server might be in intranet, prod server might be in cloud; it also create issues.

If you have public URL for prod server, you should have public URL for all the lower environments. All the testing (UT, SIT, UAT) should be done using public URL. Your server may have both, intranet and internet URL (public) . If you use intranet URL in production ,all the testing in lower environment should be done using intranet URL. You should not mix up intranet /internet URL across environment.   Also if your prod server is in cloud, you should have all the servers of QA, DEV in cloud.  It may be seen that some installation don’t have intranet URL for production. Also production server may be in cloud, whereas test environments server intranet. You should avoid this configuration. You should maintain uniformity across environments. Otherwise you will face many unforeseen issues.

 

Sandbox : This server   is uncontrolled server. Normally access is given to every developers to play with menu, interfaces, configuration etc. Also this server can be used for patch testing and version upgrade.  You don’t need to have this environment if you don’t want to invest more on infrastructure.

 

Four Nodes Server Cluster Topology

 

Four  Nodes cluster:

In this architecture topology, we have 2 nodes in primary site and 2 nodes in DR site. All the nodes are active and take loads equally. They are connected with primary site load balancer in normal situation. DR site servers points to primary site database and shared file system. If disaster strikes, the traffic is routed through to DR site load balancer and correspondingly DR site database and file system are used.  So in normal situation we have 4 nodes sharing the load and during disaster we have 2 nodes taking the load.  Also when DR site is not available, primary site nodes take entire loads.

Fault Tolerant architecture:

This is very flexible and fault tolerant server setup. In parallel it ensures equal utilization of resources. This architecture has minimum human intervention requirement in case of disaster.

Failover:  In case of disaster, no special activities to be performed for server setup perspective.    External application and database connection url   may change due to disaster if external application also affected.  In that case configuration needs to be changed accordingly.   If server uses internal database, database url may also needs to be changed assuming database will be replicated using oracle dataguard or any other technology. If local dns entry is changed, then no need to change any connection url.

Failback: No intervention required unless, external urls invoked by servers also get changed due to disaster. Internal database connection url may need to be changed. If local dns entry is changed for local or intranet application, then you may not need to change anything.

Disadvantage: This 4 node active clusters works well when primary and DR site are in close proximity and connected with high speed network. Don’t use this type of configuration if the locations are far or connected with slow network. In that case some request will be served faster and some request will be served slower.  This configuration is not recommended in that case.

Requirement Capture Template

We should have requirement capture templates for integration project which to be shared with customer. Smart customer will fill it up or you need to fill it up after discussion with the customer. Normally integration architect fill up the template after discussion with customer.  Best way is to have workshop with the customer to understand the requirement and fill up the template. Later you can use the template in URS document or FS document. The spreadsheet comes handy to understand the requirement and subsequently effort estimation.

The template contains message type, security protocol, source system, destination system etc.

To know server location is also important when multi site integration infrastructure is available. i.e you may have integration servers available in    Europe, USA.  So if source application and target application is in Europe, then you should install the integration processes in Europe. There will be latency in process execution, due to cross Atlantic traffic flow, if you install the processes in US server.

From the interface requirement sheet you can roughly calculate how much process to be developed, what design pattern to be followed. You can use this template to calculate effort and cost to implement the projects.

Following standard template you can refer and share it customer.

Part1

API Name Source Connection Protocol Destination Connection Protocol Source  Application Destination Application Location of Source end Point Location of Destination End Point Authentication Requirement (Source to ESB/ESB to Destination)
Order Master REST SOAP SFDC SAP Cloud EU Basic/Basic -HTTPS
Customer  Master REST jdbc CustMG SFDC EU Cloud Basic/2Way SSL
Location REST SOAP SCM SAP Cloud EU Basic/2Way SSL
Contacts REST REST CustMG OrdMG Cloud EU Message Signing
Organization SFTP SFTP SAP SCM US Cloud Basic/Basic
Products SFTP Jdbc SAP SCM US Cloud Basic/Basic
Utilization SFTP jdbc SAP SCM US Cloud Basic/Basic

Part2

Source Payload
(xml/json)		 Destination Payload
(xml/json)		 Data Enrichment requirement Data Transformation Requirement Number of Messages Per day(or TPS)/Files  SLA Response Time Sec File Size /Message Size  Encyption

Decryption Requirement
XML JSON  two external table lookup one field numeric to char, system date conversion  100 per day 10 sec 100KB Yes
XML JSON One external webservice lookup system date conversion  100 per day 10 sec 100KB Yes
XML JSON two external table lookup system date conversion  100 per day 10 sec 100KB No
XML JSON  two external webservice  lookup NA  100 per day 10 sec 100KB No
Text Text NA NA 5 files per Day 10 Min 10MB No
XML XML NA NA 5 files per Day 10 Min 10MB No
XML XML NA NA 5 files per Day 10 Min 10MB Yes

Part3

Exception Handling Requirement Audit Requirement Synch/Asynch Scheduling Frequency Initial Synch Up requirement Message Sequencing Requirement Other requirement
Email to Support Team Email success/failed report  by Day end Asynch, Publish Messages in Q NA Yes Yes messages to retried to post three times in case of connection failure
Email to Support Team Email success/failed report  by Day end Asynch, Fire and forget NA Yes Yes
Email to Support Team Email success/failed report  by Day end Synch NA No No
Email to Support Team Email success/failed report  by Day end Asynch,Fire and forget NA No  No
Email to Support Team No NA Daily No No
Email to Support Team No NA Daily No No
Email to Support Team No NA Weekly No No