Pagination:

This is a type of bulk processing. ESB pulls the data from source system page by page due to restriction placed in the firewall.  Each page can have multiple records. Suppose a restriction is enforced that for a web service call, no more than 6MB data can be sent through firewall. Let’s say source application needs to send 40MB of data to target. So ESB should make 7 web service calls to pull entire 40MB of data. For each call source application will only send 6MB of data. This is called a page.

Source system should support pagination. Page number should be passed as parameter to the source. Page number can be mapped to row number as well.

Sequential pagination.

Flowchart.

ESB gets the data from source, one page after another in sequence and pass to the target in sequence

Usage:

  Use this pattern, when records are to be processed sequentially. This pattern is slow.

Parallel pagination.

ESB get pages in parallel from source and send it to destination in parallel.

Usage: Use this pattern, when faster processing is required and sequential processing of records is not important.

Combining pages in ESB layer

Here ESB layer gets the pages from source, combines and send to target as one document.

Usage: Use this pattern when target needs all the records in one document.