Audit process:
Keeping audit log is very important in message oriented middleware. Sometimes standard audit log feature available with ESB tools are not enough. We need to build custom audit log. We should create a common reusable process which will be called by main process to create audit log.
There should be standard table defined which will contain audit log. But in that case it will be slower, if every process access the table to create audit records. Best practice is to create an audit queue where messages can be published. We can have a subscriber which will subscribe to the audit queue and write the audit log to the table.
Audit table should contain meta-data about messages. We can generate various reports from audit log. You can get a complete insight how ESB is performing from audit log.
Sometimes we create custom audit table without giving due importance of the fields of the table and later on we keep on adding new fields.
Typically audit table should have following attributes.
Audit Key: Primary key of the table. This should be incremented by one every time a record is inserted.
Program Name: Name of the program. This is applicable where under a single program there are multiple projects.
Project Name: Name of the project under the program.
Process Name: Name of the process producing audit log.
Event Name: Name of the Event /Message/Entity
MessageId: The MessageId uniquely identify the message. This can be used for reconciliation purpose.
Payload: This is optional. We don’t encourage to store payload in the database. It will increase size of audit tables rapidly and may deteriorate performance. Also if payload contain sensitive data, you should not store as a plan text. In that case, you need to encrypt the payload.
Source: Source System name.
Target: Target System name.
Error Code: This field will contain error code
Error Message: Detail error messages
Status: You can store success and failure status of messages in terms of successful transfer from source to target.
Received Timestamp: This will contain timestamp of when messages is received
Sent Timestamp: This will contain timestamp of when messages are sent/committed.
Log Timestamp: The timestamp where messages are logged.
Server Name: In a cluster environment it comes handy if you store the information of server which has processed the messages. You can monitor performance of each server in processing of messages.
Atribute1, Atribute2, and Attribute 3: You should have some optional attributes in the table. You may use to store additional attributes of messages.
This audit log can be used not only for transferring messaging, but can also be used to keep the audit log of other pattern such as file transfer, bulk data transfer etc.
It is better to accumulate all the data and call audit log process at the end of the processes.
You can have common sub process or you can post the data directly to the audit log queue.
We can implement queue based audit log with two processes
First process
Main process will call audit publisher messages. This process will validate the messages and will write the messages to the queue.
Second process.
This is a subscriber which will subscribe messages from the queue and write the messages in a database table.
You may directly send the messages to the audit queue. In that case you don’t need to have publisher process. In that case you need validate the messages before sending to the audit queue. Otherwise subscriber may reject the messages.