...
- We agreed on using RabbitMQ as it seemed to be the most user friendly with possibly the largest market share
- Multiple queues will be used with the basis of a singular inbound queue, multiple internal ones and possibly multiple outbound queues
- The queues will use binding keys to route the messages rather than headers
- As a default, auto ack will be disabled which means that a consumer will need to manually acknowledge a message in order for it to be removed from the queue. This should only be done after the message has been processed and and output to be either pushed on another queue or saved. This ensures that messages will not be lost when consumers fail
- We will store the amount of retries on the message and a max of 10 retries will be allowed
- The wait time (back off) algo will be 1 minute before it can be processed again
- We'll attempt to use an intelligent circuit breaker which will look at the type of exception and judge whether it would make sense to retry. Errors such as HTTP 400's wont make sense to retry as it will be an issue with the client (which will most likely require dev)
- Dead letters will be split between different dead letter queues by type
- We'll need to come back to rate limiting as we do not want to overwhelm our own systems
- we'll need to base this on some actual metrics
- We'll have duplicate queues so that we can create an audit service to record message/events
- The granularity of audits will need to be defined as too granular may create an influx of data which may not be useful
Inbound message types:
-APC Applicant confirmation message
-POS Position information
-POR Position reconciliation
-PER Person record
-ADD Address
-QAL Qualifications
-ABS Absence
-DCC Notification confirmation message
Outbound message types:
-APP Applicants
-DNC/DNF file Notifications
-RTC confirmation messages
Messages - on hold
- As we're running messages through the system, it may be possible to reduce the amount of messages within a certain period
- There may be data that we don't care about and can remove from the messages
...