...
Both systems provide plugins to extend the features of the corresponding message system
Demo
A spike
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
For the demo, we used a similar setup to production, this was changed to use a message queue instead e.g.
| Code Block |
|---|
public class TisAuditRepository implements AuditEventRepository {
private static final String AUDIT_ROUTING_KEY = "audit_queue";
private ObjectMapper mapper = new ObjectMapper();
@Autowired
private AmqpTemplate amqpTemplate;
@Override
public void add(AuditEvent event) {
try {
amqpTemplate.convertAndSend(AUDIT_ROUTING_KEY, mapper.writeValueAsString(event));
} catch (JsonProcessingException e) {
throw new RuntimeException(e.getOriginalMessage());
}
}
@Override
public List<AuditEvent> find(String principal, Instant after, String type) {
throw new UnsupportedOperationException();
}
} |
The audit logs were pulled by Logstash was used to collect the data and feed it to Elasticsearch. Logstash has modules that can ben enabled to allow the automated pull from Kafka or RabbitMQ. The setup for this was
| Code Block |
|---|
input {
kafka {
bootstrap_servers => "10.160.31.49:9092"
topics => ["audit_topic"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_temaplate => false
index => "audit_kafka"
}
} |
| Code Block |
|---|
input {
rabbitmq {
hosts => "10.160.31.49"
queue => "audit_queue"
durable => true
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_temaplate => false
index => "audit_kafka"
}
} |
| Code Block |
|---|
spring:
rabbitmq:
host: localhost
port: 5672
username: guest
password: guest
kafka:
bootstrap-servers: localhost:9092
consumer.group-id: myGroup |
This quick demo was sucessful in that we managed to automate all audit logs from going to the log files to Elasticsearch - to Elasticsearch via a message queue with logstash
Resources
https://jack-vanlightly.com/blog/2017/12/4/rabbitmq-vs-kafka-part-1-messaging-topologies
...