Syncing trainee data

Before we discuss

Read up on the summary.

Migrate Trainee Data from TIS to TIS Self Service Infra

Go over the summary

• We’ve got tis and trainee VPCs set up

• A full migration has been tested for stage data

• There are 2 main ways of sourcing events;

  1. Use custom database triggers to invoke a lambda (or similar)

  2. Use CDC from Amazon DMS to send events through Kinesis (or similar)
     
     

Data model for Trainee Self-service

Where to source data change events

Either:

• built into the Aurora DB; database triggers.

  • more configurable; able to build up ‘logical’ entities from across tables.

  • reduces overall processing (additional calls for data & restricting what events are sent)

• Use DMS or one of several CDC alternatives; database logs.

  • no need to change the database implementation or re-evaluate on every schema change (looser coupling to source).

  • better fit for existing CDC (rabbitMQ)

Sending data across VPCs

There are alternatives to using a Lambda. Lambdas can definitely access across VPCs.

Messaging / Streaming

• Kinesis (Data Streams) is comparable to alternative ‘streaming’ middleware like Kafka.

• It sits in the space where we currently use RabbitMQ for ‘messaging’.