Transaction support is being added for ESR Microservices. Transactions apply to RabbitMQ message sending and MongoDB inserts/updates.
Transactions in RabbitMQ
Nothing needs to be done on the RabbitMQ side to use Transactions. The configuration and setup that’s required is all done within our ESR Microservices.
There are 3 basic requirements to enable RabbitMQ Transactions:
call
setChannelTransacted(true)on the RabbitTemplate Spring Bean.register a
org.springframework.amqp.rabbit.transaction.RabbitTransactionManagerSpring Beanadd the
org.springframework.transaction.annotation.TransactionalAnnotations to the code which we want to wrap in a Rabbit Transaction. This Transactional Annotation works for both Mongo and Rabbit Transactions.
Transactions in MongoDB
Transactions are a relatively new feature of MongoDB (since v4.0)
https://docs.mongodb.com/manual/core/write-operations-atomicity/
To use MongoDB Transactions - we have to have the MongoDB setup in ReplicSet mode AND we have to make configuration/setup changes with our ESR Microservices.
MongoDB ReplicaSets
Simply put, a MongoDb ReplicaSet is a cluster in primary/secondary mode. All writes are sent to the master and then the data is propagated to the secondaries. You can setup a MongoDB ReplicaSet with just one node - this is good for testing Transactions but a single node ReplicaSet is not a good idea for production use cases.
If a MongoDB is setup in ReplicaSet mode, it will generally have “replicaSet=” within it’s MongoDB connection string. Here’s an example of a mongodb connection string for a replicaset:mongodb://txuser:txpass@mongo1.hee.com:27011,mongo2.hee.com:27012,mongo3.hee.com:27013/transdb?authSource=transdb&replicaSet=rs0
For further details see
NOTE: if we want to use MongoDB Transactions we must point to a MongoDB in ReplicaSet mode AND setup the code to use Transactions.
We cannot have the code to setup to use transactions and point to a MongoDB not in ReplicaSet (Transaction supporting) mode.
We cannot have the code setup without transaction support and point to a MongoDB in ReplicaSet (Transaction supporting) mode.
If we have the code setup with transaction support we must point to a MongoDB in ReplicaSet (Transaction supporting mode)
If the code is setup without transaction support we must point to a MongoDB not in ReplicSet (Transaction supporting mode)
In the code, there are 3 steps required to use MongoDB Transactions.
register a
org.springframework.data.mongo.MongoTransactionManagerSpring Bean.add the
org.springframework.transaction.annotation.TransactionalAnnotations to the code which we want to wrap in a MongoDB Transaction. This Transactional Annotation works for both Mongo and Rabbit Transactions.make sure any Mongo Collections (Collections are a similar to Tables in a typical RDBMS) you need to use within a Transaction are created outside of the Transaction, in advance. When using Mongo Transactions - you can no longer create a collection ‘on demand’ within a transaction by simply inserting into it. To work around this there is a class
com.hee.tis.esr.common.transactions.MongoCollectionInitializerConfigwhich looks for instances of Spring Data Repositories and creates any Mongo Collections required on application startup. This is a temporary solution until we are using something like mongobee to make sure Collections are created as part of the Microservice Installation process.
The MongoTransactionManager works with inserts/updates to mongo through Spring Mongo Data repositories.
There are not a great number of examples showing how Spring works with MongoDB Transactions. The example here is a bit limited.
https://github.com/spring-projects/spring-data-examples/tree/master/mongodb/transactions
It relies on custom embeddedMongoDb for testing and seems to suggest you need to use mongoTemplate when doing inserts/updates to Mongo within a Transaction. This is misleading - you can use Spring Data Mongo Repositories which are easier to use than mongoTemplates.
Global Transaction Manager
We have linked the RabbitMQ and MongoDB Transaction Managers so that they work together. To do this we create a parent/global transaction manager which delegates its commit & rollback calls to its child (Mongo and Rabbit) Transaction Managers. We use the Spring provided ChainedTransactionManager which must have 1 or more child transaction managers ( You can’t use ChainedTransactionManager without any child transaction managers). The ChainedTransactionManager is registered as a Primary TransactionManager so Spring knows which of the Transaction Managers to use when there is more than 1.
We create a RabbitTransactionManager, if required, based on a boolean property app.transactions.rabbit.enabled having value ‘true’.
We create a MongoTransactionManager, if required, based on a boolean propertyapp.transactions.mongo.enabled having value 'true’.
If have at least 1 Rabbit or Mongo Transaction Manager, then we create a ‘parent' transaction manager.
If we have neither Rabbit or Mongo transaction manager - we register a dummy/NoOp - TransactionManager. If we have code annotated with ‘@Transactional’, Spring demands that we have a Transaction Manager - so we register a dummy, NoOp TransactionManager to keep it happy. We cannot switch off the ‘@Transactional' annotation, so we just use a dummy Transaction Manager.
We have created our own class for this com.hee.tis.esr.common.transaction.NoopTransactionManager.
based on the abstract library class org.springframework.transaction.support.AbstractPlatformTransactionManager
Common Classes in the com.hee.tis.esr.common.transactions package, used to support Rabbit and Mongo Transactions. The intention is that these common classes end up in a common shared library - so the code is only in once place.
# | Class Name | Description |
|---|---|---|
1 |
| A Spring Annotation used to pull in the Transaction Support. ESR Microservices wanting to pull in TransactionSupport should add the |
2 |
| Looks for Spring Data Mongo Repositories on startup and creates the required Collections in MongoDB if they don’t already exist. |
3 |
| Noop/Dummy Transaction Manager class. See |
4 |
| On application startup, logs the actual TransactionManager being used. This is just to help with debugging. |
5 |
| Creates a |
6 |
| Creates a MongoTransactionManager Spring Bean if the property If created, this Spring Bean is named |
7 |
| Creates a Spring Bean using |
8 |
| If the property
|
9 |
| Contains public constant values related to this package. Things like Spring Bean names and important property names. |
Testing Transaction Support
We have done some work to allow us to test Transactions that apply to both Rabbit and MongoDB using Test Containers. By using Test Containers we use actual instances of MongoDB and RabbitMQ run in Docker Containers in our tests - this is useful because we are using actual instances not ‘test’ instances which might behave different from the ‘real' thing.
The base code for testing transactions is within the following packages
com.hee.tis.esr.common.transactions.test
# | Class | Description |
|---|---|---|
1 |
| This abstract class has some tests which check that Rabbit and Mongo Transactions are setup correctly. It relies on the superclass having a ‘@SpringBootTest’ which refers to the Spring Boot Application (ESR microservice ) that has Rabbit and Mongo Transactions enabled and configured correctly. The test code is common but the Configuration of the Spring Boot Application will be specific to each ESR microservice. To create an instance of this test
Instances of this test create their own Rabbit and Mongo Resources to check that each Spring Boot application has transactions correctly configured. A concrete example of this can be found in the ESR Data Export service - the class is |
2 |
| A Spring annotation which Enables Mongo Repositories and loads the Spring Config from |
3 |
| Test Spring Bean Configuration. Creates a test queue, sets up a RabbitMQ listener which will put any messages received from the test queue into another Spring Bean which is a simple list of Strings (for testing). |
4 |
| A POJO which maps to the Test Mongo Collection called |
5 |
| A Test Spring Data Mongo Repository interface for the TxMessage POJO & |
6 |
| A Test Spring Service which is used to test Rabbit and Mongo Transactions. The process method sends RabbitMQ messages and updates MongoDB and can conditionally throw Exceptions. |
7 |
| Contains public constants like test queue names, this package name ( for Spring Scanning), Test Spring Bean names, the Spring profile name for bringing MongoDb with ReplicaSet as a a Test Container. |
com.hee.tis.esr.common.autoconfigure
This package contains the class 'EmbeddedMongodbReplicaSetBootstrapConfiguration'. This class it loaded via Spring AutoConfiguration because it’s referred to in the smallsrc/test/resources/META-INF/spring.factories file.
org.springframework.cloud.bootstrap.BootstrapConfiguration=\ com.hee.tis.esr.common.autoconfigure.EmbeddedMongodbReplicaSetBootstrapConfiguration
This class starts up MongoDB in a Test Container in ReplicaSet mode. It also sets the following properties which can them be used by the test code:
embedded.mongodb.port embedded.mongodb.host embedded.mongodb.database embedded.mongodb.replicaset
We had to write custom code to setup MongoDB in a TestContainer in ReplicaSet mode so we can test transactions. Our custom code is based on the classcom.playtika.test.mongodb.EmbeddedMongodbBootstrapConfiguration from the library “com.playtika.testcontainers:embedded-mongodb:1.32“ - it starts MongoDB within a Test Container but it’s not in ReplicaSet mode.
0 Comments