...
A number of ETL’s but mainly for the NDW
GMC (requires whitelisting)
NDW
...
The moon on a stick
With a somewhat complete list of the services and applications we currently use in TIS, we have a good idea how large or at least some idea of the scale of the task at hand. We next wrote a list of things we want from the migration, ensuring we would will provide the same or better level of service (of TIS) once we migrate as well as see/improve other things such as tooling while at it.
This is what we came up with:
...
We should start from the database out
Currently using a lot of VM’s, currently consuming a lot of DevOps time
Would be good to move the DB into a managed service
Current MongoDB instance has a number of managed service alternatives - Atlas (Mongo’s official service) or DocumentDB (AWS)
MySql could be moved to Aurora or RDS
Neo4J also has a managed service in AWS called Neptune
Migration considerations - we need an investigation in the options but known option is DMS
Services/Applications
Load balancing should be used where possible (unlike the current implementation where it only routes to a specific VM depending on the source address)
Application LB is preferred where possible as it provides url based routing
Route 53 for DNS and WAF should be used (Route 53 already in place)
For applications that are already dockerised, we should prefer orchestration services (ECS, EKS) where possible as it saves time and money
Other options include Beanstalk / Fargate / Serverless (Lambda) for basic java applications, light/fast containerised apps or ETL type jobs
AWS provides and managed elastic service
Cognito seems to be the current choice for authentication
Provides a good migration path for KC
Managed services for build tools were possible, moving away from jenkins will be good, could look into CircleCI, AWS code deploy and Drone (currently trialling GitHub Actions)
Questions/unknowns/concerns
Serverless with java
Startup time for the JVM
version support of the JVM
what other languages can we use / what other languages do the team know?
ECS supports fargate as well as EC2 which could possibly allow for piecemeal migrations
EFS support in fargate?
Other forms of notifications available - could we potentially use SMS for Trainee self service
We’re currently using RabbitMQ, are there any alternatives (SQS + SNS)? is it still valid to stick with it? should we look into Kafka/Data streams?
Should we look at extending ES usage to other parts of the app?
Currently there are a couple of known limitations in GitHub Actions
The wider NHS/Public sector has an Auth service but its currently not far enough in its development, its perhaps worth keeping that in mind while working on Cognito or perhaps even delaying it until the product is ready
Whats next?
So with this laid out, we now have a better idea where we want to be, a number of investigation tasks need to happen (around the build tools, monitoring, serverless, messaging) to gain a little more knowledge on them and to assess whether they are right for us. Designs of the standard networking approach as well as the security considerations that we need to bake in so that environments for each application don't stray too far, making it difficult to manage.
Summary
All in all, this meeting has made it clear that we would like to use as much of what makes using a cloud provider good, which is to use as much of the managed services as possible - saving on costs where possible, providing features as automatic scalability and the saving limited time of the DevOps engineers.