Application Architecture - Service summaries

Owned by Reuben Noot (Deactivated)
Last updated: 22 Apr 2020 by Joseph (Pepe) KellyVersion comment

Introduction

Build out a high-level description of each service to complement the Application Architecture diagram

Matt: I have created the most current IA/Site map around TIS and it's current structure/features. The initial mapping was created with XMind with the editable documents available here: TIS Admin IA (Information Architecture) and Site Map

TIS Header features

TIS People

TIS Posts

TIS Programmes

TIS Revalidation

TIS Assessments

TIS Admin


Revalidation (UI + Service)

Summary

A doctor's licence to practice with the GMC must be renewed every 5 years, this includes trainee doctors with HEE who receive their full GMC licence after their first (Foundation) year of training. This renewal is known as Revalidation and is achieved by HEE submitting a recommendation to the GMC for those trainees who are nearing renewal (under notice). The GMC treats HEE as 13 discrete entities or Designated Bodies, each with a single Responsible Officer (RO), the individual responsible for the trainee's recommendation. In practice the RO delegates all or most of these to HEE Admins and sometimes reviews before submission. The Revalidation application within TIS enables HEE to review the data on Trainee Doctors due to be Revalidated, make a recommendation, review these and then submit to the GMC.

Revalidation is made up of an Angular 1.x UI app and a springboot application presenting a RESTful API to support the Revalidation UI. The Revalidation service uses Elastic Search indexes to hold a daily refreshed cache of trainee data and a MySQL database for the data that is persisted. The Elastic Search data is refreshed each evening by a combination of Application Architecture - Service summaries#GMC Sync and the Application Architecture - Service summaries#Revalidation ETL

Where

The Revalidation Angular 1.x UI App and the Sprintboot microservice can both be found in the same repository: https://github.com/Health-Education-England/TIS-REVALIDATION

What it does

  • Revalidation allows HEE Admins to review lists of HEE's trainee doctors, constrained to only trainees within the Designated Body the Admin has access to (retrieved from the Profile Service)
  • The lists of Trainees are presented as tabs - each one shows trainees within one (or more) status within the Revalidation workflow, these are: All Doctors | Under Notice | Review | Ready to Submit | Submitted
  • The data to populate these lists in the UI is retrieved from the Revalidation service - which holds it either in read only cache in the ES indexes, or the persistent data within the service's MySQL database
  • Selecting a Trainee takes the admin through to a series of screens displaying further trainee detail held by HEE
  • The Revalidation service provides this data from its Elastic Search cache (updated nightly), it includes programmes/revalidation history/assessments/placements/contact details
  • Once the Admin has reviewed a trainee's data they can make a recommendation to the GMC (whether the Trainee should have their licence to practice extended for another 5 years)
  • The recommendation is a simple form which gathers the required fields and saves this as a 'revalidation episode' to the Revalidation service (stored in the MySQL database)
  • The recommendation can be placed in a review status or directly submitted to the GMC (by invoking the TryRecommendation method on the /wiki/spaces/TISDEV/pages/16613389 - a SOAP based  API provided by the GMC.)
  • The UI allows the submission of multiple recommendations using one action in the UI, the service makes multiple called to TryRecommendation - users particularly like this time saving aspect compared to their alternative processes.
  • The submission to the GMC is an asynchronous process within the GMC, so Revalidation calls the CheckRecommendationStatus on the /wiki/spaces/TISDEV/pages/16613389 at regular intervals for each submitted Revalidation until its status is confirmed by the GMC - this is displayed in the 'submitted' tab
  • As the submission is accepted Revalidation takes a 'snapshot' of the data for that trainee and persists it as XML through the Revalidation service into the MySQL database as a permanent audit of the trainee record at the time the recommendation was made.
  • Aside from the external calls outlined above, the Revalidation service also authenticates with Keycloak to obtain a JWT token and authorises its permission levels with the Profile service
  • The Revalidation UI uses the Reference service, the Revalidation service doesn't make external calls to Reference though.
  • Note: if Revalidation is unavailable, Doctors can be revalidated manually via the GMC web user interface, GMC Connect, https://hee-tis.atlassian.net/wiki/spaces/TISDEV/pages/13402263/GMC+Connect, however the data in Revalidation then becomes out of sync and needs manually updating

GMC Sync

Summary

A springboot application which runs nightly to retrieve a basic Doctor dataset from the GMC to refresh the Revalidation service.

Where and When

The application has its own github repository: https://github.com/Health-Education-England/TIS-GMC-SYNC

The application is run from a Jenkins cron (00.05 UTC Mon-Fri) - here : https://build.tis.nhs.uk/jenkins/view/Prod/job/gmc-sync-prod/

What is does

  • GMC Sync first truncates the Revalidation Elastic Search (ES) indexes (removing the visibility of all Revalidation data from the Revalidation UI)
  • It then executes the following logic for each Designated Body in turn:
    • Calls the GetDoctorsForDB method on the GMC API to retrieve basic Doctor records. (/wiki/spaces/TISDEV/pages/16613389 is a SOAP API provided by the GMC.)
    • The basic record includes GMC ID, Firstname, Surname, various GMC specific dates and status (including Sanction - a flag to highlight a doctor's licence has a sanction against it.) (we ignore the GMC's Firstname and Surname in preference to the TIS one)
    • It uses the Profile service to check whether the GMC ID is already known and if not creates a TISId in the Profile service's Traineeprofile table.
    • It then writes the basic record for all doctors to the Revalidation ES index and in parallel to the Concerns ES index (Sunil Rochani (Unlicensed) to double check this element)

For pre-live environments TIS can't use live GMC data, so a GMC Sync is mocked with a Springboot application called GMC Connect, serving up SOAP responses for pre-live environments (https://github.com/Health-Education-England/TIS-GMC-CONNECT) Note: this is the same name as the GMC provided user interface, also called GMC Connect, which could cause confusion

Revalidation ETL

Summary

A springboot application which runs nightly to enrich the Revalidation ES indexes recreated by GMC Sync, with data from the TCS and Assessments services

Where and When

The application has its own github repository: https://github.com/Health-Education-England/TIS-INTREPID-REVAL-ETL

The application is run from a Jenkins cron (01.00 UTC 7 days a week) - here : https://build.tis.nhs.uk/jenkins/view/Prod/job/intrepid-reval-etl-all-prod/

What it does

  • Revalidation ETL works through each Trainee provided by the GMC through GMC Sync, by their GMC ID.
  • It uses the GMC ID stored in the Trainee ES index to identify the Person record (Person.Id) in TCS
  • It then retrieves the Person, and associated Contact, Programme, Placement and Assessment data from the TCS and Assessment services and directly updates the appropriate Revalidation ES indexes
  • It also writes a subset of this data to the Connection Discrepancies ES index

Note: Revalidation was developed before TCS and wasn't refactored as part of TCS, which is why it identifies Trainees by GMC ID and a Trainee.TISid in the Profile service but isn't more cleanly integrated into TCS.Person. This is a clear area to refactor as apart of a wider Revalidation service rework.

Concerns - (UI + Service)

Summary

To support the Revalidation service HEE can track concerns about trainee doctors. A trainee can have multiple concerns records, each record consists of metadata such as incident date, type, employer, site etc, a rich text description and the ability to upload files.

Concerns is made up of an Angular 1.x UI app and a springboot application presenting a RESTful API to support the Concerns UI. The Concerns service uses Elastic Search indexes to hold a daily refreshed cache of trainee data and a MySQL database for the data that is persisted. The Elastic Search data is refreshed each evening by Application Architecture - Service summaries#GMC Sync and the Application Architecture - Service summaries#Revalidation ETL. - see question below. Each added/amended concern once saved to the MySQL db by the service is also refreshed into the ES Index.

Q. Sunil Rochani (Unlicensed) - I'm assuming the Concerns list gets its data from the Concerns ES index via the concerns service - You said that the concerns ES index is wiped and rebuild each evening with the base record coming from GMC Sync? If so does something caches the MySQL concerns data into ES?

Where

The Concerns Angular 1.x UI App and the Springboot microservice can both be found in the same repository: https://github.com/Health-Education-England/TIS-CONCERNS

What it does

  • The UI is similar to revalidation in that it presents a list of Trainees that have a concern logged against them, constrained to only trainees within the Designated Body the Admin has access to (retrieved from the Profile Service)
  • Note this is a list of concerns rather than Trainees. It obtains this data from the Concerns service which caches this nightly in the Elastic Search index
  • The UI allows a very basic Trainee search in order to create a new concern log, using the cached Trainees in the Concerns service
  • The UI provides a form create/edit a concern record. The UI uses endpoints on the Reference service to allow the user to search and select a Site, Trust and Grade for the Trainee.
  • Aside from the external calls outlined above, the Concerns service also authenticates with Keycloak to obtain a JWT token and authorises its permission levels with the Profile service
  • The Concerns UI uses the Reference service, the Concerns service doesn't make external calls to Reference though.

Notifications - (UI + Service)

Summary

The Notifications service was created to provide HEE admins with push notifications regarding events relevant to them. Initially it included release notes for each new TIS release, but was designed with user specific notifications in mind (e.g. review Trainee X's Placement now due). With the inclusion of Microsoft Teams to all HEE Users, TIS has an alternate notification platform, now used for community wide publication of TIS Release Notes. On this basis TIS stopped actively using the notification service and alerts were disabled in the Angular 5 Admin UI.

Notifications is made up of an Angular 1.x UI app and a springboot application presenting a RESTful API to support the Notifications UI.

Where

The Notifications Angular 1.x UI App and the Springboot microservice can both be found in the same repository: https://github.com/Health-Education-England/TIS-NOTIFICATIONS

What it does

  • An 'new notifications' icon in the Angular 1.x UI App menu bar informs the user of unread notifications, clicking provides a 'slide in' list of unread notifications.
  • Clicking a notification or the Notifications tab, the user is presented with a simple 'inbox' of notifications - these can be marked as read and deleted
  • Users with TIS Admin permissions can create notifications via a simple form
  • The Notifications UI retrieves and stores all its data within the Notifications service
  • Aside from the external calls outlined above, the Notifications service also authenticates with Keycloak to obtain a JWT token and authorises its permission levels with the Profile service


Connection Discrepancies (Service and UI (within Admin UI))

Summary

As outlined in Application Architecture - Service summaries#Revalidation - The GMC treats HEE as 13 discrete entities or Designated Bodies and requires every trainee with a GMC licence to be assigned to one of these bodies. Whilst the trainees themselves are expected to self-manage their relationship with the GMC and inform them when they move locations to a new designated body, in practice HEE also needs to update the GMC's records - this is the Connection Discrepancies function.

Connection Discrepancies is a springboot application presenting a RESTful API to support the user interface which is found within the Admin UI

Where

The Connection Discrepancies Springboot microservice can both be found in the repository: https://github.com/Health-Education-England/TIS-CONNECTION-DISCREPANCIES. The UI is part of the Admin UI at: https://github.com/Health-Education-England/TIS-ADMINS-UI

What it does

  • The UI within Admin UI provides two lists, 'Create Connection' and 'Remove Connection' which provides constrained lists of trainees within the Designated Body the Admin has access to (retrieved from the Profile Service)
  • The data to populate these lists is retrieved from the Connection Discrepancies service. 
  • Create Connection highlights Trainees within the HEE Admin's local office(s) which are in a different GMC Designated Body. It allows the multi-select of these trainees and invokes the TryAddDoctor method on the /wiki/spaces/TISDEV/pages/16613389 (a SOAP based  API provided by the GMC) to re-associate the trainee with its correct Designated Body within the GMC.
  • Remove Connection highlights Trainees within the Designated Body(s) for the HEE Admin who are not on a programme within the corresponding Local Office. It allows the multi-select of these trainees and invokes the TryAddDoctor method on the /wiki/spaces/TISDEV/pages/16613389 (a SOAP based  API provided by the GMC) to re-associate the trainee with its correct Designated Body within the GMC or the TryRemoveDoctor to remove the association if one isn't appropriate << check this with Sunil Rochani (Unlicensed) - I'm sure this isn't quite correct>>

More details can be found on the https://hee-tis.atlassian.net/wiki/spaces/TISDEV/pages/49217537/Manage+connections

Admin UI

Summary

The Admin UI is the user interface that HEE admins use to interact with TIS. It allow them to manage the key business entities within TIS, People, Programmes, Posts, Placements and Assessments (4xP+A). The Admin UI constrains what a user can view or edit based on their Role, which groups up relatively granular set of Permissions. E.g. the standard 'HEE Admin' Role has read/write access to 4xP+A but can't view or edit sensitive data fields.

It is an Angular 5 application utilising the Bootstrap libraries.

What it does

  • Provides a top menu (L1 menu) with access to People, Posts, Programmes, Concerns, Assessments and Admin sections (based on the users Role)
  • Within each L1 menu dropdown, L2 items provide links to pages within each L1 section.
  • Typically these consist of a 'Search' (effectively a filterable, searchable List page of records of the object in question, e.g. People) and a CRUD (Create, Read, Update, Delete) - effectively a form allowing the CRUD of an individual record of an object - e.g. a Person record.
  • The List and CRUD pages will typically invoke the TCS or Assessment microservices to read, create and update data and use the Reference microservice to populate individual fields within the pages that display / search reference data
  • For the more complex People and Assessment objects, a Level 3 (L3) navigation is provided - e.g. tabs within an individual Person record provide access to different data or objects directly related to that Person record (e.g. their contact details, or a list of their Programme Memberships, Placements or Assessments.) Where a list is provided within a tab, the records (e.g. Placements) are editable via an inline expanding form rather than a navigation to a full page.
  • The Admin UI implements view/edit constraints for 5 Roles initially, the base HEE Admin, extensions to provide access to both Sensitive Data and Revalidation/Concerns, a full TIS Admin with access to reference data, and a Trust Admin with data access constrained to Trainees and Posts related to their Trust. Full details can be found on the Admin User Management (roles and permissions) page.
  • The Admin UI was initially built using a JHipster autogeneration model and has been incrementally improved. Some elements such as the Admin sections List and CRUD pages are early versions and lacking functionality, whereas the more widely used objects such as Person/Post/Programmes have a richer interface. A full rewrite of the List functionality is currently  in development initially for People which removes the last remnants of the JHipster 'scaffold'.
  • The Admin UI also provides the user interface to the Document Management feature within TCS, allowing documents to be stored against a Person record with limited editable metadata.
  • The Admin UI also provides the user interface to the Generic Upload microservice, more details can be found Generic Upload section
  • For a user to access the Admin UI, they must authenticate via Keycloak and the Admin UI establishes their Roles and Permissions via the Profile service

TIS Core Services (TCS)

Summary

The TIS Core Services (TCS) microservice provides the endpoints to allow the core TIS objects to be managed, People, Programmes, Posts, Placements (4xP) with supporting objects Curricula, Specialty and Rotations. It also contains the endpoints to support Document Management against People.

TCS is a springboot application presenting a RESTful API to support the user interface which is found within the Admin UI

Where

The TCS microservice can be found in the repository: https://github.com/Health-Education-England/TIS-TCS. 

What it does

  • Provides endpoints for the List/CRUD on People, Programmes, Posts, Placements (4xP) as core objects.
  • Note that People in particular is subdivided into a primary and multiple secondary database tables and corresponding endpoints, so People has a primary Person table but 'secondary tables' of ContactDetails, PersonalDetails, GMCDetails, GDCDetails, RightToWork and Qualification - all keyed off the same Person ID, i.e. 1:1 relationship
  • The relationships between the 4P Core objects are relatively complex and will be more fully documented on Entity Relationship Diagrams, the endpoints also support the creation and maintenance of these relationships
  • An example would be Placements can have many Supervisors (a subset of Person records), of two types, Clinical and Educational. These have a separate PlacementSupervisor table outlined Placement Supervisors - Solution Design.
  • Also provides endpoints for the List/CRUD on supporting objects, Curricula, Specialty and Rotations
  • Supports the relationships between Specialties & Sub-Specialties as outlined on this page - Note: unfinished relationships are highlighted here as Jira tickets still to do
  • Supports the relationships between Rotations and Programs/Posts/People, as outlined here: Programmes - Rotations - Solution Design, Field Validation & Scenarios
  • Provides the endpoints to support Document Management - the storage /metadata management of externally created documents against Person records (initially). 
  • As other services, the TCS service also authenticates with Keycloak to obtain a JWT token and authorises its permission levels with the Profile service

Assessments

Summary

The Assessments microservice provides the endpoints to allow Assessments to be managed in TIS. The UI subdivides Assessments into the event, pre-event detail, post-event detail and revalidation through Level 3 tabs. It is worth noting that some data items within Assessments are a static copy of the data item from elsewhere in TIS at the time the assessment was undertaken.

The microservice and the database tables subdivide into Assessments, 1:1 extension tables AssessmentDetail, AssessmentOutcome and Revalidation and a 1:many relationship AssessmentOutcomeReasons. Reference data for Outcome and Reason codes are also found within the service.

Assessments is a springboot application presenting a RESTful API to support the user interface which is found within the Admin UI

Where

The Assessments microservice can be found in the repository: https://github.com/Health-Education-England/TIS-ASSESSMENTS

What it does

  • Provides endpoints for the List/CRUD on Assessments - Assessments Swagger
  • The Assessment database tables and corresponding endpoints are subdivided into a primary Assessments table/endpoints and secondary AssessmentDetail, AssessmentOutcome and Revalidation tables/endpoints - all 1:1 tables keyed off the same Assessment ID. 
  • The service also supports multiple Reasons for an AssessmentOutcome, via the AssessmentOutcomeReason table/endpoint.
  • Outcome codes and Reason codes are also held with the Assessment service with discrete tables and endpoints


Generic Upload

Summary

Generic Upload (also known as Bulk Upload to users) allows HEE Admins to upload Excel spreadsheets contain multiple People and Placement records, (initially), for these to be validated, imported into TIS if acceptable, or returned to the user in a downloadable spreadsheet if not (on a row by row basis).

Generic Upload is a springboot application presenting a RESTful API to support the user interface which is found within the Admin UI.

Where

The Generic Upload springboot microservice can both be found in the repository: https://github.com/Health-Education-England/TIS-GENERIC-UPLOAD. The UI is part of the Admin UI at: https://github.com/Health-Education-England/TIS-ADMINS-UI

What it does

  • The UI presents a filterable list of all previously uploaded import files - this is global across all TIS users with access to Bulk Uploads (currently a hidden URL, with all HEE TIS Admin role members having access)
  • The Generic Upload service provides the endpoints behind this filterable, paginated list.
  • The UI allows HEE Admins to upload new import files, the Generic Upload service places these in Azure storage and the 'genericupload' database gets updated.
  • The Generic Upload service runs a background task which picks up the import files in the order they were uploaded and executes the following steps (Jayanta Saha - could you explain what the 'background task' is?)
    • It validates the input to some basic rules, specific to each import type. Errors are flagged within the spreadsheet and the row disabled from being inserted.
    • The service then calls the TCS service to create or update the Person or Placement record within TIS, for each valid row in the spreadsheet.
    • Any rows which fail to create/update are highlighted in the spreadsheet
    • The successfully imported rows are removed from the spreadsheet
    • The spreadsheet is made available through the UI for download (and correction before re-upload if required)
  • Ashley Ransoo - could you check this for functional accuracy please?
  • Jayanta Saha - could you check this for technical accuracy and/or omissions please?

TIS-NDW ETL

Summary

A spring-batch application which runs nightly to extract data from TIS, denormalise to the National Data Warehouse (NDW) schema, and write to an interim database for the NDW to import from.

Where and When

The application has its own github repository: https://github.com/Health-Education-England/TIS-NDW-ETL

The application is run from a Jenkins cron (00.20 UTC 7 days a week) - here : https://build.tis.nhs.uk/jenkins/view/Prod/job/tis-ndw-etl-cron-prod/

What it does

  • The application uses spring-batch framework to work through a series of extract and mapping tasks in discrete chunks.
  • It works through each table in the destination database (TIS-Interim, a Microsoft SQL Server DB) in turn and batches the ETL into each table
  • When starting on the next table it firsts truncates the table contents in the TIS-Interim
  • It extracts data from TIS via direct database calls to the MySQL service databases (as opposed to going through the REST service layer - in order to provide adequate performance)
  • It writes data to TIS-Interim using Microsoft SQLServerBulkCopy for MS SQL Server databases and JDBC Batch for other databases
  • It accesses TCS, Assessments and Reference databases to assemble the required data
  • The mapping is contained in discrete SQL mapping files in https://github.com/Health-Education-England/TIS-NDW-ETL/tree/master/src/main/resources/queries which the application uses to extract data from TIS and normalise it to the single table in the NDW schema
  • Note: The destination schema is held in the repository for each destination table but these need to be dropped and recreated manually if they are changed, it's not flyway enabled.
  • The service through spring-batch has instrumentation on its progress which it writes to the TIS-Interim database - tables Batch Job Execution and Batch Step Execution can be used to review or debug the application progress through tables and batches.

ESR (Service and ETL)

Summary

  • TIS-ESR interface acts as a bridge between ESR and TIS to exchange junior doctors and placements data between the systems. The data is exchanged between the systems using pre-defined fields and format as outlined in the ESR specification.
  • TIS-ESR interface comprises of two applications.
    • ESR ETL: A spring-boot application which runs on a daily basis multiple times with different load types. 
    • ESR service: A spring-boot application running as a service and is used to persist and retrieve ESR data.

 

Where and When

The applications have their own github repositores.

The ESR-ETL application is run from a Jenkins cron: 

What it does

  • ESR-ETL receives and sends different files from and to ESR. The details of files and data sets are outlined here: ESR - Datasets (Historical)
  • It receives RMT files in changed (RMC) or full (RMF) mode. It processes and stores POR and POS records from the files.
    • The POR and POS records are persisted in ESR database (mySQL) using ESR service.
  • Each of the above files are received per deanery/local office and all the files produced are as well per deanery.
  • For each of the deanery posts ESR requests corresponding APP file containing applicant records in placement for the post is generated. The following restriction applies for the selection:
    • Only the future applicant placements are selected.
    • The selected future placements should have a start date between 2 days from that day to 3 months. 
    • It uses TCS service to fetch the post, placement and trainee details.
    • The applicant records are stored in ESR database using ESR service.
  • A notification file (DNF - full notifications and DNC - changed notifications) is sent per deanery on a daily basis.
    • The notification records tracks all the events happening in TIS and creates different notification types outlined in the ESR spec. This is a way of updating ESR about the changes happening in TIS.
    • The TIS notification events are tracked and stored in TCS database in the EsrNotification table using the TCS service. This records only the TIS contents.
    • All of the final notification records generated are stored in ESR database using the ESR service.
  • An acknowledgment is generated for each RMT file processed and similarly an acknowledgment file (APC) is received from ESR for each APP file sent.
    • These APC records are stored in the ESR database using the ESR service.
  • The ESR service and ESR-ETL use profile/keycloak for user authentication and have an ESR specific user configured for the same.

SYNC (Service)

Summary

  • TIS-SYNC is a container for background and long-running 'batch' jobs that synchronise data and update cached values.
  • It currently is one package with a dependency of TCS persistence and reference client.
  • It currently comprises 5 separate jobs (listed in "What it does") which assume they are run in a particular order.
  • The jobs are currently duplicated in TCS but are scheduled to run a minute earlier by default

Where and When

SCM:

Deployment:

  • Currently deployed on

The 5 jobs are triggered by cron schedule on methods in a Java class for each job.  The schedule can be set in the container by defining the cron as an environment variable or startup parameter.

  • The default schedules for the cron jobs can be found here: ETL Timings

What it does

There are 7 Jobs:

  • PersonOwnerJob: This triggers a stored procedure that updates the local office for Person records.
  • PersonPlacementEmployingBodyTrustJob: After running this job, tcs.PersonTrust table will be truncated and resynchronised with populated EmployingBody Trust data.
  • PersonPlacementTrainingBodyTrustJob: Will also synchronise populated trainingBody Trust data into tcs.PersonTrust table.
  • PersonRecordStatusJob: Updates the 'training status' (DB field: status) based on a ProgrammeMembership starting today or ending yesterday.
  • PostEmployingBodyTrustJob: After running this job, tcs.PostTrust table will be truncated and resynchronised with populated EmployingBody Trust data.
  • PostTrainingBodyTrustJob: Will also synchronise populated trainingBody Trust data into tcs.PostTrust table.
  • PersonElasticSearchSyncJob:  Will truncate index of persons in ElasticSearch and then create new index and save documents of populated PersonView for ElasticSearch.

Note:

  • PersonPlacementTrainingBodyTrustJob should be triggered after PersonPlacementEmployingBodyTrustJob.
  • PostTrainingBodyTrustJob should be triggered after PostEmployingBodyTrustJob.



Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213