ESR Streamlining

Streamlining – ESR/TIS interface – Briefing Note
Introduction
The purpose of this note is to provide a briefing on issues arising from the interfacing/data exchange between the NHS's National Electronic Staff Record system (ESR) and Health Education England's Trainee Information System (TIS). To do this the paper will outline the background and history of work to date and identify issues relating to this work. The paper seeks advice and direction in relation to the significant risks associated with the identified issues.
Background
ESR is a payroll and HR system used by Trusts throughout the NHS. 470 organisations in England and Wales are currently using the system. In the context of this paper it is used to get junior doctors on payroll and manage training and development. The primary functions of the ESR Interface to TIS are;

• Reconcile the Inbound Positions from ESR against Posts in TIS to make sure that both systems match.
• Send Placements from TIS to ESR for future posts, this includes sending new starters who have never been placed before.
• Receive Trainee profile updates from ESR to TIS,
• Send Placements updates from TIS to ESR where changes occur after they've been sent to ESR.

The ESR solution is based on Oracle's e-Business suite R12 and has been in place since 2006 providing almost 12 years of great benefits to the NHS. It is currently integrated with Wales Deanery Intrepid solution delivered by Hicom, the Empower solution delivered by Northgate and the Trainee Information System (TIS) solution delivered by Health Education England for the purpose of transferring junior doctors' placement information.
Whilst the benefits derived from the integration have been extensive to the NHS Employer organisations, it is limited to what it can provide when compared to modern interfaces in many respects. We recognise that the historic framework contracts ESR has had are ineffable and makes it difficult to react into current practice.
TIS Programme and ESR work to date
HEE had used the Intrepid system for approximately 20 years which is supplied by Hicom. Each Local office (13 of them grouped into 4 regions), used Intrepid individually and built up separate implementation and databases.
The TIS programme was put together to:

• develop an in-house team within HEE to build a replacement for Intrepid
• develop a national system providing consistency across local offices
• give HEE the capability to replace the modules
• provide the ability to continue to develop the product in a use-centred approach and meeting the business needs.

TIS-ESR One-directional flow
As part of the TIS solution, a replacement for the Intrepid-ESR interface was developed to carry on the automatic transfer of junior doctor placement between TIS and ESR to avoid a downgrade in the service when switching to TIS. The Minimum Viable Product (MVP) included exchanging trainee placements, changes to their placements by way of notifications to ESR and data quality reports generated where gaps in the data are identified. This is also known as the one-directional flow. The TIS-ESR interface involves exchanging of files generated by either parties in a sequential order every evening via scheduled jobs similar to the Intrepid-ESR interface. The specification provided to TIS to integrate with ESR was largely one that was developed between Hicom, ESR and IBM.
TIS-ESR Bi-directional flow
There is a future requirement to build a bi-directional link to exchange updates of trainees personal details to feed into TIS. In this arrangement; whilst TIS masters training information, ESR would be deemed as master for the trainees' details once they start employment.
The bi-directional flow that existed between ESR-Intrepid initially had HEE Local Offices faced with several data quality issues regarding their trainees' details updated inaccurately on Intrepid. As a result, most of the Local offices switched off the bi-directional flow or accepting a bare minimum set of data to be updated with on Intrepid.
TIS has started scoping the dataset that needs to be included in a bi-directional flow for TIS, if one is to be built. It is currently being analysed with the various systems in use by HEE where trainees have self-service access to. Given the numerous issues we have encountered with the current method; it is not deemed sustainable to build onto it a bi-directional flow using a similar approach to Intrepid.
ESR Interfaces
There are basically 3 interfaces on ESR through which the above are received or sent:

• Deanery Update Interface/Deanery Reconciliation and Medical Trainee Outbound Interface
• Deanery Inbound Recruitment Interface
• Generic Inbound Notification Interface

The diagram below illustrates these interfaces points and associated data flows.


Key Issues
The use of the phrase "system interfaces" is not technically correct. Modern systems exhibit clear technical interfaces which allow an on-demand connection and provide a documented interface specification through APIs.
Most modern applications expose APIs that can be used by clients to interact with the application. These are broadly understood to be accessible and adhere to standards such as REST or gRPC. They are documented, versioned and designed with platform independence and service evolution in mind giving them the ability to evolve independently from client applications
The current "interface" between TIS and ESR is essentially a file transfer and import into the various data storage arrangements. It is this approach which generates issues in two key areas:
Data transfer approach – The current method uses FTPS on N3/HSCN which has a number of inherent issues:

• Security

FTPS uses multiple port numbers. The initial port number (default of 21) is used for authentication and passing any commands. However, every time a file transfer request (e.g. get or put) or directory listing request is made, another port number needs to be opened which means therefore a range of ports in your firewalls have to be opened to allow for FTPS connections, which can put your network at risk and weaken your cybersecurity defences.

• N3

The Health and Social Care Network (HSCN) Health and Social Care Network - https://digital.nhs.uk/services/health-and-social-care-network is a new data network for health and care organisations which replaced N3. It provides the underlying network arrangements to help integrate and transform health and social care services by enabling them to access and share information more reliably, flexibly and efficiently, while at the same time reducing costs and complexity, standardising networks, enabling service sharing, and extending the parameters of collaborative working.
Implicit in that statement is acceptance that N3 is no longer fit for purpose and certainly not compatible with the vision and aspirations enshrined in the Five Year Forward View and National Information Board's Personalised Health and Care 2020. Personalised Health and Care 2020 - https://www.digitalhealth.net/includes/images/news0254/PDF/0172_NHS_England_NIB_Report_WITH_ADDITIONAL_MATERIAL_S8.pdf
Data management within receiving systems - the current approach requires the import of data from a delimitated file. This is essentially a text file with a record per line with the end of record indicated by a marker (a comma or pipe symbol). This traditional approach contains many points for potential error – markers missing, file corruption or malformed document.
The current approach is essentially a "bulk" import and export of data. This means that data are processed on a daily/weekly basis, with an inherent "lag" in the systems (the time taken to export and import data) of 24 hours plus, this carries a clear number of usage risks.
Specific issues
In addition to the technical limitations of the approach and also linked, the below are more specific issues we have encountered on the interface:

• encountered issues where markers are either missing or use of other special characters in the file rendering the whole of it invalid and therefore not processed
• National Post Numbers not matching the format that TIS expects and use of special characters breaking the processing of the file
• Limitation of ESR to handle retrospective reference data that would have been correct at the time of recording on TIS but not acceptable to ESR as they only expect a specific version of the reference data.
• Limitation of ESR to handle trainee placements information beyond 3 months' future time projected hire date
• The exchange of notifications, i.e. changes to placements subsequent to them being transferred to ESR being inadequate. The changes to placements are not exchanged in real time but relies on changes audited for the day to be built and sent in a file at the end of the day. The changes are limited to only 5 different types of notifications that ESR can infer. E.g. a medical rotation is due to take place against a post, an applicant has withdrawn/replaced from a post etc.
• If for some reason, there is an outage, it involves a lot of manual work by an experienced individual to process the backlog of files to bring the interface back up and sync the data
• The time at which the files are currently exchanged are out of normal working hours, which means if there are issues, they are not noticed until the next day
• Time restrictions for files being sent at specific times of the day rather than processed in a stream means that failures at certain parts of the day have cascading effects.

Associated risks – Data Quality
The implications of the above specific issues are detrimental to data quality in many ways;

• A file being unprocessed means trainees' placements are not transferred to ESR on time to follow the code of practice and have a direct impact on trainees' payroll due to delays in records update
• National Post Numbers mismatch, has a similar implication as the above unless noticed on time
• Changes to placements not transferred in real time but at the end of the day, means out of date information are sent to ESR, again with a similar implication to the above
• Inferring reference data on trainees supplied information rather than being able to retrospectively maintain them by ESR, creates inaccuracies in reporting and would create inaccuracies if a future bi-directional flow is to be built
• Over-reliance on a 3 months' future start date rule, means ESR does not have the full picture of oncoming trainees for the posts beyond 3 months' time
• Data inaccuracies – manual interfering and fixing backlog of unprocessed files are prone to human error and therefore impacting on data quality
• The time taken to bring the data back in synch in an event of an outage (e.g. due to network issues), means that there could be further inaccuracies or data loss created by missing applicants which were due to be transferred during that period
• In addition to the delays caused by afore-mentioned issues, there is also an added delay by the ESR queue processing of files, normally 24 hours' plus

Managing risks and issues
As detailed above the bi-directional "interface" has a number of significant challenges.
Due to the age and contractual arrangement for ESR approaches to date have been forced down the technical route detailed above, this puts particular pressure on the TIS development team as there is more flexibility in the technology in use on that side.
Form the TIS perspective our only mitigation route at the moment will be for the team to increase the levels of data validation (filtering/checking) and to increase the level of data monitoring/audit. Our concern is that, the underlying issues will remain and data error can be expected to occur with consequent reputational and operational damage.
Conclusion
There are significant risks and issues in this area of work that are detailed above.
The mitigation of these risks from a TIS perspective will require significant person resource investment by the TIS team.
The team require guidance as to how much of this work should be conducted in the knowledge that the process will be error-prone.
As the team are unaware of the future development program for ESR the above question is one that the TIS team cannot address.


R.Hill, A.Ransoo, C.Mills – 7/12/18