Trainees must now manually link their FormR submissions to specific programme memberships, and indicate whether the form is submitted as a new starter or for ARCP. There are fairly large numbers of forms that were submitted before this feature was included that consequently lack this linkage information. This makes reporting more complex, and hinders attempts to improve the FormR submission process and compliance.
This document comprises the outcomes of an investigation into the automation of the form R linking, retrospectively and in the future.
Outputs of investigation
Rules to link Form R to Programme with confidence measure
Rough numbers in each confidence level of the above
Rules to link Form R to ARCP vs New Starter with confidence measure
Rough numbers in each confidence level of the above
Approach
FormR and TIS data as of early Oct 2024 were retrieved and used to evaluate potential methods of linking FormRs and programme memberships. Both submitted and draft forms were assessed.
Given that a subset of the these FormRs had been manually linked by trainees to particular programme memberships, these manual linkages were used to assess the potential accuracy of automated linkages. If the programme membership automatically selected for a given FormR was the same as that chosen by the trainee, then it was considered a ‘correct’ linkage; if it differed then it was considered erroneous. Two factors made this assumption less clear-cut:
Trainees manual form linkages are not infallible, and in some cases appear to be incorrect
The subset of manually linked forms do not comprise a particularly large number, are limited to a specific recent time period, and are not necessarily representative of ‘business as usual’ (in particular, they do not cover the ARCP-intensive period of Apr-May).
Nevertheless, having manually linked forms allowed us to take some basic rules for automated linking and to refine these with respect to edge-cases and other scenarios not initially considered.
An iterative ‘divide-and-conquer' approach was used to establish automated linkages. This permitted more clear-cut scenarios to be dealt with before handling more complex situations, without having to deal with every eventuality in a single step. A disadvantage of this method is that no overarching rule is available, and that the linkages established are highly dependent on the sequencing of the steps involved: later steps are implicitly dependent on earlier steps having already dealt with forms that the rules of the later steps do not address (or would link incorrectly).
Pre-processing and standardisation
FormR local offices are captured manually by trainees, and as such my vary from the ‘canonical’ name. In addition, official local office names have recently been updated as part of the standardisation to merge Health Education England into the NHS proper.
Details of the rules for standardising FormR local offices are item A.
To make TIS programmes and specialties more consistent, TIS programme names were standardised and assigned a parent as defined in items B1 and B2.
Linking rules
A. Trainee has only a single programme membership
Rule:
If a trainee has only one programme membership, then any forms they have submitted must relate to that programme membership. SQL.
Matches:
FormR PartA: 33271 (27.9% of test dataset); 798 manually linked forms
FormR PartB: 40381 (28.6% of test dataset); 1310 manually linked forms
Errors:
7 trainee-linked forms referred to programme memberships that no longer exist, and were mis-linked to new programme memberships. As such, any rule to link forms in this way must allow for the possibility of programme memberships being deleted after having being linked to a form. This may best be handled by automatically delinking any form referring to a programme membership that is being deleted.
B. Unique current programme membership with matched local office and programme details
Rule:
Trainee has only one current programme membership starting more than 6 months ago, finishing more than 6 months in the future, with matching local office and programme details. SQL.
Matches:
FormR PartA: 339814 (33.4% of test dataset); 1096 manually linked forms
FormR PartB: 50379 (35.6% of test dataset); 1605 manually linked forms
Errors:
3 trainee-linked forms were mismatched:
Form PartB forms:
The relevant details from TIS:
In both instances, the trainee linked to a PM that had finished some time before the submission date of the FormR (first form submitted 28 Aug 2024, PM finished 3 Oct 2023; second form submitted 27 Sep 2024, PM finished 6 Feb 2024), while having an ongoing PM with the same LO and programme curriculum specialty. It is not clear if these were trainee errors or not.
FormR PartA forms:
The relevant details from TIS:
Similarly, the form was submitted on 28 Aug 2024 for PM that finished on 3 Oct 2023, while having an ongoing PM with the same LO and programme curriculum specialty.
Add Comment