Trainees must now manually link their FormR submissions to specific programme memberships, and indicate whether the form is submitted as a new starter or for ARCP. There are fairly large numbers of forms that were submitted before this feature was included that consequently lack this linkage information. This makes reporting more complex, and hinders attempts to improve the FormR submission process and compliance.
This document comprises the outcomes of an investigation into the automation of the form R linking, retrospectively and in the future.
Outputs of investigation
Rules to link Form R to Programme with confidence measure
Rough numbers in each confidence level of the above
Rules to link Form R to ARCP vs New Starter with confidence measure
Rough numbers in each confidence level of the above
Summary
Using the two-step matching process described in more detail below, 82% of FormRs can be linked with no obvious errors. This leaves ~46 000 FormRs that cannot be linked automatically.
Additional steps to link ~95% of FormRs can be added (leaving ~12 000 unlinked) with error rates estimated to be 0.05% (projected to 115 mis-linked forms).
Approach
FormR and TIS data as of early Oct 2024 were retrieved and used to evaluate potential methods of linking FormRs and programme memberships. Both submitted and draft forms were assessed.
Given that a subset of the these FormRs had been manually linked by trainees to particular programme memberships, these manual linkages were used to assess the potential accuracy of automated linkages. If the programme membership automatically selected for a given FormR was the same as that chosen by the trainee, then it was considered a ‘correct’ linkage; if it differed then it was considered erroneous. Two factors made this assumption less clear-cut:
Trainees manual form linkages are not infallible, and in some cases appear to be incorrect
The subset of manually linked forms do not comprise a particularly large number, are limited to a specific recent time period, and are not necessarily representative of ‘business as usual’ (in particular, they do not cover the ARCP-intensive period of Apr-May).
Nevertheless, having manually linked forms allowed us to take some basic rules for automated linking and to refine these with respect to edge-cases and other scenarios not initially considered.
An iterative ‘divide-and-conquer' approach was used to establish automated linkages. This permitted more clear-cut scenarios to be dealt with before handling more complex situations, without having to deal with every eventuality in a single step. A disadvantage of this method is that no overarching rule is available, and that the linkages established are highly dependent on the sequencing of the steps involved: later steps are implicitly dependent on earlier steps having already dealt with forms that the rules of the later steps do not address (or would link incorrectly).
Pre-processing and standardisation
FormR local office names are captured manually by trainees, and as such may vary from the ‘canonical’ name. In addition, official local office names have recently been updated as part of the standardisation to merge Health Education England into the NHS proper.
Details of the rules for standardising FormR local offices are item A.
To make TIS programmes and specialties more consistent, TIS programme names were standardised and assigned a parent as defined in items B1 and B2.
Linking rules
A. Trainee has only a single programme membership
Rule:
If a trainee has only one programme membership, then any forms they have submitted must relate to that programme membership. SQL.
Matches:
FormR PartA: 33271 (27.9% of test dataset); 798 manually linked forms
FormR PartB: 40381 (28.6% of test dataset); 1310 manually linked forms
Errors:
7 trainee-linked forms referred to programme memberships that no longer exist, and were mis-linked to new programme memberships. As such, any rule to link forms in this way must allow for the possibility of programme memberships being deleted after having being linked to a form. This may best be handled by automatically delinking any form referring to a programme membership that is being deleted.
B. One matching programme membership active within 1 year before and 4 month after submission date
Rule:
Trainee has only one current programme membership in the time-period window of interest (from 12 months before form submission date to 4 months after form submission). SQL.
Matches:
FormR PartA: 62934 (52.8% of test dataset); 1788 manually linked forms
FormR PartB: 76637 (54.2% of test dataset); 2542 manually linked forms
Errors:
None detected.
C. One matching programme membership for submission local office and specialty active within 1 year before and 4 month after submission date
Rule:
Trainee has only one current programme membership with matched local office and specialty in the time-period window of interest (from 12 months before form submission date to 4 months after form submission), with matching local office and programme details. SQL.
Matches:
FormR PartA: 17156 (14.6% of test dataset); 629 manually linked forms
FormR PartB: 17886 (15.3% of test dataset); 659 manually linked forms
Errors:
27, of which 4 may be considered non-trainee errors.
0 Comments