Content Comparison

...

Rules to link Form R to Programme with confidence measure
Rough numbers in each confidence level of the above
Rules to link Form R to ARCP vs New Starter with confidence measure
Rough numbers in each confidence level of the above

Summary

Using the two-step matching process described in more detail below, 82% of FormRs can be linked with no obvious errors. This leaves ~46 000 FormRs that cannot be linked automatically.

Additional steps to link ~95% of FormRs can be added (leaving ~12 000 unlinked) with error rates estimated to be 0.02% (projected to 50 mis-linked forms).

Certain explicitly linked forms were tagged as dubious and ignored for the purposes of estimating error. These were forms linked to programme memberships that had finished more than 18 months ago, had started more than 18 months ago but were linked in a non-ARCP capacity, where the programme membership no longer exists, where (for PartA) the form start and end dates and specialty match a different programme membership to the one linked to, etc. These attempts to identify and mitigate against user-error would need to be reviewed, as would the set of linking rules applied.

ARCP / non-ARCP metrics have not yet been assessed: given our sample data excludes the peak period for ARCPs, this might need to be more strongly rule-based as opposed to modelled off existing form linkages. While all trainee-linked forms for ARCP were submitted on or after the start of the programme, so were a significant percentage of the non-ARCP forms.

Approach

FormR and TIS data as of early Oct 2024 were retrieved and used to evaluate potential methods of linking FormRs and programme memberships. Both submitted and draft forms were assessed.

...

Pre-processing and standardisation

FormR local offices office names are captured manually by trainees, and as such my may vary from the ‘canonical’ name. In addition, official local office names have recently been updated as part of the standardisation to merge Health Education England into the NHS proper.

...

7 trainee-linked forms referred to programme memberships that no longer exist, and were mis-linked to new programme memberships. As such, any rule to link forms in this way must allow for the possibility of programme memberships being deleted after having being linked to a form. This may best be handled by automatically delinking any form referring to a programme membership that is being deleted.

B.

...

One matching programme membership active within 1 year before and 4 month after submission date

Rule:

Trainee has only one current programme membership in the time-period window of interest (from 12 months before form submission date to 4 months after form submission). SQL.

Matches:

FormR PartA: 62934 (52.8% of test dataset); 1788 manually linked forms

FormR PartB: 76637 (54.2% of test dataset); 2542 manually linked forms

Errors:

None detected.

C. One matching programme membership for submission local office and specialty active within 1 year before and 4 month after submission date

Rule:

Trainee has only one current programme membership starting more than 6 months ago, finishing more than 6 months in the futurewith matched local office and specialty in the time-period window of interest (from 12 months before form submission date to 4 months after form submission), with matching local office and programme details. SQL.

Matches:

FormR PartA: 339814 17156 (3314.4% 6% of test dataset); 1096 629 manually linked forms

FormR PartB: 50379 17886 (3515.6% 3% of test dataset); 1605 659 manually linked forms

Errors:

3 trainee-linked forms were mismatched:
Form PartB forms:

...

The relevant details from TIS:

...

In both instances, the trainee linked to a PM that had finished some time before the submission date of the FormR (first form submitted 28 Aug 2024, PM finished 3 Oct 2023; second form submitted 27 Sep 2024, PM finished 6 Feb 2024), while having an ongoing PM with the same LO and programme curriculum specialty. It is not clear if these were trainee errors or not.

FormR PartA forms:

...