Assessments research

Scope

Assessments development will also include the initial stages of design and development for the Trainee Portal
'Assessments' refers to the progression reviews carried out at least annually, but may be more often, and not to the workplace-based assessments that are managed on the various ePortfolios
The assessments function will support reviews across medicine, dentistry, pharmacy and healthcare sciences

Objectives

To understand from users' perspective:

the assessments process and how it differs between professional groups, localities, training stages and specialties
who does what and the detail of the tasks they perform (what, why, where, when and how)
their experience of the process - what's good and what's not
what do they need from a system to support the process

6 local teams:

East Midlands
Kent, Surrey & Sussex
North East
South West
Thames Valley
Yorkshire and the Humber

(Others to be substituted if no availability with initial list - to total of 6 for Alpha)

Objectives

Methods

Contextual Enquiry

Sitting with an administrator and observing what they do
Discussing observations, issues and ideas as they arise
Duration as long as the task takes but minimise disruption of work
Feedback anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas

Interviews

Structured but informal interviews to talk through processes (e.g. revalidation) and discuss any issues and ideas
May be 1-2-1 or include more than one person working within a team (not more than 2 preferred, max 3)
Usually last about 1 hour
Feedback anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas

Workshops

Group of colleagues – either working in similar roles or representing users from end to end of the process
Work together to create a common view of process pain points and opportunities
Feedback can be attributed to whole workshop and not to individuals

Model research schedule - 1-2 days with local team, depending on numbers, availability and location

Can be flexible depending on availability, but should generally start with an observation session with a validation officer/manager who actually carries out the tasks involved in administering the process. Whether we use 1-2-1 or workshops depends to a certain extent on the numbers of people available/willing to participate, but I'd suggest we stick to 1-2-1 (or 1-2-2) for the first couple of local teams at least, and move on to workshops if it feels like we're getting a consensus that can be played back to bigger groups (and as we develop assets that can be used as workshop stimulus).

What	Who do we need at this session	Notes
Contextual Enquiry (c. 2hrs?)	Validation officer/manager who actually undertakes the tasks required for revalidation (RV1-11) in summary: Identify - identify trainees who are due for revalidation Preliminary checks trainee status/records ARCP CTT Check for concerns Form-R Live concerns records - serious & untoward incidents ARCP panel concerns Exit/Employer reports Collate concerns & compile recommendations Submit to Responsible Officer Submit to GMC (after approval) & check status	If these tasks are carried out by different roles, observation to be with each of these roles individually e.g. if a validation officer is responsible for preliminary checks but a separate person/team responsible for gathering and considering evidence on concerns, then we'd like to go through the part of the process with both of these people idividually.
Depth interviews (3 x 1hr)	2 or 3 1-2-1 interviews with other people who perform the role as above. 1-2-1 Interview with a validation manager if they are not covered in the Contextual Enquiry.	As above, need to cover different roles where these are involved (e.g. 1 from Validation team, 1 from Specialty team)
Interview (ideally 1hr but we'll take what we can get!	1-2-1 with Responsible Officer	This will be to understand his/her experience of the process, including audit/reporting requirements, plus wider insight into working practices (e.g. self service or supported), digital take-up etc NB - we may not be able to speak to RO in every local team, but will be happy to talk to as many as we can
Depth interviews 1hr	1-2-1 with trainee	Number of trainees depends on schedule - if visit is spread across 2 days this can be increased