Page Comparison

Scope

• Assessments development will also include the initial stages of design and development for the Trainee Portal
• 'Assessments' refers to the progression reviews carried out at least annually, but may be more often, and not to the workplace-based assessments that are managed on the various ePortfolios
• The assessments function will support reviews across medicine, dentistry, pharmacy and healthcare sciences

Objectives

To understand from usersadministrators' perspective:

• who does what in the assessments process and how it differs between professional groups, localities, training stages and specialties who does what and
• the detail of the tasks they perform (what, why, where, how)
• the inputs and outputs
• tools and systems used
• their experience of the process - what's good and what's not
• what do they need from a system to support the process

To understand from panel members' perspective:

• the detail of the tasks they perform (what, why, where, when and how)
• tools and systems used
• their experience of the process - what's good and what's not
• what do they need from a system to support the process

6 local teams:

• East Midlands
• Kent, Surrey & Sussex
• North East
• South West
• Thames Valley
• Yorkshire and the Humber

(Others to be substituted if no availability with initial list - to total of 6 for Alpha)

Objectives

To understand from trainees' perspective:

• overall experience of the process - what's good and what's not
• the tasks they are required to perform - submitting information, attending meetings
• tools and systems used
• how they would like to interact - information, communication (content & channels)

Methods

Contextual Enquiry

• Sitting with an administrator a user and observing what they do
• Discussing observations, issues and ideas as they arise
• Duration as long as the task takes but minimise disruption of work
• Feedback can be anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas

...

• Structured but informal interviews to talk through processes (e.g. revalidationARCP) and discuss any issues and ideas
• May be 1-2-1 or include more than one person working within a team (not more than 2 preferred, max 3)
• Usually last about Can be completed in 1 hour but more time useful for more detailed exploration
• Feedback can be anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas

Workshops

• Group of colleagues – either working in similar roles or representing users from end to end of the process
• Work together to create a common view of process pain points and opportunities
• Feedback can be attributed to whole workshop and not to individuals

Model research schedule - 1-2 days with local team, depending on numbers, availability and location

Can be flexible depending on availability, but should generally start with an observation session with a validation officer/manager who actually carries out the tasks involved in administering the process. Whether we use 1-2-1 or workshops depends to a certain extent on the numbers of people available/willing to participate, but I'd suggest we stick to 1-2-1 (or 1-2-2) for the first couple of local teams at least, and move on to workshops if it feels like we're getting a consensus that can be played back to bigger groups (and as we develop assets that can be used as workshop stimulus).

...

Contextual Enquiry

(c. 2hrs?)

...

Validation officer/manager who actually undertakes the tasks required for revalidation (RV1-11) in summary:

• Identify - identify trainees who are due for revalidation
• Preliminary checks
  • trainee status/records
  • ARCP
  • CTT
• Check for concerns
  • Form-R
  • Live concerns records - serious & untoward incidents
  • ARCP panel concerns
  • Exit/Employer reports
• Collate concerns & compile recommendations
• Submit to Responsible Officer
• Submit to GMC (after approval) & check status

...

Depth interviews

(3 x 1hr)

...

2 or 3 1-2-1 interviews with other people who perform the role as above.

1-2-1 Interview with a validation manager if they are not covered in the Contextual Enquiry.

...

As above, need to cover different roles where these are involved (e.g. 1 from Validation team, 1 from Specialty team)

...

Interview

(ideally 1hr but we'll take what we can get!

...

This will be to understand his/her experience of the process, including audit/reporting requirements, plus wider insight into working practices (e.g. self service or supported), digital take-up etc

NB - we may not be able to speak to RO in every local team, but will be happy to talk to as many as we can

...

Depth interviews

1hr

...

Usability testing

• Reviewing users actually interacting with prototypes or developed software
• Can be moderated and observed - either face-to-face or remote using specialist software (e.g. or screen sharing (e.g. Webex)
• Can be self-directed by user using specialist software (e.g. User Zoom)
• Specialist software enables collection of quantitative data (time, clicks, heat maps) as well as qualitative (comments, opinions and observations)
  
  

Target coverage

During Alpha development we will aim to gather views from across the four professions in scope, appropriate to the level of development of the assessments process (Dental undergoing change due to devolvement to local teams; HSC & Pharmacy are currently piloting processes so do not have a large cohort of existing users), with the sample including a range of locations and specialties (not exhaustive but sufficient to give us a good feel for the requirements).

The Assessments Subject Matter Leads (SMLs) group includes representatives from each profession and plays a consultation and sense checking role. Get details of who they are and their activities.

The Assessments Design Process gives details of research actually undertaken and is updated regularly.