...
Scope
- Assessments development will also include the initial stages of design and development for the Trainee Portal
- 'Assessments' refers to the progression reviews carried out at least annually, but may be more often, and not to the workplace-based assessments that are managed on the various ePortfolios
- The assessments function will support reviews across medicine, dentistry, pharmacy and healthcare sciences
Objectives
6 local teams:
- East Midlands
- Kent, Surrey & Sussex
- North East
- South West
- Thames Valley
- Yorkshire and the Humber
(Others to be substituted if no availability with initial list - to total of 6 for Alpha)
Research context
It's important to acknowledge that some of theses people/teams have been involved in previous research during Discovery and undertaken by Angelita during her business process mapping. Please refer to User Research Spreadsheet for details.
Objectives
To understand from users' perspective:
...
To understand from administrators' perspective:
- who does what in the assessments process
- the detail of the tasks they perform (what, why, where, how)
- the inputs and outputs
- tools and systems used
- their experience of the process - what's good and what's not
- what do they need from a system to support the process
To understand from panel members' perspective:
- the detail of the tasks they perform for revalidation (what, why, where, when and how)
- tools and systems used
- their experience of the process - what's good and what's not
- what do they need from a system to support the process
To understand from trainees' perspective:
- overall experience of the process - what's good and what's not
- the tasks they are required to perform - submitting information, attending meetings
- tools and systems used
- how they would like to interact - information, communication (content & channels)
Methods
Contextual Enquiry
- Sitting with an administrator a user and observing what they do
- Discussing observations, issues and ideas as they arise
- Duration as long as the task takes but minimise disruption of work
- Feedback can be anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas
...
- Structured but informal interviews to talk through processes (e.g. revalidationARCP) and discuss any issues and ideas
- May be 1-2-1 or include more than one person working within a team (not more than 2 preferred, max 3)
- Usually last about Can be completed in 1 hour but more time useful for more detailed exploration
- Feedback can be anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas
Workshops
- Group of colleagues – either working in similar roles or representing users from end to end of the process
- Work together to create a common view of process pain points and opportunities
- Feedback can be attributed to whole workshop and not to individuals
Model research schedule - 1-2 days with local team, depending on numbers, availability and location
Can be flexible depending on availability, but should generally start with an observation session with a validation officer/manager who actually carries out the tasks involved in administering the process. Whether we use 1-2-1 or workshops depends to a certain extent on the numbers of people available/willing to participate, but I'd suggest we stick to 1-2-1 (or 1-2-2) for the first couple of local teams at least, and move on to workshops if it feels like we're getting a consensus that can be played back to bigger groups (and as we develop assets that can be used as workshop stimulus).
...
Contextual Enquiry
(c. 2hrs?)
...
Validation officer/manager who actually undertakes the tasks required for revalidation (RV1-11) in summary:
- Identify - identify trainees who are due for revalidation
- Preliminary checks
- trainee status/records
- ARCP
- CTT
- Check for concerns
- Form-R
- Live concerns records - serious & untoward incidents
- ARCP panel concerns
- Exit/Employer reports
- Collate concerns & compile recommendations
- Submit to Responsible Officer
- Submit to GMC (after approval) & check status
...
Depth interviews
(3 x 1hr)
...
2 or 3 1-2-1 interviews with other people who perform the role as above.
1-2-1 Interview with a validation manager if they are not covered in the Contextual Enquiry.
...
As above, need to cover different roles where these are involved (e.g. 1 from Validation team, 1 from Specialty team)
...
Interview
(ideally 1hr but we'll take what we can get!
...
This will be to understand his/her experience of the process, including audit/reporting requirements, plus wider insight into working practices (e.g. self service or supported), digital take-up etc
NB - we may not be able to speak to RO in every local team, but will be happy to talk to as many as we can
...
Depth interviews
1hr
...
Usability testing
- Reviewing users actually interacting with prototypes or developed software
- Can be moderated and observed - either face-to-face or remote using specialist software (e.g. or screen sharing (e.g. Webex)
- Can be self-directed by user using specialist software (e.g. User Zoom)
- Specialist software enables collection of quantitative data (time, clicks, heat maps) as well as qualitative (comments, opinions and observations)
Target coverage
During Alpha development we will aim to gather views from across the four professions in scope, appropriate to the level of development of the assessments process (Dental undergoing change due to devolvement to local teams; HSC & Pharmacy are currently piloting processes so do not have a large cohort of existing users), with the sample including a range of locations and specialties (not exhaustive but sufficient to give us a good feel for the requirements).
The Assessments Subject Matter Leads (SMLs) group includes representatives from each profession and plays a consultation and sense checking role. Get details of who they are and their activities.
| Profession | Target | Progress (updated 27/02/17) | Required |
|---|---|---|---|
| Medical | |||
Administrators from 5 localities - including foundation, core and at least 5 different HST/GP | Interviews & workshops: EM / L&SE / NE / SW / WM / Y&H covering foundation, core, GP and > 5 specialties | No additional requirement | |
| Trainees from 5 localities - including foundation, core and at least 5 different HST/GP trainees | 1-2-1 interviews: CT2 Surgical - East Midlands ST6 General Surgery - Thames Valley Workshop: 9 trainees from different specialties - North East | Trainee/s from 2 further localities | |
| 3 x TPDs from different localities and specialties | 3 x TPDs | ||
| Dental | |||
| Administrators from 3 localities |
| Administrators from 1 x locality | |
| Healthcare Scientists | |||
| Healthcare Sciences improvement team |
| ||
| 2/3 trainees involved in initial pilot | 2/3 trainees | ||
| ?? Panel members ?? | Poss not required for Alpha? | ||
| Pharmacy | |||
| Pharmacy Training Improvement project colleagues | Interviews with Rosalyn Cheeseman (East Midlands) & Amandeep Doll Amandeep Doll is a member of the Assessments SMLs group | ||
?? Trainees ?? ?? Panel members ?? | Trainees involved in the pilot were only aware piloting ePortfolio and not that these were being reviewed for assessment, so not sure of value of this - Pharmacy pilot should be doing it's own research as part of process development? Pharmacy not focus for Alpha (HSC is being used in prototype as example of TIS accommodating different profession/processes). Panel members similar. To discuss. |
The Assessments Design Process gives details of research actually undertaken and is updated regularly.