Assessments research

Scope

• Assessments development will also include the initial stages of design and development for the Trainee Portal
• 'Assessments' refers to the progression reviews carried out at least annually, but may be more often, and not to the workplace-based assessments that are managed on the various ePortfolios
• The assessments function will support reviews across medicine, dentistry, pharmacy and healthcare sciences

Objectives

To understand from administrators' perspective:

• who does what in the assessments process
• the detail of the tasks they perform (what, why, where, how)
• the inputs and outputs
• tools and systems used
• their experience of the process - what's good and what's not
• what do they need from a system to support the process

To understand from panel members' perspective:

• the detail of the tasks they perform (what, why, where, when and how)
• tools and systems used
• their experience of the process - what's good and what's not
• what do they need from a system to support the process

To understand from trainees' perspective:

• overall experience of the process - what's good and what's not
• the tasks they are required to perform - submitting information, attending meetings
• tools and systems used
• how they would like to interact - information, communication (content & channels)

Methods

Contextual Enquiry

• Sitting with a user and observing what they do
• Discussing observations, issues and ideas as they arise
• Duration as long as the task takes but minimise disruption of work
• Feedback can be anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas

Interviews

• Structured but informal interviews to talk through processes (e.g. ARCP) and discuss any issues and ideas
• May be 1-2-1 or include more than one person working within a team (not more than 2 preferred, max 3)
• Can be completed in 1 hour but more time useful for more detailed exploration
• Feedback can be anonymised – issues will be aggregated and generalised in project outputs e.g. user journeys and personas

Workshops

• Group of colleagues – either working in similar roles or representing users from end to end of the process
• Work together to create a common view of process pain points and opportunities
• Feedback can be attributed to whole workshop and not to individuals

Usability testing

• Reviewing users actually interacting with prototypes or developed software
• Can be moderated and observed - either face-to-face or remote using specialist software (e.g. or screen sharing (e.g. Webex)
• Can be self-directed by user using specialist software (e.g. User Zoom)
• Specialist software enables collection of quantitative data (time, clicks, heat maps) as well as qualitative (comments, opinions and observations)
  
  

Target coverage

During Alpha development we will aim to gather views from across the four professions in scope, appropriate to the level of development of the assessments process (Dental undergoing change due to devolvement to local teams; HSC & Pharmacy are currently piloting processes so do not have a large cohort of existing users), with the sample including a range of locations and specialties (not exhaustive but sufficient to give us a good feel for the requirements).

The Assessments Subject Matter Leads (SMLs) group includes representatives from each profession and plays a consultation and sense checking role. Get details of who they are and their activities.

The Assessments Design Process gives details of research actually undertaken and is updated regularly.