Assessments - Feedback

The following fields are reported by teams as not being used:

• Review time, portfolio review date, trainee notified of outcome nor any of the additional fields.
• Review Time and Portfolio Review Date
• we don't have any use for the curriculum specialty (14) or sub-type (15) as far as I'm aware.
• Comments (12) should possibly be removed, or at least made only visible to local teams

Remove:

• review time
• portfolio review date
• trainee notified of outcome

The Assessment Status (4) field could be useful if we could report on it properly; it's not useful in and of itself currently to us

PYA (23) is not just for the completion of GIM - it's for any Medicine (JRCPTB) curriculum.

Academic curriculum assessed (1) refers to the academic curriculum the trainee is on (e.g. ACF, ACL) and was assessed in (Academic Outcome, 2)

Comments such as "Trainee progressing well" should now be included in the Supplementary Detail section under Detailed reasons for recommended outcome (1) or Additional comments from the panel (6), depending on the outcome/details

The field 'next review date' used to auto-populate the next ARCP form on the system, but no longer does this. This is a particular feature I used to use a lot and would be really useful to me if there could be something like this on TIS

The 'next review date' field; should this date be one year exactly after the assessment you are inputting? On the most part yes, but often we need to have a review before that, for those out of sync trainees who are due to gateway before the year is up or for trainees who have had an adverse outcome

Additional Fields, we use Panel Members. Employer/Exception reports is a legacy field to avoid data issues (this process was replaced with the LEP Report). In theory, the other 3 Additional Fields could be useful, but our ability to customise these has been very limited, and I'm not sure how much use they'd be going forwards

There's kind of two parts to this; I'll talk about the second in a bit (how the fields could be improved), but the first is more of a national/HEDG/whatever decision, which is whether these fields are still necessary? As far as I'm aware, the documentation/activities/exams fields are a carry-over from the old RITA days, when panels might need to record the specific evidence seen. However, now the panels just view the ePorfolio; there might be particular pieces of evidence that are especially relevant to the ARCP outcome, but these are should be commented on in the appropriate Supplementary Detail fields anyway (i.e. if the reason for an O3 is exam failure, "Detailed reasons for recommended outcome" should be, for example, "MRCPCH Part 2 failure"; if an exceptional O3 is being awarded following recommendation from OH, "Mitigating circumstances" would include "Letter from OH regarding sickness"). I personally think that the "Activities", "Exams" and "Documentation" fields are surplus and not useful now, but that would probably require agreement from HEDG/similar

If we are expected to continue to use these fields, for starters, being able to have 'default' lists for certain curricula would be useful. For example, having the MRCGP exam results available as an option for evidence considered would be a complete waste in Paediatrics. We should also be able to isolate this by regions; for example, our Paediatrics programme uses summary documents, which the trainees complete to summarise their evidence/training over the period being assessed, and one of the panel members uses this in the ARCP (adding their own notes having reviewed the portfolio ahead of time) to 'present' the portfolio to the panel. I don't know of any other regions doing this, but having the ability to include "Summary document" in Documentation would be useful to us. This would probably require having something like a national reference table with the ability to add ARCP Acitivites/Exams/Documentation to a curriculum (or vice-versa), and within that, mark which regions it would be available to (similar to the new Associated Local Offices field in Intrepid)

Panel members we use anyway, but would need to have some kind of reference table to populate this with, or a more streamlined approach would be to have panel members 'book' on to certain ARCP panels, with admins then having the facility to remove them from entire panels (if they didn't show up) or from individual ARCPs (if they left part-way through the day/had to step out as they were a particular trainee's Educational Supervisor/whatever)

Employer/Exception Reports is a legacy field; I think this should probably be kept as a read-only legacy field for the time being, as it'll only apply to ARCPs between April 2013 (when revalidation started for junior doctors) and Winter 2015 (the last round of Employer Reports before Live Reporting was introduced with the LEP Reports)

• Joanne Watson (Unlicensed) to get feedback on this
• leave all fields as is for now

the Concern Summary should have text from the ARCP panel with regards to revalidation (e.g. "Trainee declared unresolved significant event; no current concerns but awaiting coroner's report"), whereas the Responsible Officer's Comments are for any comments the Responsible Officer needs to make on the form (as the ARCP form is the means of transferring information to a doctor's next Responsible Officer); this can either be regarding the information noted on the ARCP form by the panel (e.g. "Coroner's report received 06/11/2017 and reviewed by RO; no concerns") or to add additional information not covered by the ARCP panel

Currently working on GMC ARCP return and what I think would be really useful going forward is the drop down options for Grade at next rotation should include the drop down options required by GMC as well as relevant grades, for example, F2 ARCP 6 needs "Not known from ARCP as trainee applies to specialty training" or Specialty Training ARCP 6 "Trainee completed training - not applicable"

• Change values within "Grade at Next Rotation Field" drop down (Joanne Watson (Unlicensed) to get GMC ARCP return recognised fields)
• Within post-assessment detail field #9

Journey

• bulk upload of assessments within Assessments megamenu
• view/search within assessments with filtering
• person record shows outcome only
• click through to assessments detail page within Assessment
• indicate on navigation to show them where they are (STEVE)
• back button to take user back to previous location

Navigation

• display Assessments at the top level

as long as there's still legacy outcomes for RITAs/LATs/FTSTAs, I don't think we'll need those any more (RITA and FTSTA are gone, and I think LATs are only continuing in the other nations) Otherwise it looks fine

There should be an option to create an assessment in a  person’s record too

Also I do not believe that a front page for Assessments showing all assessments, for all trainees of all specialties, is particularly useful. I am not sure I would ever navigate to a trainee’s ARCP pages from this page as it brings up all trainees of the same name?

Being able to batch create/update assessments for a number of trainees at a time would however be useful

Under Event ‘trainee ID’ needs to be GMC number

Layout of page is clunky. Forename and surname are better suited next to one another  

One page to add all ARCP info would be easier to navigate than 4 different pages

Adding an assessment via the trainee record has been included in interation two of component - as demoed in the S&T 1/2/18.

Bulk update functionality is in backlog of tasks

Aim to remove the trainee ID from all areas of the UI - frequently reported feedback.

UI flow has been designed to go down each column rather than along the row.

I can add the date, type and Status, but I am not able to add anything under Programme number - I can't even click into the box.  It also shows Programme name is populated but underneath in red highlights that Programme name is required.  As I can’t save this information, I am unable to test further in adding an assessment.

IE issue.

IE compatibility issues are in the process of being worked through and added back in to E2E testing.

It would be more helpful if there were clues of how to use this page like ‘type in GMC number first’.

RITA shouldn’t be an option under ‘type’ for anything but legacy items. No trainee should be awarded a RITA anymore.

I’m unclear how the type of ‘academic’ would be used so I’m not sure how useful that would be. All academic assessments are by definition also an ARCP.

It feels like there is a lot of blank space on the page and fields could be pulled together a little so you don’t have to scroll down/worry you’re missing something.

Tool tips

Is this a common agreement?

Post-assessment

Links to having all the info on one page issue.

I can create an assessment and add information in the fields, but I am unable to save this information in Pre – assessments and Post – assessment limiting me to further testing. 

1. Dropdown options for grade at next rotation don’t include CT1, CT2, ST1 to ST8 options or not applicable/other
2. Grade at time dropdown not available
3. There is nowhere to record the outcome of the assessment
4. It is unclear how much space is available to record comments on reasons for outcome etc. This appears to display as a single line of text rather than an expanding text box.
5. Rather than true or false, could this be yes or no?
6. It would be better if dates were shown as dd/mm/yyyy rather than yyyy/mm/dd
7. Can the assessments be displayed with the most recent first
8. It would be much easier if all information was on one page, rather than 4 separate tabs.
9. When searching for a trainee in assessments it appears to bring up trainees several times (for each recorded ARCP), can it just display the trainees once
10. I don’t appear to be able to add new assessments for current trainees, though this could be because the system is set up for testing on a limited number of test trainees or I am missing something
11. Can we have the option to search for a trainee by GMC/DRN in the assessment section

1. Post-assessment
2. Should autopopulate based on trainee's placement at Review Date - not meant to be editable.
3. Post-assessment
4. Not a comments box. Adverse outcomes and Not assessed have reasons.
5. Agree, need to look at the use of True and False
6. Date format - speak to devs again - may link to more focussed filtering though.
7. On assessments list? Yes, there is a Sort function - will demo.
8. Previously mentioned
9. Brings up each assessment event the trainee has had - to enable the end user to see the assessment history of the trainee.
10. Second person to mention this - will check permissions
11. GMC/GDC definitely

1. Where has the trainee ID come from (apologies if I missed this as I couldn’t join the assessments Skype last week)?  Without it I cannot create an assessment yet I don’t know where the number is derived from.  When searched for the trainee in people I can then find the number but it seems a random number?
2. Assessment type does not list IRCP and RCP (Dental assessment types)
3. PYA is only relevant to medical specialties; could this therefore not show unless a medical specialty is selected?
4. Grade at next rotation – not applicable to all, DFT for example is a 1 year programme therefore this field is not applicable yet is a required field.
5. Where exactly do you select the assessment outcome?!  Cannot see an option to select.
6. Could true and false be yes or no?
7. Dates not in consistent format, would be better to be DD/MM/YYYYY
8. Would be better if all one tab rather than separate ones
9. Cannot see a way to bulk create assessments?

I can create an assessment by entering a trainee ID (randomly typing in numbers and picking one, but I have no idea what the Trainee ID is).

It defaults a programme but I don’t know on what basis, the programme associated with the trainee completed in 2015.

If you start tying in a Trainee Surname or First Name the drop down boxes appear blank, though you can select a record by clicking somewhere in the blank box.  When I typed in Rich (in the surname box) and clicked in the blank drop down box I ended up with Richard John Brown (Rich was not part of the Surname).

The programme name for Richard John Brown was populated but the programme number wasn’t (though it is visible via the Programme Membership screen).

The calendar doesn’t work properly as reported in other streams.

• Good can search for GMC number, but field needs to be updated to show GMC number and not Trainee id

New assessment name box- when typing it pops up and shows empty, no names but if you click on empty box, a selection is made…. can’t see who is being selected

• In Post-Assessment detail, academic curriculum assessed field drop down needs to be sorted.
• In Post-Assessment detail, academic curriculum assessed field – cannot select an option from the drop down list.

• I tried to type in a trainees name and no names appear on the dropdown list to select.
• I typed a trainee’s GMC number (7529299) into the ID field and it converted it to their internal system ID. The internal system ID should never appear on the front end.
• The Status field automatically populated with “Appealed” when I chose the trainee ID. The status should only automatically populate after the review date field has been populated so that they system can calculate if the status should be “scheduled” or “overdue”.
• The programme number should automatically populate the way the programme name automatically populates.
• The Type, Status, and ProgrammeName fields automatically populate but red warnings still show up below them saying values are required.

• We think Trainee ID should be changed to GMC number.
• We would prefer to be able to type dates rather than using the calendar.
• We think the true/false drop-down options should be replaced with yes/no.
• 4 pages to complete one assessment seems excessive, we think it would be better for this to be on one page.
• We don’t like the layout of the fields, it is unclear how you navigate the page (the order in which you should fill in the fields).
• When the programme name is blank, you can not progress, as the field is still required.
• When the grade at time is ‘not available’, we would like to be able to type over this.
• Where do you complete the outcome of the assessment?
• When saving the event page, it loops back to the assessments list rather then staying on the page for you to complete the other sections.

I would want to be able to create an assessment from within a person’s record.

Would want an option to filter by specialty on the front page of assessment.

I am not able to access the drop down for type or programme name

Assessments are not in date order

Pre Assessment:

PYA is only associated with Medicine ARCPs in Peninsula

Feels a little unnecessary to have to click through four tabs of information for one assessment – could all information be on one page?

On reviewing assessments for Dr Husam Jamil and Jenna McMinn, it hasn’t pulled through all details:

Missing: Review date, status, grade at time, (Portfolio review date – is this field necessary as there is already a review date) Curriculum Specialty, completion of training date prior to review, 10% Lay and External have not been pulled through.

Defaulting to ‘False’ where the field has been left blank is distracting.

The field ‘Time out of Training’ can’t be required as sometimes trainees don’t submit their Form R (and get a 5 for it). They would subsequently submit it, but that is sometimes after the ARCP is awarded. I also don’t think it should be listed in the pre-assessment section as this section is often filled out a long time before the ARCP while we may only just get TOOT a few days before the panel.

I don’t see anything under ‘documents considered’, I imagine that will be fixed in the next iteration.

PYA probably doesn’t need to be a required field and could be a ‘tick’ box as the vast majority of ARCPs will be a no. PYAs only happen for a few trainees and only in 1 year of their training so it seems like a lot of ‘no’ entries to make for a few ‘yes’ entries. In addition, if you go to older ARCPs, this wasn’t a required question but rather a tick box and it looks like you haven’t mapped a non-tick to mean a ‘no’ in this question so whenever you go to an old ARCP, you would need to fill out this ‘no’.

Grade at Time: sometimes it says ‘Not Available’ – I assume this is because the post doesn’t have the right information. In this case, we should be able to manually override the post information so that this field can be filled in from the user’s knowledge.

I haven’t provided any feedback on the post-assessment detail and revalidation as you said those aren’t ready yet.

On reviewing some trainee's record's, I have noticed missing/ not pulling through information such as curriculum specialty, title, grade at time.

Academic curriculum is mandatory and not all trainees are academic.

I am able to view trainee's records and edit the information but not able to save these changes.

·        In pre-assessment detail-Is the portfolio review date the same as the ARCP event date? Do we need both?

·        Would WTE months OOPR/T during period pre-populate if this was applicable? Unable to add any details into this field manually.

·        Do we need both “WTE months OOPR/T” and “Months OOPR/T” fields? Is there any difference between these two fields?

·        Does PYA need to be a mandatory field?

·        Should days out of training be in the pre-assessment detail section, as we would check this information from their Form R which is reviewed at the ARCP Panel? As this is a mandatory field, I need to add something into this field to be able to move on, which I wouldn’t be able to do before the ARCP Panel.  Would make more sense to have it under post-assessment detail.

·        Post-Assessment Detail – unable to select appropriate grade. Only options started “zzz” or “academic”. Perhaps this should be a drop down list rather than start typing? Need to type 3 letters before it brings up whole list, no matter what letters you enter. You can over-write these options with your own choice e.g. ST1 – is this right?

·        Academic curriculum assessed should not be a mandatory field.

·        I like the free text boxes for supplementary details, but more space to type would be useful. It would also be better if we could see all text at once rather than having to scroll across.  

·        Post Assessment Detail did not save the fields that we completed. We only completed the mandatory fields, perhaps this was because of the problems with the “grade at next rotation” field (please see previous feedback)?

·        There is nowhere to record the outcome of the ARCP.

·        We also need to be able to record if the trainee was not assessed at this ARCP and the reason for this, e.g. mat leave, sick leave, non-engagement, etc.

·        It would be helpful if the ARCP “status” changed to complete once you have completed the post-assessment details, rather than having to do this manually.

·         In post assessment – if I select something from the drop down menu, I cannot then go back to having nothing in this field.

·        We run a report from Intrepid to let us know the dates of upcoming ARCPs for all of our trainees to help us with scheduling – will we still be able to do this using TIS? This is critical for us and we would be unable to manage the ARCP process without the ability to do this. 

·        “Go back to Person record” does not always work.

In outcome window- under appeal, 10% audit, external trainer, trainee notified of outcome all have options of true or false. Perhaps Yes or No would be better suited?

Where is the outcome field? Past outcomes not pulling through into post assessment. When on post assessment screen in one of the previous outcomes, there is no indication of date/outcome you’re looking at. Ie, if a trainee has 8 ARCPs in history and you select one- can’t really tell which one has been selected…

Grade at next rotation is a free text box and allows you to type and save nonsense. No drop down, is it possible to have a drop down to select as currently on Intrepid with highlight of the current grade so we don’t have to go back out to check and come back in to select?

Next review date lets you select a date in the past (may be needed for amending past records, not certain about this), doesn’t let you manually type a date

Academic curriculum is currently a mandated field- not all trainees will have an academic curriculum and there is no option to select 'none'. Also, it should be alphabetised, and allow a drop down which currently there isn’t. It also lets you type and save nonsense.

When searching a trainee is assessments, it pulls through all previous ARCPs with oldest at the top…can this be the other way round? Oldest at the bottom..

• Dates are in inconsistent formats throughout assessment component
• On the list of assessments for a trainee, the column Academic curriculum assessed is always blank. This column should show the Curriculum name (DR2:   [DataRepositoryV2.Intrepid.HEE].[R0].[vwAssessment].[CurriculumName] and should have a generic title like Curriculum Assessed.
• Filter on status field – shows status.[overdue|completed|appealed|scheduled]. Would be better to have the prefix removed.
• Clicking on Go back to person record in Post-assessment detail takes you to a blank screen; Event, Pre-assessment and Revalidation all work as expected.

• The incorrect field of “Academic curriculum assessed” is appearing on the list of assessments for GMC 7529299. The field should be curriculum. The curriculum displayed needs to be the curriculum for the trainee’s clinical assessment not the curriculum for their academic assessment. Only academic trainees ever receive academic assessments.
• The review date and outcome fields are not populating with any values on the assessments list view.
• The “status” (overdue, scheduled, etc.) of the assessment is not displayed on the assessment list. This field to me seems more relevant to this view than to the assessment full details view.
• I don’t understand the completion of training date (prior to review) field. What is this?
• I don’t understand why the pre-assessment details and post assessment details are split in two totally different views. I would argue it is better to have them in two labelled sections in the same view. Otherwise, someone could easily get confused as to what assessment they are actually filling in post-assessment details against!

It isn’t clear if the page has saved or not when a change has been made.

Assessments not appearing chronologically

Not able to see outcomes

Pre Assessment:

Grade at time appearing as not available. Not able to manually enter this.

PYA should not be a yes/no box

Should days out of training be mandatory at this stage of the creating the record?

When you enter the assessment record you are then in the assessment section and if you click back you are directed back to assessment home page not the people page

In this view, as Academic curriculum assessed is the first field they are all blank as not all trainees will be academic.  The review date and outcome columns are all blank.  The dates are American  - can this be changed to British view eg. 23/01/2017. 

As there are 4 pages, once you have clicked into another tab of an assessment, you can no longer see what date or curriculum you are reviewing.  The reasons (U codes ) do not show in the outcome.  Assessments are not in a chronological order.

Supplementary details on post assessment tab does not expand with the text.  In order to read the information you have to put your cursor into the field and move it along with the arrow key .  This is not very practical or sensible.  Text could be deleted in error.

Revalidation section on Intrepid logs which documentation has been reviewed:

Form R, LEP report, ES report, SOAR (For NES Only) and Other.  These fields don’t appear in TIS and therefore that information has not pulled through.

I can view and edit previously and newly created assessments. The overall outcome list on trainee’s list of assessments is not currently useful as it looks like ‘review date’ only shows on this list if it is a newly inputted event rather than one that existed in Intrepid. Can this field be mapped to the existing field in Intrepid? Likewise, there are no outcomes listed against any ARCP (probably because it looks like this field doesn’t exist yet in TIS). ‘Academic Curriculum Assessed’ – will that map to the curriculum assessed? It isn’t currently doing that for existing or new ARCPs.

When you navigate from the person record list of Assessments, logically clicking ‘go back to list’ (upper left corner) would take you back to the list of ARCPs for that person. However, this actually takes you back to the overall list of all assessments, which isn’t actually useful. You will often want to go back and forth between a couple of ARCPs for a given trainee. This does happen when you click on ‘go back to person record’ when you are in ‘pre assessment detail’ but if you are in ‘post-assessment’, this link doesn’t work and takes you to a ‘assessments’ page (as if it isn’t referencing that ‘person’ record anymore).

The assessments spread over 4 pages is not very helpful. Once you move on you lose the date your reviewing etc. and it would be more useful to see the assessment as one whole event.

Dates set out in American style can this be change to a British view? 

You cannot expand text boxes (text in one long line) which makes reading the information very difficult indeed.

No information in the in the review date and outcome section in the assessment tab. Also the academic curriculum assessed is the first field which I don’t think is very helpful as not all trainees are academic.

Yes but why do we have to go here to add the detail rather than via the create assessment screen? I don’t appear to be able to select what I need and Save in this screen.

In person’s record, academic curriculum assessed is blank even though it is populated as can be selected, same with review and outcomes and some periods to and from dates column.

Post and pre assessments screens have no name of trainee or identifiable information -Forgot whose details was filling. If was completing this and had to go away and come back, it would be difficult to remember which trainee record you’re on and if it’s not saved, will have to go out and start again. Can identifier be added to the screen? -Not TIS ID because if transferring information from paper outcome, this is not something we have recorded so wouldn’t help in identifying

Supplementary details comment windows being on one line is really difficult to read if there is a longer paragraph. Not very user friendly. Can this expand into a box as currently on Intrepid so you can see full comment at a glance?

Saving a new assessment- no indication that it's been saved other than a flash of the screen. Would it be possible to add a message to say it's saved?  Changes to outcomes don’t appear to be saving

Pre-assessment details don’t seem to allow changes to be made

‘Documents considered’ in pre assessments seems to be floating, there is nothing below it, no option to add anything and can’t click on it… Should this field be in pre assessment? Would we not populate this after the assessment rather than before? Do we need this at all?

Navigating from post assessment to pre assessment brings up a blank screen

• When opening an assessment from person record, it opens to the post-assessment details page. We think it would be better for the assessment to open on the events page.

Data that is there looks correct but as above it has not pulled through everything.

This appears to be correct, but sometimes information isn’t pulling through – see feedback above for more details.

Doesn’t look like all data is being pulled through. 

• Doesn’t appear to be pulling all data through, for example review dates and comments made at previous ARCPs don’t appear to be recorded

• Status and Review date under Event have not gone across to TIS from existing Intrepid record.

Names don’t appear to be alphabetised, same as reported in other components

Not all data pulling through…can’t see previous assessments, review dates and outcomes on assessments page