Evaluation of assessment of tools to enable remote usability testing for the TIS project. Following options are all Cloud-based SaaS (Software as a Service)
NB. Features analysis based on published information
...
| Software | Description |
|---|
| Minimum system requirements (for desktop testing) | Licence | Indicative cost | Government Digital Marketplace |
|---|
| Devices | Formats | Remote access | Moderated testing? (Live interaction with tester) | Records screen | Records user - |
|---|
| audio | Records user - |
|---|
| video | Written survey responses | Activity tracking | Reporting | Comments |
|---|---|---|---|---|
Specialist cloud-based software that enables remote user testing in a 'virtual lab' | Browser IE, Chrome, Firefox(versions not specified) Microphone
Requires users (research participants) to install browser add-on |
Standard public sector:
|
£24,140 pa (1 year minimum contract) | Yes |
https://www.digitalmarketplace.service.gov.uk/g-cloud/services/763266489702926
| Desktop, tablet, mobile | Websites, mobile websites, apps, prototypes | Yes | Yes | Yes | Yes | Yes | Heat maps & clickstreams | Automated dashboards Data export into Office Suite | Most widely used Only one I can find on G-Cloud (Govt digital marketplace) | ||
User testing and research software platform - partnered with JustInMind prototyping tool which is being used in TIS UI design |
Basic
Basic
$79 per month
Browser Google Chrome v51+ Operating system Windows 7 and above Microphone Uses a Chrome app, which automatically downloads and installs at the beginning of a test | Professional:
| $199 / month (Basic licence available for $79/month but this does not induce moderated testing) | No | Desktop, tablet, mobile | Live websites, mock HTML, images, prototypes (Invision, Azure, Market, JustInMind | Yes | Yes | Yes | Yes | No | Yes | Path reports (which pages users viewed) Average time on page Page exits | Automated reporting Charts, filtering CSV download | ||
| TryMyUI | Remote usability testing service | Browser requirements not specified on website (and couldn't fine via Google search either) Microphone and webcam Users must download |
a recording app to enable them to feedback and a test file for each new test they take part in | Team: 10 tests a month 10 test results "from your own users" (i.e. not ones they've recruited) Multi user login for sharing research 10-min highlight video reels of results Agency/Enterprise 100 tests a month Unlimited test results "from your own users" Multi user login for sharing research 20-min highlight video reels of results One-click report generation with integrated video playback | Team $299/month Agency/Enterprise price on application | No | Desktop, tablet, mobile | Images, wireframes, websites, mobile websites | Yes | No | Yes | Yes | No | Yes - up to 4 questions Built-in questionnaire - System Usability Scale (SUS) and Single Ease Question) incorporated | Screen capture of user interactions - clicks, taps, swipes | Automated reporting Task duration and completion rate info (But clicks etc only visible on screen recording?) | Seems to be more focussed on running tests on clients behalf (incl. recruiting panels) than a truly DIY tool. Appears to be quite structured and incorporates standard questions - allowing us to ask 4 additional questions. Doesn't appear to be 'self-service' - they are running the tests and editing the feedback? So doesn't seem as flexible as other tools? | ||
| Userlytics | Remote usability testing service | Browser
Microphone and webcam Users must download an app which checks they have the necessary equipment (e.g. webcam) and takes them through the test tasks | Pro Plan | $13,900 pa Estimate based on 24 tests (2 per month) with 20 participants each | No | Desktop, tablet, mobile | Websites, prototypes (incl. wireframes, static images), videos, mobile apps, display ads, search and social behaviour | No | Yes | Yes | Yes | Yes - up to 100 questions | Captures screen interactions (computer and mobile devices) | Not automatic - get video and annotations but need to request analysis and executive summary of the results | As above, seems to be more focussed on running tests on clients behalf (incl. recruiting panels) than a truly DIY tool. | |
| User Testing | Remote usability testing platform | Browser IE8 + / Firefox 3.5 + / Safari 4 + / Chrome Operating system PC - Windows XP + Mac - OS X 10.7 + Microphone and webcam Users must download a screen reader | Basic Pay per tester Video feedback only (no analytics etc) Pro Plan Full functionality but website doesn't detail extent of testing or testers (details negotiated on application) | Basic $99 per video ($49 for first 10) Pro Plan Price on application | No | Desktop, tablet, mobile | Websites, prototypes (including wireframes), apps, | Says it does but I can't see any evidence on website - service element includes providing moderators so not sure able to DIY | Yes | Yes | No | Yes | No | Yes - of survey results (Main output is videos) | "The World's most popular user testing platform" As above, seems to be more focussed on running tests on clients behalf (incl. recruiting panels) than a truly DIY tool. Qualitative and survey data only | |
| Software | Description | Minimum system requirements (for desktop testing) | Licence | Indicative cost | Government Digital Marketplace | Devices | Formats | Remote access | Moderated testing? (Live interaction with tester) | Records screen | Records user - audio | Records user - video | Written survey responses | Activity tracking | Reporting | Comments |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Cisco WebEx | Online meetings audio/video conferencing | User must download and use prototype WebEx has some limitations and issues in Windows 10 (Microsoft Edge & Firefox limitations - IE recommended browser) | Basic Up to 3 people per meeting Premium 8 Up to 8 peopler meeting Additional features (not necessary for user testing) | Basic Free Premium 8 £15 pm | Yes | Desktop, tablet, mobile | N/A | Yes | Yes | Yes | Yes | Yes | No - although allows commenting | No | No | This is a good option to avoid potential issues with permissions for browsers/add ons Does not provide tracking or metrics - testing is by facilitator observation only |