Evaluation of assessment of tools to enable remote usability testing for the TIS project. Following options are all Cloud-based SaaS (Software as a Service)
NB. Features analysis based on published information
| Software | Description | Minimum system requirements (for desktop testing) |
|---|
| Licence | Indicative cost | Government Digital Marketplace | Devices | Formats | Remote access | Moderated testing? (Live interaction with tester) | Records screen | Records user - |
|---|
| audio | Records user - |
|---|
| video | Written survey responses | Activity tracking | Reporting | Comments |
|---|---|---|---|---|
Specialist cloud-based software that enables remote user testing in a 'virtual lab'
| Browser IE, Chrome, Firefox(versions not specified) Microphone
Requires users (research participants) to install browser add-on |
Microphone (essential) and webcam (desirable) required on user's PC
| Standard public sector:
| £24,140 pa (1 year minimum contract) | Yes |
| Desktop, tablet, mobile | Websites, mobile websites, apps, prototypes | Yes | Yes | Yes | Yes | Yes | Heat maps & clickstreams | Automated dashboards Data export into Office Suite | Most widely used Only one I can find on G-Cloud (Govt digital marketplace) | |
| User testing and research software platform - partnered with JustInMind prototyping tool which is being used in TIS UI design
| Browser Google Chrome v51+ Operating system Windows 7 and above Microphone Uses a Chrome app, which automatically downloads and installs at the beginning of a test |
Microphone (essential) and webcam (desirable) required on user's PC
|
Professional:
| $199 / month
(Basic licence available for $79/month but this does not induce moderated testing) | No | Desktop, tablet, mobile | Live websites, mock HTML, images, prototypes (Invision, Azure, Market, JustInMind | Yes | Yes | Yes | Yes |
| No | Yes | Path reports (which pages users viewed) Average time on page Page exits | Automated reporting Charts, filtering CSV download | |
| TryMyUI | Remote usability testing service | Browser requirements not specified on website (and couldn't fine via Google search either) Microphone and webcam Users must download a recording app to enable them to feedback and a test file for each new test they take part in |
Microphone (essential) and webcam (desirable) required on user's PC
Team: 10 tests a month 10 test results "from your own users" (i.e. not ones they've recruited) Multi user login for sharing research 10-min highlight video reels of results Agency/Enterprise 100 tests a month Unlimited test results "from your own users" Multi user login for sharing research 20-min highlight video reels of results One-click report generation with integrated video playback | Team $299/month Agency/Enterprise price on application
| No | Desktop, tablet, mobile |
| Images, wireframes, websites, mobile websites | Yes | No | Yes | Yes |
| No | Yes - up to 4 questions
Built-in questionnaire - System Usability Scale (SUS) and Single Ease Question) incorporated | Screen capture of user interactions - clicks, taps, swipes
| Automated reporting Task duration and completion rate info (But clicks etc only visible on screen recording?)
| Seems to be more focussed on running tests on clients behalf (incl. recruiting panels) than a truly DIY tool.
Appears to be quite structured and incorporates standard questions - allowing us to ask 4 additional questions. Doesn't appear to be 'self-service' - they are running the tests and editing the feedback? So doesn't seem as flexible as other tools? |
| Userlytics | Remote usability testing service | Browser
Microphone and webcam Users must download an app which checks they have the necessary equipment (e.g. webcam) and takes them through the test tasks |
Microphone (essential) and webcam (desirable) required on user's PC
Pro Plan
| $13,900 pa Estimate based on 24 tests (2 per month) with 20 participants each | No | Desktop, tablet, mobile | Websites, prototypes (incl. wireframes, static images), videos, mobile apps, display ads, search and social behaviour | No | Yes | Yes | Yes | Yes - up to 100 questions | Captures screen interactions (computer and mobile devices) | Not automatic - get video and annotations but need to request analysis and executive summary of the results | As above, seems to be more focussed on running tests on clients behalf (incl. recruiting panels) than a truly DIY tool. | ||||
| User Testing | Remote usability testing platform | Browser IE8 + / Firefox 3.5 + / Safari 4 + / Chrome Operating system PC - Windows XP + Mac - OS X 10.7 + Microphone and webcam Users must download a screen reader
| Basic Pay per tester Video feedback only (no analytics etc) Pro Plan Full functionality but website doesn't detail extent of testing or testers (details negotiated on application) | Basic $99 per video ($49 for first 10) Pro Plan Price on application | No | Desktop, tablet, mobile | Websites, prototypes (including wireframes), apps, | Says it does but I can't see any evidence on website - service element includes providing moderators so not sure able to DIY | Yes | Yes | No | Yes | No | Yes - of survey results (Main output is videos) | "The World's most popular user testing platform" As above, seems to be more focussed on running tests on clients behalf (incl. recruiting panels) than a truly DIY tool. Qualitative and survey data only |