The Benefits of User Testing with Disabled Users
Jul 06 2012
Disabled users are users.
This might seem like a straightforward or even a trite statement, but its a point worth making since Usability (or UX – User Experience) is a growing field which has the fantastic goal (and one very close to my heart) of making websites, or pretty much you interact with, less annoying, more intuitive and generally just work better. Usability, however, is rarely seen by its experts as being linked to the accessibility concerns of disabled users.
Disabled users are a big segment of the population, however. The Australian Bureau of Statistics states that in Australia 18.5%, or 4.0 million people, reported having a disability in 2009. And since valid usability testing should discover the usage patterns of valid demographic groups, ignoring up to 18.5% of the population doesn’t sound like good representative sampling to me.
User testing is a part of usability testing, but user testing with disabled users also adds accessibility findings to the normal usability findings.
This post is about the advantages and special considerations for doing user testing with disabled testers.
A bit more about user testing
User testing is part of usability testing and is designed to bring out any problems your website audience has when trying to find information, buy things or complete any other action that’s essential on a site.
This involves developing realistic scenarios of tasks that your users would normally do on your website, for example, sign up to the mailing list.
People are then recruited who match your website audience demographics to work through the scenarios while recording any issues they have and note areas for improvement.
There are two broad approaches to user testing:
- Qualitative: one on one testing involving observing people trying to complete tasks on a website to find specific issues they have doing them.
- Quantitative: running workshops with larger numbers of people to get more generalised findings from more people.
User testing is different from accessibility testing which normally involves doing an accessibility audit against the WCAG 2.0 guidelines using automatic and manual testing by an accessibility specialist.
Testing with disabled usersAs I said at the start of this post, disabled users are users too. But as well as the ethical reasons for including them in user testing, gathering data from disabled testers is good practice because of the rich demographic data they bring to the findings as well.
- User with impairments fit into two broad demographic groups:
- Disabled users – who may be a non-specific group of disabled users or a grouped by specific impairments such as
- Visual impairments (blindness, low vision, colour blindness)
- Hearing impairments
- Motor skill impairments
- Cognitive impairments
- Impairments due to aging (some combination of any of the above)
Disabled users may also be included within other demographic groups for user testing where their profiles match the group being recruited.
Disabled users may be employed or unemployed; educated or under-educated; affluent or from poor socioeconomic backgrounds; native English speakers or ESL; technology ‘power users’ or computer illiterate.
The results of testing with these users can therefore add considerable depth and validity to accessibility audits and usability research projects to show hidden accessibility and usability problems.
What to test on a website
Typically, the areas of a site that have the most usability problems are the ones involving the most interaction with users, for example completing a purchase, viewing videos, searching for information or filling out forms.
These are also exactly the same kinds of things that should be tested with disabled users since these features also typically turn up the most errors in accessibility audits against the WCAG guidelines.
Since solving the usability issues for these kinds of features involves the most amount of work, this is one really good reason to include user testing with disabled users in the process at the same time. How counterproductive would it be to solve the usability issues for some users but not the usability needs of disabled users for exactly the same website features? Or worse, to increase accessibility problems while improving usability (for some)?
The testing environmentDisabled users have specific needs from websites and correspondingly have Assistive Technology (AT) requirements to allow them the level of access they normally get. Assistive Technologies enable people to perform tasks that they would otherwise be unable to accomplish, or had great difficulty accomplishing and include accessible keyboards and voice recognition software and screen readers.
Therefore to conduct user testing with disabled users, the test environment needs to be accessible for those particular users, otherwise the data that can be obtained from testing will be restricted.
This either means that the testing environment needs to be low tech – as in the case for workshop style testing – or set up for the particular impairments the testers have. As this can be difficult to manage, arranging one on one testing or remote in the tester’s home or office environment may be the only workable solution.
Recruiting for user testing normally requires that the level of technical savvy of the tester is known and documented. This is just as important with when testing with disabled users as difficulties they have with their assistive technology can be confused with difficulties they have with the site being tested.
Communicating user testing results
Presenting the results of user testing is excellent for showing the impact of usability and accessibility issues on real people. Summaries from testing sessions along with quotes from actual people can be very persuasive and make more of an impact than stating which WCAG checkpoints a site does or doesn’t comply with.
However, best is to have both. You can’t always test with the amount of users you’d like due to time and money restrictions., The Worldwide Web Consortium (W3C) page on user testing says, “This is where accessibility standards like WCAG really come into their own. By following such guidelines, you can increase your chances of getting a foundation of accessibility even for user groups you are not able to test.”
Also detailing how user testing results match up against the WCAG 2.0 guidelines is brings more validity to the findings because the guidelines are designed to meet the access needs of older users, users who have English as a second language (ESL), low literacy users and users with slow Internet connections as well as disabled users.
including findings against the WCAG 2.0 guidelines also means that the sufficient techniques listed for each of the checkpoints can be applied to give standardised guidance on the best approaches to fix accessibility issues that user testing turns up.
Therefore user testing with disabled users should be accompanied by an accessibility audit as well to ensure that the recommendations from user testing with disabled users don’t contradict accessibility guidelines. (That would be ironic, wouldn’t it?)
In summary, testing with disabled users adds richness to usability testing findings, highlights accessibility issues at the same time and ensures recommendations for fixing usability problems also fix accessibility problems.
Richard Corby is a web accessibility expert and is a partner at Webbism.