Using Disabled Users for Accessibility Testing
As mentioned in Ken’s initial post for this series, there are strong opinions about how accessibility testing should be performed and by whom. We are often asked if our accessibility reviews are done by users of screen readers or by users of other types of assistive technology. The simple answer is…no, we usually do not include disabled users as part of our initial accessibility reviews.
You may ask, why not? Let me explain; when a new client contacts us for help, rarely do they have a robust accessibility process in place as part of their product or web development lifecycle. In most cases, accessibility is mostly an unknown to these teams or they only have a basic idea of what is required to conform with the various accessibility standards and guidelines. While using disabled users to perform accessibility testing seems like a good idea, there is a lot that these teams need to understand before effectively incorporating this type of testing.
WCAG is an Elephant
We like to think of accessibility as a process. You may have heard the saying, “How do you eat an elephant?” Of course, the answer is, “One bite at a time!”
The first challenge is that most are looking to conform with the WCAG guidelines. As you may know, conforming with the guidelines is good but does not always result in the best user experience. Yet, WCAG is currently how we measure and demonstrate accessibility support within a product. So, the first part of the ‘process’ is understanding where your product or website currently stands regarding WCAG.
While the WCAG criteria and techniques give us a way to measure accessibility support, it is far from perfect. What do I mean by that? It means that these are technical guidelines and can be difficult to understand and to know if you have implemented them successfully. We see this all the time with customers who claim they have fully implemented WCAG yet miss some of the most simple and basic aspects of WCAG and end up facing a legal complaint as a result. A good example is WCAG Success Criteria 1.3.1 Info and Relationships. This criterion states, “Information, structure, and relationships conveyed through presentation can be programmatically determined or are available in text.” What does that really mean? In our WebAlign product, we broke this criterion down into seven Core Concepts to make sense of what this means and how to apply it! This just further highlights that incorporating WCAG is not as simple as some may think.
This is where using a quality third party accessibility services consultant, such as Converge Accessibility, can provide huge benefits. Obtaining a baseline accessibility audit helps to identify accessibility issues within the product and helps identify areas of weakness in the teams understanding and implementing of the WCAG criteria and techniques. A baseline accessibility audit also helps to educate the designers, copy creators, developers, and QA/Testers around WCAG along with the impact the various issues identified can have on different user groups.
Testing with Disasbled Users
While we do not typically use disabled users as part of our accessibility reviews, we do use assistive technology in our testing. This is also an area where there must be an awareness of the various caveats in doing so. For example, screen readers are not testing tools. They are created to allow visually impaired or blind users access to content. In so doing, many screen readers make assumptions for inaccessible coding. The JAWS screen reader will often guess at what text should be used to as a label for an input field when the coding does not specifically link the text label with the control. A novice screen reader user may hear the correct label description announced yet never realize that this was an assumption made by the screen reader and therefore incorrectly conclude that the input fields are correctly labeled. If JAWS is reading these labels okay, what does it matter? Because other assistive technologies may not make these same assumptions and you can end up with an inaccessible experience for other users. Also, not correctly linking a text label with the input control is a direct failure of WCAG Level A.
Using Real-World Assistive Technology Users
Digital accessibility is starting to see a lot of attention, yet it is still not easy to find accessibility testers with a strong background in testing and/or an understanding of WCAG and assistive technologies. You can find a lot of people who are willing to do accessibility testing, but most are new to the field and are still working through the learning curve that comes with WCAG, assistive technologies and accessibility in general. This is also true when it comes to finding real-world assistive technologies users to perform accessibility testing. Being an assistive technology user does not make one a skilled tester. Here is why I say that:
Inexperience
Software testing has been around for a long time and there are a lot of good software testers out in the world. Yet, there is a marked difference between a good software tester and one that can test the user interface (UI). This is because UI testing can be tedious and requires a lot more attention to detail than writing a script and evaluating the results that are returned from that script. The same is true with accessibility testing. It takes a certain aptitude to be able to focus on the accessibility aspects of the UI and controls along with being able to test for and understand the impact an issue may present for different types of users. While there are some very skilled accessibility testers who are also users of assistive technologies, the majority do not have a background in software testing and lack the experience to properly document an issue or fully understand and explain the associated WCAG criteria.
Inability to Fully Test the Entire UI
Another area of concern is that if a product has content that is completely inaccessible, a disabled user may not even be aware of the content and therefore never be able to fully test that part of the product. I have personally seen this occur. About 13-14 years ago I had the privilege of working with the Veterans Health Administration’s Section 508 Office. We were assigned to observe several screen reader users interact with a product to determine how well it supported accessibility. We were not allowed to provide any guidance or input and were to observe only. The users could verbally explain to us how they “saw” the interface. It was fascinating to hear their descriptions as each has their own vision of how the product was arranged on the screen and they were very skilled at learning how to use the product without any instruction. Yet there was one big issue, there was a panel in the UI that was completely inaccessible to a screen reader and the users were never given any indication this panel existed! As a result, they were missing key functionality that was required to use the software properly.
While we were able to report this issue because of observing this testing, if the testers were left to report their results without any oversight, this would have left a huge accessibility issue in this product even though the rest of the UI seemed to be very accessible.
User Bias
While we do not typically use disabled users as part of our initial testing efforts, we have worked with individuals in this capacity previously because of contract obligations or because the product was well enough along in an established accessibility process that it made sense to do so. Yet there is still another area where caution needs to be exercised with disabled user testing, that is user bias.
As a screen reader user, keyboard-only user, color blind user, etc., individuals tend to be biased toward their own impairment, which is completely understandable. This is not often considered when incorporating assistive technology user testing which can lead to missing key aspects of testing and/or issues that affect other groups of users.
Normalized Configurations
Another area to be aware of is the configuration of the assistive technology. The purpose of assistive technology is not for testing but to allow users equal access to the content and functionality of a product. As such, assistive technology not only “auto corrects” for poor coding at times but also allows for personal configurations. If someone is a real-world assistive technology user, chances are they are not using the technology in an out-of-the-box configuration or configured in a way to ensure that most issues will be detected. JAWS has the ability to configure its output so that graphics without an ALT attribute will be completely ignored. Many JAWS users may enable this because they get tired of hearing the cryptic source string for those images being read. This can lead to missing obvious WCAG failures.
Cost
The cost of accessibility testing is also a huge consideration for most. If using disabled users for testing, there will have to be others involved in that testing to ensure it is being done completely as noted earlier. If cost is a concern, then finding an accessibility tester who has experience with the various assistive technologies being used is critical to ensure testing is being done adequately and within budget.
User Group Testing
While we are not against using disabled users as accessibility testers, the fact is that there is a lot that needs to be accounted for in addition to the increased cost of a bigger testing team. If you can include these users as part of your testing, do so! If not, all is not lost. One of the biggest overlooked opportunities to gain valuable feedback from real world assistive technology users is usability testing.
Usability testing (or user group or focus group testing) is a great way to include real-world assistive technology users. This type of testing does not require a background in software testing to be beneficial and helps identify issues that are not discovered by accessibility conformance testing alone. The W3C provides a nice resource for involving users in evaluating web accessibility and I recommend you check it out if you are ready to move in this direction.
Summary
Incorporating accessibility into a product is a process. If you do not currently have an established and proven process, it may not be the right time to include disabled user testing. Instead, consider including these users as part to your usability testing. The feedback you obtain from those sessions will likely be more valuable than the cost of involving these users in the accessibility testing effort. Again, if you are at a point in the process to do so and the cost is within your budget, then do both!
We understand that there are various opinions on this topic. We would love to hear your view. If you have something to add, please continue the discussion by adding your comment to this blog.
While you are at it, please consider signing up for our mailing list and following us on social media!
Want to See More Content Like This?
Want the latest blog posts, videos, white papers, and announcements? Sign up for our mailing list and stay in the loop!
We're Here to Help When You're Ready
Take a deep breath. Then feel free to reach out to our team when you're ready to discuss your accessibility needs.
0 comments on “Viewpoint: Disabled User Testing”