AKA The Pitfalls of Accessibility Testing
Have you ever had a simple project around the house that spiraled out of control? For example, that dripping bathroom faucet. Along with that constant drip…drip…drip, there is also a nagging voice reminding you that your hard-earned money is slowly going down the drain. So, you finally decide to fix that faucet! Sometimes this is an easy task, but if you are like me, you often find you have opened a Pandora’s box of home repair. The leaky faucet you thought would take a few minutes to fix ends up being broken beyond repair.
You diligently obtain a replacement faucet only to discover that the last guy who replaced it had jury-rigged the setup. It is a miracle you did not have a major leak! This small leak that seemed easy to address has now grown into a monster you are not able to conquer. In comes the expensive plumber who says he can fix the faucet but without missing a beat also informs you that there are other major plumbing issues on the verge of catastrophic failure! About this time, you start wishing you were not a homeowner.
Web accessibility testing can often go south just as quickly. Last week, I had one of those projects and wanted to share my experience.
Any developer or tester can easily "see" the design and HTML for a web site when scoping out what should be tested for accessibility. We have been doing type of testing for over 20 years and in that amount of time you would think we have just about seen everything. Most of the time, when we scope out an accessibility review, we are spot on with the selected test sample and time required to complete the task. But sometimes, just like the plumbing project mentioned earlier, there can be problems that are not obvious until you are deep into the testing process.
It All Seemed So Straightforward
We were contacted by an east coast firm that creates marketing sites for customers. On the surface, the site under review seemed simple—just some basic information about a product or service, a few images, and several links to a backend eCommerce platform for processing the transaction.
"Easy-peasy lemon squeezy," I thought. And because the client was on a tight timeline and we normally do fixed-cost contracts on this type of work, we gave them a quick proposal and thought nothing more about it.
Then the accessibility testing began. One accessibility issue after another started popping up. Eventually, it was like peeling layers off an onion. With every layer peeled back, there were more issues to document. When you are under a time constraint, discovering more issues to document can be as painful as each layer of that onion is to the eyes! As this was a simple site, I was expecting to identify maybe 15-20 errors at the most. I ended up documenting 56 unique issues! To give you a point of reference, that is more than the number of issues we typically identify for a test sample of 25-30 pages.
This site made ample use of ARIA, while at the same time failed to address some of the basics of HTML accessibility. This showed that the developers were somewhat familiar with accessibility but had not first taken the time to understand HTML accessibility. Unfortunately, we see this a lot now days.
While none of the issues we documented were necessarily new, there were some honest and simple mistakes that were well hidden.
Example Issue
One such example was a button that displayed a selection of options. When testing with a keyboard, everything worked as expected. When testing with a screen reader everything appeared to work as expected, as the screen reader announced the name of the button and would announce the options displayed as you navigated through them. What was not announced was the role of the control (a button) and the current value, or state of the control (expanded or collapsed).
Upon code inspection, all the pieces seemed to be there. What was off, was the assigned role was coded as aria-role="button". If you are familiar with ARIA, you will immediately recognize that this should have been coded as role="button". That simple syntax error caused the JAWS screen reader to ignore the other ARIA attributes such as aria-expanded. Once the syntax error was fixed, JAWS announced all the necessary details as expected.
While obvious when explained, this issue was not so obvious after hours of manual testing and looking at the page source. To add insult to injury, many of the ARIA parsers did not pinpoint that issue. The only scanning tool that identified that issue directly was the axeDevTools browser extension.
Fixed Cost Contracts are Your Friend
Of course, it is not our client’s fault that the site was as complicated as it was. We are normally good at estimating the scope of projects but, like every accessibility consultant, we have some projects that lose money and some projects that earn money. In every situation, fixed cost contracts benefit our customers. While in some cases the actual effort may be lower than expected, fixed cost contracts still give customers a clear budget to work with. And in projects like the one from last week, they can save you a ton of time and effort.
Hidden Problems Are Real Problems
There is a more important reason for sharing this story than saving you money. When we first looked at the client’s site, we initially thought it was relatively accessible. But, just like my plumbing story, there are sometimes that only an expert can really identify the deeper looming accessibility problems. And if our client tried to evaluate the site for accessibility on their own or had decided to work with a less experienced consultant, they would have likely concluded that it was in rather good shape—just as we initially thought at first glance.
It takes expertise to understand and accurately identify some accessibility problems. When you are ready to take accessibility seriously, we are ready to help you see a clear picture of where your content stands in conforming with WCAG and what you need to do fix any existing issues.
Want to See More Content Like This?
Want the latest blog posts, videos, white papers, and announcements? Sign up for our mailing list and stay in the loop!
We're Here to Help When You're Ready
Take a deep breath. Then feel free to reach out to our team when you're ready to discuss your accessibility needs.
0 comments on “Why Can Manual Accessibility Testing Take Longer Than Expected?”