Conducting a Moderated Usability Testing for Screen Reader Users and Keyboard Navigation Users

This post on usability testing is being re-published with permission by Evelyn Sakura Brokering from her original post on Medium.com

Note: I would like to mention that although I have tried my best to write in non-offensive language regarding people’s limitations and disabilities, I may have used some terms in a way that someone might object to. Please let me know if you come across any such terms, as I am looking to be sensitive to offensive nuances. Thank you!

Have you ever navigated a computer only using the keyboard?

If you are in front of a computer with a keyboard, try starting by tapping the Tab key. You can begin navigating the screen, right? This is one of the many commands you can use on a keyboard to navigate a page, besides using a mouse, which requires orientation to where the pointer is. When you do this, you can also use a screen reader that will read out the selected content, helping you navigate the page on a screen.

Before I dive into this topic, I would like to explain who typically uses a screen reader and keyboard to navigate a page on a screen. Credit for the explanation below goes to Joel Isaac, who helped me educate myself regarding this topic.

Keyboard users: People with mobility impairments may use a keyboard, mouse or alternative mouse/switch depending on the nature of their impairment. A keyboard user may differ between a keyboard-only user and a screen reader keyboard user. Sometimes a screen reader can change how the keyboard interacts with the material being read/navigated.

Screen reader users: Low vision users may use a screen reader, but are more likely to use a screen magnifier that can enlarge their view and can provide some extra color contrast and formatting for easier readability. Since people with low vision can generally see, they would be more likely to prefer a mouse over a keyboard, though this may be an evenly mixed choice based on personal circumstances.

Screen reader with a keyboard or in conjunction with a braille display: Usually users who are blind will use a screen reader with a keyboard or in conjunction with a braille display that may also act as a keyboard. Blind users rarely use a mouse because the use of a mouse requires orientation to where the pointer is on the screen; a difficult task when you’re not seeing the screen.

A person using a Keyboard with a Braille Display.

User using keyboard with a braille display Photo by Sigmund on Unsplash

Project Background

I had the pleasure to conduct a moderated usability testing with screen reader users and keyboard navigation users for Lime Connect’s newly designed platform before its launch.

Who is Lime Connect?

Lime Connect is the world’s largest network of high-potential students and professionals with disabilities, connecting them to top careers and rebranding disability through achievement. It is essential that Lime Connect’s platform be user-friendly for individuals using screen readers and keyboard navigation users, since the primary users are people who have disabilities and who may rely on assistive technology.

Asking advice from Experts!

Since this was my first time conducting usability testing with screen reader users and keyboard navigation users, I reached out to all the people I knew who were experienced in this field.

Special thanks to everyone for the in-depth help in reviewing my guide to conduct this study. This could not have been a success without everyone’s help.

Now that you know the sources I followed are credible… I will talk about the steps I took to prepare.

Preliminary Step: Understanding the difference between accessibility vs usability

Even if a site is “accessible,” it doesn’t always mean it is usable. Take this as an example: The picture below shows a screen shot of the Safeway site. Is the footer accessible with the keyboard? Yes. But it takes 58 times tab key presses to get to the footer. So this is NOT usable for a key board navigation user.

Screen shot of the Safeway.com site.

UXR with Participants with Disabilities By Sheri Byrne-Haber

To check if the platform complies with accessibility guidelines, you would need to conduct QA testing. Ideally, it’s best to determine if the site is accessible before testing for usability to minimize potential obstacles.

Meeting the Web Content Accessibility Guidelines (WCAG), or any accessibility guidelines, does not necessarily mean it is “usable” for all users.

Step 1: Recruiting Participants

It is recommended to compensate participants in some way for a study, as people with disabilities often face disadvantages when seeking employment.

Ideal Number of Participants (8 people):

  • 3 Screen Readers: (Users who are legally blind)
  • 3 Screen Readers: (Users who are low vision)
  • 3 Keyboard Navigation users: (Users who are Low vision)
  • 3 Keyboard Navigation users: (No visual impairment, users with physical disability)

For this study, I recruited 5 participants who rely on screen readers and keyboard navigation for web browsing.

Recruiting participants with varying levels of tech confidence, software knowledge, and familiarity with commands yields a well-rounded result.

Step 2: Pre-Research Questions — Ask Questions Before the Interview

Knowing before the interview the technology participants use and how familiar they are with it is crucial, as everyone has different levels of familiarity. I took some time to try out the technology myself beforehand by using various screen readers and navigating the site using only the keyboard. This helped me gain a better understanding of the user flow that users will follow when conducting the tasks I will be asking them to perform.

I asked three questions:

  1. What assistive technologies are you currently using? Could you please provide details of both the software and hardware, including the model numbers?
  2. How long have you been using this technology?
  3. On a scale of 1 to 10, how well and familiar would you say you can use the technology?

Know what technology participants already use and how well they can use the technology before the interview.

Step 3: Create a Usability Testing Guide!!

This is going to be your best friend on the day of testing. For those who have experience conducting moderated usability testing, it is not too different. Create research goals, research tasks, methodology, interview questions, etc… My guide turned out to be 8 pages long. Here are some questions I prepared specifically for usability with assistive technology.

Assistive Technology Specific Questions:

  • How easy or difficult was it to navigate the website using your assistive technology?
  • Were you able to understand the content and information presented on the website using your assistive technology?
  • Did you encounter any barriers or challenges while using the website?
  • Were you able to access all the functionality and features of the website using your assistive technology?
  • Was the website designed in a way that made it clear what actions you could take and what information was available to you?

Step 4: Conduct the Test!!

Once I was confident with the testing guide and had spent some time using the software myself beforehand, I was ready to conduct the test!

Below are the 7 tips that helped me conduct the test:

Tip 1: Help users feel as comfortable as possible!

  • Be vulnerable: Here’s how I introduced myself: “I have dyslexia and use text-to-speech technology every day. My goal today is to make this platform as accessible as possible for everyone.”
  • Let users know that “I am testing the product, not you. There is no ‘right’ answer.”
  • Ask if they’d like a description of you or any of the visual elements with which they’ll be interacting.

Tip 2: Give time to adjust

When introducing the participants to the initial screens, allow them about 5 minutes to familiarize themselves by exploring the screen in a way that is most comfortable for them (Joel Isaac).

Tip 3: Check the “Share Sound” button on Zoom

When the participant is sharing their screen in Zoom, make sure they check the ‘Share Sound’ button to hear the screen reader for recording. Mute yourself as they complete the task.

Screen shot of the Zoom Share Sound checkbox.

Tip 4: Ask users to open the “Speech Viewer”

When users open the “Speech Viewer,” you can visibly see what users are listening to. This will help you follow along when the screen reader output is fast.

Screen shot f the NVDA Speech Viewer dialog.

Tip 5: Ask users to slow down

If the screen reader output is too fast for you to understand, you may want to ask the participant to lower the speed of the screen reader. But keep in mind the slower speed may affect their performance. (Joel Isaac)

Tip 6: Ask users to pause the screen reader when speaking

Ensure that the participants pause the screen reader when they are speaking or when you want to talk. It may be difficult for you to understand when both the participant and the screen reader are speaking at the same time.

Tip 7: Ask Questions!!

I acknowledged that I didn't have a complete grasp of their workflow, so it was necessary to ask lots of questions during the testing.

Before the test, I wondered if participants would be comfortable turning on their camera over Zoom. Usually, I ask participants to turn on their camera to see their facial expressions and identify frustrations. However, I was thinking that if the participants cannot see me, it would not be fair. Joel Isaac told me to “Build a rapport with your participants so there’s no fear around asking potentially sensitive questions. Basically, when in doubt, ASK!” Part of me was nervous to ask this question of participants, but to my surprise, some participants were happy that they got to turn on their camera.

Results

The usability study was a significant success; I was able to identify over 25 usability obstacles. I conducted rounds of testing with participants with vision, and to my surprise, some of the key findings were overlapping. The improvements that will be made for the screen reader users and keyboard navigation users will ultimately help people with vision as well. I may write a separate article just for the findings.

(Explanation below goes to Makoto Ueki.)

It is essential to identify the root causes of the issues that have been found. For each problem, it’s advisable to assess which of the following four categories it falls into and then use this information to inform your subsequent improvement plan.

  1. User agents (browser / screen reader)
  2. Web content authors (HTML / CSS / JavaScript, etc.)
  3. Authoring tools (CMS / blog / Any other platforms, etc.)
  4. User’s skill

If it is due to #2, it’s pretty straight forward. Just fix it 🙂

Conclusion

To my surprise, I was able to understand most of the screen reader output. Since I use text-to-speech technology every day, my strength in comprehending fast text-to-speech technology helped during this session.

I witnessed first-hand why some WCAG criteria are set in place, such as the importance of ARIA Landmarks; even making small changes in the order of H-tags can make a huge difference when navigating a platform. This test transformed my perspective as a UX designer from “I want to design accessible platforms” to “I HAVE to design accessible platforms.”

Technology plays a significant role in some people’s lives. I see the unlimited potential that assistive technology has to empower people and raise our standard of living, particularly for individuals who have specific limitations; it can be a life-changing tool to use. In my case, text-to-speech technology is something I cannot live without. We have to take accessibility into account as we design, since the people who need it heavily rely on it.

If you ever have an opportunity to conduct usability testing for screen reader users and keyboard navigation users, DO IT!! It will provide you with so many new perspectives as someone who creates digital products.

Thank you, Lime Connect, for valuing inclusivity in your platform and allowing me to contribute to making it more useful for all users.

About the Author

Portrait of Evelyn Sakura Brokering.

Hi, I am Evelyn Sakura Brokering.

I'm a bilingual Japanese-American UX Designer specializing in highly accessible, user-centric designs. I am a strong advocate for assistive technology/accessibility and its ability to empower people. As a UX designer, I am determined to use my voice to promote accessibility and inclusivity in design.

My personal connection to accessibility stems from my own experience with dyslexia and my bi-cultural background, which provides me with a unique perspective on non-verbal visual cues and data visualization. Let's collaborate to create intuitive and accessible products for your users!

My Page: https://www.evelynbrokering.com/about

WebAlign

If you are looking to add understanding and clarity to your web accessibility efforts, consider using WebAlign. With your WebAlign subscription you have access to guidance for your designers, developers, and QA/Test teams on how to incorporate, code, and verify accessibility on your sites. You also have weekly access to our live Zoom calls where you can discuss accessibility topics with our experienced accessibility professionals. And there is our WebAlign community user forum to post questions and bounce ideas off other WebAlign users who may have already dealt with issues you are currently facing.

Stickman with a cup of coffee.

For less than what most spend on coffee at their favorite coffee shop each week, you can have access to the most up to date accessibility compliance resource available today.

To find out more about how WebAlign can help you and your team successfully design, create, and maintain accessible web content, please visit our What is WebAlign page.

Want to See More Content Like This?

Want the latest blog posts, videos, white papers, and announcements? Sign up for our mailing list and stay in the loop!

We're Here to Help When You're Ready

Take a deep breath. Then feel free to reach out to our team when you're ready to discuss your accessibility needs.

0 comments on “Conducting a Moderated Usability Testing for Screen Reader Users and Keyboard Navigation Users

Leave a Reply

Your email address will not be published. Required fields are marked *

Click to access the login or register cheese