TL;DR
Cliff Notes Version. If you want to use a great product like WebAlign to change your organization's accessibility culture, your organization will likely ask you to perform a pilot project (aka "proof of concept"). This is an opportunity to embrace so don't blow it! This post describes how to plan a pilot project to get the results you want.
It's the Sad Truth of Working in Accessibility
In 99.9% of most modern organizations, accessibility professionals face one unavoidable reality: we "don't get no respect". When our organizations get sued with an accessibility complaint, our leadership asks us why we didn't take care of the problem earlier. Yet, when we ask for budget to avoid these problems in the first place, they will hem and haw-- and then ask for detailed cost-benefit analyses. After all, the money to pay for that company picnic doesn't grow on trees (that's actually a true story BTW-- we know of one IT company with a large accessibility practice that diverted the entire annual marketing budget for the accessibility team to pay for an expensive corporate retreat-- and then later asked why sales were down for the year for that team). And, even when we present organizations with an obvious solution that will solve accessibility issues in a very cost-effective way (cough... WebAlign.. cough, cough), they will still insist on a pilot project (or "proof of concept") to validate its worth.
And, within the last two weeks, three of our prospective customers have announced that they are kicking off pilot projects around WebAlign. Two of them are large, multinational corporations and the third is a medium-sized municipality. And, as they started outlining their approach, it seemed that the skills we have as consultants at Converge Accessibility could be really helpful in making sure that they were successful.
Why You Need to Plan Your Proof of Concept
Without careful planning, a proof of concept is doomed to failure. And, what your leadership may not be telling you, is that they may expect-- even want-- you to fail at a subconscious level. Why is that?
It's because change is hard. When an organization has been doing things the same way for years, there's a strong resistance to suddenly change that way of doing things. It's a natural survival mechanism that's been programmed into us by billions of years of evolution. After all, the reason there aren't dinosaurs still roaming around gobbling up unsuspecting humans is that dinosaurs could not adapt to rapid environmental change. Deep down inside, all of us loathe change because we feel that change may hurt us, possibly even kill us.
Most of web accessibility takes place in a change resistant world. For instance, when we perform an accessibility audit and give our results to our clients, we know deep down that the accessibility improvements will only be temporary-- and our clients will be opening themselves to lawsuits within a year. The reason is that they resist change-- even if they know they can do things better, they just go back to the same old ways of designing and coding websites. In my opinion, even regular testing-- whether done with automated testing or with teams of (expensive) manual testers-- rarely yields substantial changes in the way that most organizations design, build, and test websites. Instead, the changes only affect the specific content that gets remediated (leaving them vulnerable in other places that were not tested) and lasts only as long as the organization continues paying for expensive testing services.
As you know, WebAlign is about changing an organization's accessibility culture by setting clear requirements in design, development, and testing that reinforce each other and that foster compliance with WCAG in the quickest and simplest way possible. While it's compatible with the way every organization works, it's still imposing change. And, because change is always hard, organizations may resist that change.
Use Data to Your Advantage
When I was at the Justice Department, I managed the first three government-wide surveys of the Federal government's Section 508 implementation efforts. It was obviously a massive job as the Federal government has countless agencies and over 9 million employees. We surveyed every executive branch agency and I got to know folks from all of them, ranging from tiny agencies like the 12-employee Marine Mammal Commission up to the Department of Defense, which had a little over 2 million civilian (not uniformed) employees. In conducting those surveys, I learned that there are two ways to conduct a survey-- one way is to just glean objective, scientific data and the other is to use a survey to shape the outcome that you want. As an accessibility professional, I suggest you focus your efforts on the latter approach.
But isn't it untruthful-- even deceitful-- to use a survey to gather incorrect information? I'm not suggesting that you focus your efforts on misleading results. What I am suggesting is that you need to try to shape the questions and answers to factor out the natural resistance that people have towards change. And, because that natural resistance can be very strong, it's important to leave no room for that resistance to sneak in.
What is a Proof of Concept and Why You May Want to (or Have to) Use One
A "proof of concept" (aka "pilot project") is a means of proving that an idea, initiative, or plan makes sense for an organization. Instead of rolling out a change across an entire organization, it's first safer and cheaper to institute that change in a much smaller format-- and then assess whether it was a good idea or a bad idea. This way, the entire organization doesn't have to go through testing a bad idea. And the costs are contained because it only needed to be implemented in a small group.
For instance, let's say your organization has a massive web development team. Or let's say you rely on vendors to develop specific projects. In either case, pick an upcoming project in the next few months-- preferably one that involves content or a team that has had accessibility challenges in the past. Also make sure that the project includes creative teams (UX designers, copy authors, and graphic artists), developers, and testers-- the personnel that WebAlign was designed for. Then, roll out WebAlign ONLY with that team on that one project. That is a perfect pilot project. Then, you analyze the results of that pilot project. What specifically do you analyze? We'll get to that below.
If you're interested in using WebAlign to change your organization's culture around accessibility, it makes sense to start with a pilot project-- even if you're not being asked to do it. The reason is because it shows your leadership that you're analytical and objective. Sure, you may be fully onboard and passionate about the idea of WebAlign, but that feeling may not be shared by everyone in your organization. Plus, by rolling it out in a smaller way first, you can discover different strategies for making it work even better. This way, when you do decide to go for the organization-wide rollout, your solution will be a more customized to the way that your teams work.
There's also a second reason why you should consider a pilot project: leadership buy-in. As I mentioned at the start of this post, accessibility is the ultimate Rodney Dangerfield in IT initiatives, but starting at the bottom of the hierarchy (in terms of respect) doesn't mean that you have to stay there. A successful pilot project supports logical business decisions. If you can show a cost-savings over using the usual consultant/auditing model, that affects Return on Investment. If you can show a genuine improvement in accessibility, that improves the overall corporate risk profile. These are solid business concepts that organizational leadership responds to. Keep it up and maybe the IT Security guys won't keep teasing you at the water cooler.
Use Data to Prove Your Proof of Concept
Objective Data
There is one obvious objective measure: "is the product more accessible?" I think that the best way to measure this is to have a third-party team test a sample of web pages from (1) a group of pages that were included in your pilot project and (2) a group of pages outside of your pilot project. Then compare the results and see how much accessibility has improved. But, as long as you have some real accessibility testing expertise in your organization, you could also do the testing internally.
Another objective measure is improvements in knowledge. How does that work? Say, for instance, your organization's web content always has trouble with heading structure. They just put <H4> elements directly <H1> elements and the headings themselves often don't seem to be used for structure. In that case, asking the following question both before and after the proof of concept may be very helpful.
Do you understand how header structure needs to be conveyed in both wireframes and HTML in order to comply with WCAG?
-
-
- Absolutely, I can easily find the correct code – and how it needs to be conveyed from design
- I have a reasonable understanding but I’m not absolutely sure
- I have some idea but don’t feel at all comfortable explaining this to others
- I don’t understand this area at all
-
I would bet that, after using WebAlign intensely on a pilot project, you survey will show a huge improvement here. And it's so easy to develop this kind of question-- just try to find weak areas in your team's web products and develop a few questions around them. Then, you'll be able to report great data by making statements like, "in the top five areas where our team has struggled in the past, the proof of concept team created code that had zero errors and the team also reported a 96% improvement in overall confidence in handling this kind of content." If that doesn't win you respect, start working on your resume.
Subjective Data
Objective data like we talked about above is relatively straightforward; subjective data takes a bit of careful planning. While subjective data can still be a winning formula, it can also be a trap. For instance, because of the natural resistance to change, you never, ever want to ask completely open-ended questions like,
"Overall did you find adding this new process to be enjoyable?"
"Would you like to add this new process to the work you already have?"
In fact, even open-ended questions like, "Do you have any other thoughts about this new process?" is just inviting trouble. Instead, it's important to focus on closed-ended questions (e.g. multiple choice) and to be very strategic with open-ended questions by setting boundaries.
Proposed Subjective Questions
Closed-Ended Questions
Closed-ended question (e.g. multiple choice) are usually the safest. The following list will be updated as time goes on so that all of our WebAlign prospects can pick-and-choose the questions that fits your needs. Please feel free to steal and modify!
Do you feel confident in your knowledge of what you have to do to make [OUR ORGANIZATION'S] web content comply with WCAG?
-
- Absolutely confident
- Somewhat confident
- Not confident
- No idea what I have to do
Do you feel that WebAlign would help you make [OUR ORGANIZATION'S] web content comply more easily with WCAG?
-
- Absolutely, I think WebAlign would help
- Yes, I think WebAlign will generally help
- Maybe, but I’m not sure WebAlign will help
- No, WebAlign won’t help at all
Do you feel that you have a clearer understanding of your role in improving web accessibility at [OUR ORGANIZATION]?
-
- Yes, WebAlign helped me understand how my role fits into accessibility better
- Somewhat, but I’m still not clear on my role or the roles of others when it comes to accessibility
- Not really. I’m still not entirely clear on how teams work together to achieve accessibility
- No, I don’t have any understanding of accessibility at all
Do you think that the lessons learned from using WebAlign will become easier to use over time?
-
- Yes, the ideas in WebAlign are straightforward and the format is helpful to make sure I didn’t miss anything
- Yes, the ideas in WebAlign are useful but accessibility will always be difficult
- No, WebAlign won’t be easier to use even as I use it more
- No, WebAlign simply doesn’t make sense to me at all
How effectively do you feel that WebAlign helps achieve full WCAG compliance from different roles (including yours) at [OUR ORGANIZATION]?
-
- WebAlign does a great job ensuring that all aspects of WCAG compliance are addressed
- WebAlign does a reasonable job but some WCAG Success Criteria are not completely addressed
- WebAlign does a fair job but there are better tools.
- WebAlign does not do a good job
Do you think that WebAlign can help catch web accessibility problems earlier in the process?
-
- Yes, if managers could see team member checklists, they could spot where problems occur earlier in the process
- Maybe, but team members will likely not respond to managers asking them about problems they identified on their checklists
- Maybe, but it is unlikely managers would focus on accessibility problems and would allow them to reach production
- No, there is simply no way to avoid problems earlier in the design process
If your colleague needed to have a set of web pages be designed, built, and tested to fully meet WCAG, how likely would you be to recommend WebAlign?
-
- There is a high likelihood I would recommend WebAlign
- There is a good likelihood I would recommend WebAlign
- I may recommend WebAlign but only after recommending several other products
- I wouldn’t recommend WebAlign at all
Open-Ended Questions
Open-ended questions get a little trickier because they can provide a gateway for users to express their resistance to change. Here, setting boundaries can really help though. Again, feel free to steal and modify!
"In using WebAlign at [OUR ORGANIZATION], what other things would make it easier to do your job?"
"In rolling out WebAlign, are there specific things that [OUR ORGANIZATION] can do to further improve accessibility?"
In reading over these questions, you may notice that they each imply that the decision to use WebAlign has already been approved (although that is never stated). This makes it harder for users to express their resistance to change and suggest that WebAlign is an added burden.
Suggestions Wanted
We would love to hear about other ideas you have to help ensure that a proof of concept succeeds. If you're planning a proof of concept for WebAlign, please let us know-- both to help your proof of concept as well as our other customers. We need to all help each other. After all, we don't get no respect.
Want to See More Content Like This?
Want the latest blog posts, videos, white papers, and announcements? Sign up for our mailing list and stay in the loop!
We're Here to Help When You're Ready
Take a deep breath. Then feel free to reach out to our team when you're ready to discuss your accessibility needs.
Beautiful post Ken. Another thought to add to reasons for a proof of concept (POC): when introducing accessibility into an organization, generally the teams are willing, but have no idea how to start. A POC can operationalize the approach to accessibility. It brings a variety of knowledge, and implementing accessibility can go from daunting to achievable.
Great point, Joel! A proof of concept is great for making an abstract idea “real” to an organization.