Manual vs. Automated Accessibility Testing: Why You Need Both

Organizations preparing for a Section 508 evaluation often ask whether automated scanning tools are sufficient for compliance. The short answer is no. Automated tools are useful, but they cover only a portion of the requirements. Full compliance validation requires manual testing by a trained evaluator.

Understanding what each approach covers, and where each falls short, is essential for planning an effective accessibility strategy.

What Automated Testing Covers

Automated accessibility scanning tools analyze web pages and documents against a set of programmatic rules. They are effective at identifying issues that can be detected through code analysis, such as:

• Missing alt attributes on images
• Insufficient color contrast ratios
• Missing form labels
• Broken or invalid ARIA attributes
• Heading hierarchy errors
• Missing document language declarations
• Duplicate element IDs

These tools can scan hundreds of pages in minutes, making them efficient for identifying surface-level issues across large digital properties. They are particularly useful as a first pass to establish a baseline understanding of an application's accessibility posture.

The DHS Trusted Tester process includes 63 test criteria. Automated tools can reliably address approximately 42 of those 63 criteria. That is a meaningful portion, but it leaves 21 criteria untested.

What Automated Testing Cannot Do

The remaining 21 criteria require human judgment. Automated tools cannot evaluate:

Keyboard navigation logic. An automated tool can verify that interactive elements are focusable, but it cannot determine whether the tab order is logical, whether focus is managed correctly in modal dialogs, or whether keyboard traps exist in complex interfaces.

Screen reader experience. Automated tools do not use screen readers. They cannot evaluate whether announcements are meaningful, whether dynamic content updates are communicated, or whether the reading order matches the visual presentation.

Meaningful alt text quality. A tool can detect that an alt attribute exists, but it cannot determine whether the text is actually descriptive. An image with alt="image" will pass an automated check but fails the accessibility requirement.

Complex interaction patterns. Drag-and-drop interfaces, custom widgets, tabbed panels, and other non-standard components require manual evaluation to verify that they are operable and understandable for users with disabilities.

Cognitive load assessment. Automated tools cannot evaluate whether content is written clearly, whether error messages are helpful, or whether the interface is confusing for users with cognitive disabilities.

The DHS Trusted Tester Methodology

The Department of Homeland Security developed the Trusted Tester process as the standard methodology for evaluating Section 508 compliance. Certified Trusted Testers are trained to conduct manual evaluations against the full set of 63 test criteria using a defined, repeatable process.

A Trusted Tester evaluation includes everything automated tools cover plus the 21 manual criteria that require human judgment. The result is a comprehensive assessment that can withstand scrutiny from contracting officers, prime contractors, and federal accessibility coordinators.

The Trusted Tester process produces an evidence-based report. Every finding is documented with the specific WCAG success criterion, a description of the issue, and the testing conditions. This level of detail is what separates a defensible accessibility audit from a surface-level scan.

When to Use Each Approach

Use automated scanning when:

• You need a quick baseline assessment before committing to a full audit
• You are monitoring a large number of pages for regressions during development
• You want to identify and fix obvious issues before an evaluator arrives
• You need to triage which areas of your application have the most accessibility risk

Use manual Trusted Tester evaluation when:

• You need a compliance determination for a federal deliverable
• You are preparing an Accessibility Conformance Report (ACR)
• A contracting officer or prime contractor has requested Section 508 evidence
• You need to validate that remediation efforts actually resolved the issues
• Your application includes complex interactions, multimedia, or custom components

In most federal contracting scenarios, both approaches are used in sequence. An automated scan with a tool like our free, open-source AccessAudit identifies the obvious issues and establishes scope. A manual audit addresses the full set of criteria and produces the documentation required for procurement.

The Risk of Relying on Automated Tools Alone

Organizations that rely exclusively on automated scanning expose themselves to significant risk. An automated scan that reports zero issues does not mean the product is compliant. It means the tool did not find issues within the 42 criteria it can evaluate.

The 21 manual criteria include some of the most impactful accessibility requirements, keyboard navigation, screen reader compatibility, and meaningful content. These are precisely the areas where users with disabilities encounter the most barriers.

A contracting officer who tests your product with a screen reader will quickly discover issues that no automated tool would flag. If your ACR is based solely on automated testing, it will not hold up under scrutiny.

A Practical Approach

For organizations subject to Section 508 requirements, a practical compliance strategy uses both approaches:

1. Start with an automated risk scan to identify surface-level issues and establish scope
2. Remediate the automated findings before engaging a manual evaluator
3. Conduct a full manual audit using DHS Trusted Tester methodology
4. Prepare the ACR based on comprehensive test results
5. Re-test after remediation to confirm issues are resolved

This approach minimizes cost by addressing the easy issues first, then investing in manual evaluation when the product is closer to compliance.

---

Simkins & Elgazar offers both AI-powered risk scanning through our free, open-source AccessAudit tool and comprehensive manual audits using DHS Trusted Tester methodology. We help organizations understand where they stand and what it takes to get to full compliance. Contact us to discuss your testing needs.