Skip to content
Interface language: English
Audit / Signal

AI says your site is accessible. What did it actually test?

A green AI accessibility report is not proof. Check tested states, rule coverage, inconclusive findings and manual verification.

Article language: English

Aeneia world concept: stone terraces with shallow water and a parallel dry path lead towards a bay.
In this article

The report has a green heading: “No accessibility issues found.” Beneath it is a screenshot of a page with a “Book an appointment” button. The tool never opened the form, triggered an error or checked the confirmation window. The report never captured the rest of the booking journey.

This is a hypothetical report, not the result of a scan conducted for this article. The problem isn’t that the tool found nothing. It’s that a narrow test has been given a sweeping meaning. “No issues detected in the state visited” and “the site is accessible” require different evidence.

AI can help run checks, organise results and write a report. When reviewing the work, establish which actions were actually performed, what they covered and how the final conclusion was reached. A fluent explanation doesn’t expand the scope of a test.

One URL, several different states

Imagine a service for booking bicycle repairs. A customer chooses the type of repair, an appointment and a location. Clicking a button opens a window; submitting an incomplete form brings up a message about a missing phone number. None of these steps necessarily creates a new page address. A list of URLs checked therefore doesn’t describe the whole journey.

Before reading the result, draw a short map of the states that matter to the task. For our service, that means entering the booking process, opening the window, choosing an appointment, encountering a form error and seeing the confirmation. Note how to reach each state. “Form error” without a record of what was submitted may lead to a different message when the test is repeated.

In the test record, distinguish, for example, the page on arrival, the appointment window opened with the button, and that same window showing “Choose an appointment”. Keep a reference to the evidence for each state. If the report contains only a screenshot of the initial page, ask for evidence from the others. Three entries with the same URL can be more informative here than a long list of addresses where only similar headings were checked.

Record the conditions too: device type or viewport size, whether the user is signed in, and the service option selected. A business customer’s form may contain fields absent from an individual customer’s form. A report about one version shouldn’t automatically be taken to cover the other. You don’t have to test every possible combination indefinitely; you do need to explain which ones were included and which remain to be checked.

If a tool navigates the site on its own, ask for a trace of the path it took. A list of clicks helps, but also check whether each action actually produced the intended state. A button may fail, data may still be loading or an error may prevent a window from opening. A scan run at that point will assess something other than what was planned.

Pay particular attention to views that depend on data. A calendar while it is loading and one showing available appointments may contain different elements. Establish how the tester recognises that the intended state is ready, and record what indicates readiness. A calendar with no available appointments may show different content again. If that is part of the service’s normal operation, give it a place in the scope instead of dismissing it as a case where “nothing appears”.

If part of the process requires test data or access, arrange it before testing. Being unable to enter an area is not the same as checking it successfully. Mark it in the report as “not covered — no access” and explain how the check can be completed. That is a concrete gap in scope, not an accessibility defect established in that area.

What a tool can tell you about the state it tested

W3C explains that tools cannot automatically check every aspect of accessibility and that human judgement is needed. Its guide to selecting accessibility evaluation tools describes them as an aid to evaluation. Adding AI to the process does not remove the need to show what was actually checked.

Separate the rule-based scanner from the model describing its results. The first may identify an element and a specific rule; the second writes an explanation or recommendation. If a model assesses a screenshot itself, it has access to different information from a tool that analyses the document’s structure. The report should name the method behind each finding, rather than label the whole thing simply “AI audit”.

Check that a recommendation refers to the same element as the finding. A model might explain a field-label problem correctly but attribute it to the neighbouring button. The person implementing the change needs the location, the rule and the conditions for reproducing it. An elegant explanation of why accessibility matters is no substitute for those details.

The axe-core documentation distinguishes, among other results, incomplete items that require manual review and inapplicable rules for which no matching elements were found. The definitions in the axe-core results documentation don’t support treating either category as a pass. Keep them separate in the report.

Ask for an explanation of omitted rules and areas. If an embedded calendar from an external provider was excluded, the result doesn’t cover the entire booking process. There may be a sound technical reason for the exclusion, but customers still use that calendar. Identify its owner in the summary and say how it will be checked. The same applies to elements deliberately hidden during the test to produce a quieter screen. Changing what was examined changes the scope of the conclusion.

If a model summarises the raw result, compare the two. Did an item marked “needs review” remain in the summary? Did “no matching elements” become “passed”? Was a tool execution error described as an error-free page? A change in the report’s tone can conceal exactly what a person needs to know.

Aeneia conceptual photography

The number of checks can also be used to overstate an automated result. One hundred rules run do not mean one hundred screens checked or one hundred requirements confirmed in full. Establish what the figure in the heading actually counts. If the tester cannot explain the denominator, the percentage is decoration dressed up as precision.

Distinguish the levels of the report as well. A finding may concern one button, an entire open window or a task carried across several screens. Combining such items into one number requires an explained rule. Two instances of the same label error do not always mean two different problems for a user, while one difficulty getting back to a window can block the whole journey. Set priorities by their consequences, not just by the number of markers.

What a green summary cannot confirm on its own

The presence of text beside an element does not prove that a person understands its purpose. In our form, a “Next” button might have a valid name in the code yet still fail to explain whether the next step will confirm the booking. Separate the technical finding from an assessment of what the message means in the task.

Likewise, an image can have alternative text that doesn’t serve its purpose. If it shows where to collect the bicycle, calling it “building” adds little. The page context is needed to judge it. That doesn’t mean automatically writing a long story about the image; establish what information it needs to convey to the user.

For keyboard use, test the route through opening the window, making a selection and closing it. Inspecting the initial page alone cannot confirm that the focused element stays visible later or that focus returns to the right place when the window closes. Those tasks have to be performed in the agreed configuration.

A screenshot doesn’t record the sequence of keyboard movements. In evidence from a manual test, state the actions and what happened after each one. “Press Tab three times, then Enter” is meaningful only with a starting state and a recognisable target. A different number of fields can change the order on a later run. Record the name of the element each action reached, so the test remains understandable after a small content change.

When items can be reordered, check an alternative way to do it. An interface may respond beautifully to mouse dragging while leaving other people unable to achieve the same result. For the background, see the article on alternatives to dragging (in Polish). For this report, the question is whether both methods were actually included in the test.

A technical result can also be supplemented by a properly planned study with people who use assistive technologies. It shows how one participant, drawing on their own experience, performs the task in a particular configuration. It does not automatically replace a complete conformance assessment, but it contributes evidence that a green scan heading cannot provide. The article on research with assistive technology users (in Polish) explores related considerations.

Agree separately on the scope of manual testing, the criteria and the tester’s competence. The article on business website accessibility (in Polish) looks more closely at assessing the site’s functions. Here, we are judging the credibility of a tool’s report: whether its conclusion has an adequate basis and whether unresolved questions are visible.

From a finding to a fix you can verify

Describe a defect through its conditions and effect. “After opening the booking window, I can’t reach the appointment selector with the keyboard” gives someone enough to prepare a test of the fix. “Navigation problem” is too broad on its own. If the finding comes from a specific tool rule, retain its identifier and the location of the element in the version tested.

Include the versions of the site and tool, along with the configuration, in the report. If the form changes the next day, the earlier result remains evidence about the earlier state. Don’t automatically attribute it to the new release. Equally, don’t combine results from different interface variants under one date without explaining what each covers.

Check the fix in the same task that exposed the problem, then inspect the related steps. Adding a close button may resolve one difficulty while leaving the return to the form unclear. Acceptance should show the effect for the person performing the task, rather than stop at an item disappearing from the report.

When comparing reports from before and after a fix, check that their scopes match. If the second scan never opened the window, the disappearance of its issues does not confirm a repair. A similar problem arises when a loading error displays a simplified fallback page. First confirm that the correct screen and state were tested; only then compare the rule results. This helps prevent a drop in alerts from masking a reduction in the elements actually checked.

If a model proposes a code snippet, distinguish the proposal from a change that has been implemented and tested. A revised label may fit the example but not the button’s current function. Fixes still go through the normal acceptance process. An acceptance checklist helps here, particularly when connecting a fix to the actual task journey. For a closer look, see the article on website acceptance (in Polish).

When AI evaluates its own recommendation, check the basis of that second assessment. “The change ensured compliance” is not a record of a repeat test. The article on an AI model acting as a judge (in Polish) examines the independence of that kind of scoring. In this case, return to the specific state of the page and the evidence of what was checked.

A summary can retain three honest lists: confirmed defects, properties checked within a stated scope, and tasks requiring further work. Assign an owner and a way to close each gap. A “further review” status should lead to a specific task, not disappear while the summary is being edited.

You can discuss the reporting process and rules for using AI with Aeneia Audit. At the close of the work, the answer should be specific: which states and properties were checked, what was found, and what remains unknown. The green heading can stay, provided it stops saying more than the document beneath it.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear