Skip to content
Interface language: English
Audit / Signal

An AI agent in the browser: a click is not a completed task

Assess a browser agent against the right record, the current screen, delays and a confirmed result, rather than a click alone.

Article language: English

Sheets tied with a dark ribbon and a seal lie on a stone surface beside a pen.
In this article

The agent selected a row, clicked “Save” and reported that the task was complete. But the application changed a different company with a similar name. In the recording, everything looks smooth: the pointer reaches the button, the window closes and a green message appears. The success belongs to the wrong record.

When an agent operates a browser, distinguish the intention, the action taken and the confirmed result. A tool can perform a click correctly while the overall task still fails. An assessment therefore needs to follow the instruction from the intended object to the state visible in the target application.

Start in a controlled environment with fictional records. You can test similar names, delayed windows and uncertain outcomes without changing real customer data. An impressive recording can help, but it does not replace an agreed way to verify the task.

What exactly should change?

Suppose the agent has to fill in a field on a fictional company record. The database contains “Studio Północ” and “Studio Północ - branch”, and the list truncates both names. The instruction should identify the right object in a way that can be checked, for example by ID and name. Otherwise, the agent and the person assessing it may be working on two different tasks.

Before the test, record the initial state and the expected change. Name the field, value and scope: one record, a selected group or the whole list. Also identify what the task does not cover. A sound assessment checks both the new value and the absence of unintended changes to similar records.

Define evidence of the outcome. It might be reopening the correct record and reading the saved field. Another process might require a request status or a view of a downloaded file. The evidence needs to match the task. Closing an edit window does not by itself show that the application saved the change.

OpenAI's computer use documentation explains that the completed status of a call means the model has finished generating the request; the application still has to execute it. After execution, the application returns the current screen. This separation shows why the name of a tool status cannot by itself replace acceptance of the user's task.

In a responsible AI agent workflow (in Polish), assign these stages to roles and limits. Who identifies the object, who performs the operation and who checks the result? If every stage is simply called “the agent did it”, it becomes hard to see where information or confirmation was missing.

Change the layout, not the task

First run a simple test on a familiar screen. Then keep the same task but change the list order or window width. Does the agent still find the right record, or does it repeat a movement to the old position? Record the observation behind its decision and what it saw immediately before making the change.

Anthropic's guidance on click problems identifies possible causes including differences between screenshot and display scale and ambiguous, similar-looking elements. These are concrete failure modes. But not every wrong operation is a misplaced pointer: sometimes the object or the goal itself was identified incorrectly.

Place two similar buttons in the test, but avoid a puzzle designed only to make the tool fail. Use a layout that occurs in the application, such as “Save” for a draft and “Save and send”, or actions on adjacent rows. The expected choice should follow from the instruction and the current screen.

Introduce a delayed window: a suggestion, confirmation or application message. Check whether the next action reaches the intended element or the new layer. Keep the image from before the action and the state afterwards. This helps distinguish a misread screen from a change that appeared after the screen was observed.

Change the list after scrolling or refreshing too. Records can move from their earlier positions. Check whether the agent returns to the current state before acting, especially if the preceding step took time. An old screenshot records an earlier situation; it is not a permanent map of the buttons.

Check the browser tab and account as well. Two similar screens may belong to different projects or organisations. Before changing anything, the process should identify the context in a way the user can inspect. The page title alone may be too general when both tabs say “Customers”.

In an AI automation test environment (in Polish), create this distinction deliberately: two fictional projects with one similar record in each. Verify both the correct change in the first and the unchanged state of the second. The evidence then covers the boundary of the task, not just the new value found.

Wait for a state that supports the next decision

After a click, the application may take time to process the change. Check what the agent does while it waits. If it sees no response, does it repeat the action, move on or inspect the state? Add a delay in the controlled test and observe the whole sequence. Do not assess only the fastest path, where every window appears immediately.

Include a partial result too. A text field may have been saved while an attachment was rejected. If the task requires both, the agent should show their different states instead of choosing one general message from the first success. Record whether the user can safely finish the missing part without repeating the part already confirmed.

Aeneia conceptual photography

Add form validation: one field needs correction while the remaining entries stay in place. Check whether the agent identifies the right message and changes only the affected part. If it starts the whole form again, compare the final values with the approved proposal. Correcting one error should not quietly change the rest of the task.

Prepare a save error separately. The agent should distinguish an absent result from confirmed completion. Record what it shows the person and what state it assigns to the task. “Could not confirm” can be an honest result, but it should not become “nothing changed” in the report when the state remains unknown.

For actions with external effects, agree the approval boundary and the environment's technical capabilities beforehand. If the agent also uses files or network access, check the actual limits of its sandbox (in Polish) instead of inferring them from the environment's name. The test does not need to send a real message or change access rights. It can inspect the preparation and preview using a substituted operation that records the proposal. The assessment should reveal behaviour without creating real side effects.

In human approval of AI actions (in Polish), show the object, change and scope before approval. Consent for one record must not silently cover the whole results page. Test refusal too: does the agent stop the relevant action, and does the application avoid carrying it out by another route?

For a file download task, check the file rather than only the press of “Download”. In a controlled test, compare its name, type and contents with what was expected. Separately prepare a state in which the application generates the document later. The agent should recognise that it is waiting instead of pointing to an empty or older file as the new result.

If the task requires a preview before saving, assess what the person actually sees. Is the full object name and the field being changed shown, or only an approval button? Approval without the right context can leave an error undiscovered even though the process history contains a formal acceptance step.

When the agent receives page or document content, that content should not expand the assignment on its own. Testing that boundary needs a separate scenario with controlled material. In this assessment, at least keep the source of the instruction recognisable and the operation's scope established. Do not treat every message on the screen as new consent from the person who gave the task.

For a review with Aeneia Audit, prepare one task, a controlled application view and a definition of the final result. That makes it possible to assess the process without relying solely on the model's own declaration.

Returning after an interruption is a separate task

Interrupt the test after a click, before the agent receives confirmation. On resumption, check whether it reads the current state before retrying. If the previous operation did succeed, repeating it can produce a second effect. An unknown result calls for a way to establish the state, not an automatic restart of the whole task.

Record the object ID and the stage at which work stopped. A long account of pointer movements is not a substitute. The person taking over should know what was confirmed, what was only proposed and what could not be read. This helps them continue even if the interface looks different on return.

For a larger set of items, associate each response with the correct input. This is explored in checking AI batch results against records (in Polish). “Batch complete” does not explain which records have a confirmed change, which were rejected and which still need checking.

If accepting new code or a tool is part of the work, keep a separate decision as described in checking dependencies proposed by AI (in Polish). The agent may operate the panel smoothly and still have no basis for expanding the environment's scope. Being technically able to click does not decide whether the operation belongs to the assignment.

For every run, record the choice of object, the attempted operation and confirmation of the outcome separately. If the error appears during selection, the correction belongs in identification or the instruction. If the right record was chosen but the save was not verified, a different change is needed. One overall task result helps with reporting, but it does not replace diagnosis of the stage.

Compare the reviewer's effort too. If they must reopen every similar record after each action, the solution has shifted a substantial part of the work into verification. Record that cost alongside the scope of confirmation. The decision to keep using it can then account for the whole process, not just the time visible in a cursor recording.

For the handover, prepare a short human-readable record: object, change, confirmation and unresolved items. Ask someone who did not watch the test to identify the next step. If they cannot tell whether to retry the operation or only read the state, the summary needs work. That is a separate usability problem in the process, even when all earlier clicks can be reproduced.

In the final decision, limit approval to the tasks and conditions actually assessed. If one type of form was tested, record that scope and when to review it again after an application change. The team then knows when it can use the agreed path and when a new test is needed.

After a fix, repeat the case that exposed the problem and a few neighbouring variants. If the fix addressed identification of the right row, test a different sort order and a similar name. Record what the test confirmed. One successful recording should not become a promise that the agent will work through any change to the application.

A sound assessment ends with a statement that can be checked: this value was saved in this record, the other records within the check retained their state, and the evidence is here. The agent can then say “done” with a useful explanation. The green message on the screen is still welcome, but now it belongs to the right record.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear