Computer Use Agent

AskUI executes tests with a Computer Use Agent (CUA): an AI agent that operates real software through the same channel a human tester does: the screen, the mouse, the keyboard. There is no selector, no recording, no object map. You write the test as numbered steps in a Markdown file, in the same wording you would hand a colleague, and the agent carries it out.

The loop

For every step of your test the agent repeats this cycle, often several times, until the step's expected result is reached or its attempts run out:

1. Screenshot: read the screen of the device.
2. Decide: interpret your step against what is actually visible, like a tester following instructions.
3. Act: through a tool: click, type, scroll, or anything else the toolbox allows.
4. Verify: compare what happened against the step's expected result. If it matches, the step gets its status and the agent moves on; if not, it loops again (within its attempt limit). Either way it lands in the report.

One cycle in the conversation log

The same cycle as the app records it, the log is where you watch it happen:

The stage labels on the left are this page's four stages; the app itself shows the narration, the screenshots inline, and the tool-call chips with their results (plus tokens and latency per step). The cycle is the same whether the agent acts on the screen or calls a tool like the HTTP client. Only the "act" differs. Every element of the log: Runs.

Why not a script?

A script is written for a finite set of application behaviors: one path through one version of one application, in one environment. Anything outside that set fails it: a renamed button, a machine that has drifted since the test was written, a survey pop-up nobody planned for.

The agent decides from the live screen instead, so it also handles what nobody wrote down: another application, another environment, a state that has decayed. A random pop-up costs it a click, not the run. That same freedom is why two runs can take different paths.

What a CUA agent is made of

What it is made of: two parts you choose, two we provide. These are not the products you install, they are the pieces a run is assembled from:

• A CUA model: the intelligence, and the part you pay per token for. It receives the screenshot and the step text and answers with the next action, so it must be vision-capable: a text-only model can't see the screen. Either you consume it through us (billed with your AskUI workspace) or you bring your own provider and pay that vendor directly. Which model you pick drives both the run's cost and how reliably it reads your UI.
• Tools: everything the agent can do, and the boundary of what a test can possibly do. We ship most of them: the device tools (screenshot, click, type) come with the device kind, and the Tool Store adds ready-made ones like SQL, HTTP or screen zoom. You extend from there where your product needs it, your own C# custom tools, or existing MCP servers, and you decide which of them a project actually gets.
• AskUI Desktop & AgentOS: the infrastructure layer you don't have to build yourself. AgentOS gives the agent eyes and hands on a machine (Playwright covers browsers, adb covers Android), turning "click 1180, 96" into a real click on a real machine, across operating systems, remote machines, emulators and physical phones, with connection, sessions and screen capture already solved. AskUI Desktop is where you author tests, connect devices, start runs and read the results.
• The AskUI harness: the test automation harness around the model: the plumbing a test framework normally gives you, built for an agent. This is the part we provide, and it decides how disciplined a run is (in detail):
  • The loop: screenshot, decide, act, verify, repeat, as above.
  • Prompt assembly: your prompt files plus the current test's steps, the folder's rules, and the names of available secrets, stitched into the instructions the model sees.
  • The conversation: the running transcript of screenshots, decisions and tool results. It is the agent's memory, re-sent on every turn, which makes it the run's evidence and its cost, the reason tools should return little.
  • Stop conditions: two attempts per step and no creative retries, so a PASSED means the step actually passed; an infrastructure error ends the run as BROKEN instead of grinding on.
  • The run lifecycle: setup, the tests, teardown per folder, and the report written while it all happens (details).

Next: how a whole run walks setup, tests and teardown (Test automation harness), how to write steps this loop executes reliably (Writing good tests), and what it means to bring this into your environment (Enterprise fit).