Test results and reports

A run records three things: the result of each test, an execution report per test, and the run-level summary report. Where they live and how to open them: Runs.

Test result

The agent ends each test by recording its result: one status and a short reason, for example FAILED, "Expected the dashboard, saw an error banner.". The recorded result decides whether the test passed. The summary report, the Dashboard, and the Management Report all count it.

The agent picks the status by AskUI's status definitions. The status is one of:

• PASSED: the actual result matches the expected one.
• FAILED: the test executed, but the result didn't match.
• WARNING: the expected outcome was observed, but something noteworthy went wrong on the way (slow response, minor visual deviation).
• SKIPPED: not executed: the agent skipped it on purpose (e.g. a precondition was not met), or the test never started (see Tests that never started).
• BROKEN: the test ran and could not finish: a crash or an infrastructure failure.

The reason names what was expected and what was observed. A test that ends without a recorded result is BROKEN, with the reason "No result recorded". A test that stops on an error is BROKEN, with the error as the reason. If the agent records more than once, the last result counts. The result is saved next to the test as <test_name>_result.json.

Tests that never started

A test that never started is SKIPPED, never BROKEN. Its note in the summary report names the reason:

A skipped test counts as nothing, so it changes neither the pass rate nor the totals. The run still fails, because it did not reach every test it was started for (when a run passes). For runs from older versions, the app shows a test that the summary listed as broken without a duration as SKIPPED.

Execution report

The execution report is the written record of one test: what the agent did, what it expected, and what it observed, with screenshots. The agent that ran the test does not write it. A separate agent writes it after the test ends: it reads the test's conversation and screenshots and follows your project's report format. The next test starts at once, so a report never slows a run down.

Whether a test gets a report, and which model writes it, you set under Extending → Reporting. A test without a report can get one later from the Runs page: Write a report later.

# Test Case Report: login_test

**Test Case ID:** login_test
**Date:** 2025-11-12
**Status:** WARNING · **Cost of this test:** $0.33 ($0.41 with reporting)

## Summary

The agent logged in using the QA credentials and verified the dashboard was
shown — but sign-in was unusually slow (about 15 seconds).

## Preconditions

1. Application is running and the login page is visible.

## Test Steps

1. **Navigate to the login page** — Status: **PASSED**
   - **Agent Interpretation:** Opened browser and navigated to the login URL.
   - **Expected:** Login form visible.
   - **Actual:** Login form visible with username and password fields.
   - ![Step 1](screenshot_2.png)

2. **Enter QA credentials and submit** — Status: **WARNING**
   - **Agent Interpretation:** Filled the form using the QA credential secrets.
   - **Expected:** Dashboard page shown after successful sign-in.
   - **Actual:** Dashboard loaded with the welcome banner — but only after
     roughly 15 seconds.
   - ![Step 2](screenshot_6.png)

## Postconditions

1. User is signed in and the dashboard is visible.

## Issues

Sign-in took about 15 seconds — well above the usual 2–3 seconds. The
dashboard did load completely, so step 2 is WARNING rather than FAILED, but
this is worth watching before it becomes a timeout.

## Conclusion

The login flow works, with one caveat: sign-in is unusually slow.
Functionally PASSED behaviour, reported as WARNING because of the response
time.

Section by section:

• Test metadata: who, when, and how it went.
  • Test Case ID: ties the report to the test case, its title ID or file name.
  • Date: when the test executed.
  • Status: the result the test recorded. The report states it as given and cannot change it.
  • Cost of this test: right behind the status. The test's execution cost comes first, and the total including this report follows in parentheses, for example $0.33 ($0.41 with reporting), in whole cents. When only the report's cost is known, it reads $0.08 for the report. Without a known price, the cost is left out. The cost also adds to the run's total, which the summary report in the app shows as Cost of this run:.
• Summary: the outcome in two sentences, read this first.
• Preconditions: whether the assumed starting state was actually met, a test that began from the wrong state fails here, not misleadingly in a step.
• Test Steps: one entry per step.
  • Agent Interpretation: how the agent understood your instruction, check this when a step surprises you.
  • Expected / Actual: the step's contract and what was actually observed.
  • Status: the step's own status: PASSED, FAILED, SKIPPED, WARNING, or BROKEN.
  • Screenshot: the step's evidence, exactly one per step: the screenshot that shows the step's outcome. A second one appears only as a rare exception, for example to compare before and after. This holds whatever your report format says. The writer picks the screenshot from those the agent took during the test, so the numbers in the file names (screenshot_<n>.png) need not run in order.
• Postconditions: whether the overall pass criterion held at the end.
• Issues: anything worth a look that isn't a failure, the incident notes. In the example above: the 15-second sign-in.
• Conclusion: the closing assessment of the whole case.

The report's shape is defined by your project's report format. To share a report with someone without the app, export it as PDF, JSON, or JUnit XML: Export. What to do with a failing report, triage, root cause, where the fix belongs: Analyzing failures.

Files of a test

Each test has its own folder in the run folder:

Summary report

summary_report.md in the run folder summarizes the whole run: a totals head (test-case count, per-status counts, run duration), one Markdown table across all tests with status and execution time per test, and detail sections that quote the recorded reason of each warned, failed, broken, or skipped test. Broken tests are listed under 💥 Broken, tests that never started under ⏭️ Skipped.

# Summary Report

**Date:** 2026-08-06

**Total:** 3 Test Cases
- **✅ Passed:** 2
- **❌ Failed:** 1
- **⏱️ Duration:** 00:04:03

| Test Case ID | Test Case Name | Status | Duration |
| --- | --- | --- | --- |
| login_test | login_test | ✅ Passed | 00:01:12 |
| settings_test | settings_test | ✅ Passed | 00:00:48 |
| checkout_test | checkout_test | ❌ Failed | 00:02:03 |

# ❌ Fails

## checkout_test
The pay button never appeared after entering the voucher code.

The summary comes from the recorded results, not from the execution reports, so it is complete even for tests without a report. A test that ran but recorded no result reads "No result recorded". Tests that a test plan selected but the run never started are listed under ⏭️ Skipped with their reason, so nothing goes missing silently. The summary report pane exports the run as PDF, JSON, or JUnit XML: Export.

In CI, upload the whole workspace folder as a build artifact so the reports and screenshots stay attached to the run even when tests fail:

- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: askui-run
    path: agent_workspace/