Run report

Two written reports per run, the test report per case and the run-level summary. Where they live and how to open them: Runs.

Test report

Each test gets its own <test_name>_report.md (with a step_<n>.png screenshot for every step), the agent writes it while it executes. It is the primary record of what the agent did, what it expected, and what it actually observed.

# Test Case Report: login_test

**Test Case ID:** login_test
**Date:** 2025-11-12
**Status:** WARN

## Summary

The agent logged in using the QA credentials and verified the dashboard was
shown — but sign-in was unusually slow (about 15 seconds).

## Preconditions

1. Application is running and the login page is visible.

## Test Steps

1. **Navigate to the login page** — Status: **PASSED**
   - **Agent Interpretation:** Opened browser and navigated to the login URL.
   - **Expected:** Login form visible.
   - **Actual:** Login form visible with username and password fields.
   - ![Step 1](step_1.png)

2. **Enter QA credentials and submit** — Status: **WARN**
   - **Agent Interpretation:** Filled the form using the QA credential secrets.
   - **Expected:** Dashboard page shown after successful sign-in.
   - **Actual:** Dashboard loaded with the welcome banner — but only after
     roughly 15 seconds.
   - ![Step 2](step_2.png)

## Postconditions

1. User is signed in and the dashboard is visible.

## Issues

Sign-in took about 15 seconds — well above the usual 2–3 seconds. The
dashboard did load completely, so step 2 is WARN rather than FAILED, but
this is worth watching before it becomes a timeout.

## Conclusion

The login flow works, with one caveat: sign-in is unusually slow.
Functionally PASSED behaviour, reported as WARN because of the response
time.

Section by section:

• Test metadata: who, when, and how it went.

  • Test Case ID: ties the report to the test case, its title ID or file name.

  • Date: when the test executed.

  • Status: the overall result, the **Status:** line is what the runner parses as the run's result. One of:

    • PASSED, actual result matches expected.
    • FAILED, the step executed but the result didn't match.
    • SKIPPED, intentionally not executed (e.g. precondition not met).
    • WARN, passed, but with a caveat (slow response, minor visual deviation).
    • BROKEN, could not execute: crash or infrastructure failure.

    The overall status is the worst of the steps: BROKEN if any step broke, else FAILED if any failed, else WARN, else SKIPPED if all skipped, otherwise PASSED.

• Summary: the outcome in two sentences, read this first.
• Preconditions: whether the assumed starting state was actually met, a test that began from the wrong state fails here, not misleadingly in a step.
• Test Steps: one entry per step.
  • Agent Interpretation: how the agent understood your instruction, check this when a step surprises you.
  • Expected / Actual: the step's contract and what was actually observed.
  • Status: the step's own status, same vocabulary as above.
  • Screenshot: the step's evidence (step_<n>.png).
• Postconditions: whether the overall pass criterion held at the end.
• Issues: anything worth a look that isn't a failure, the incident notes. In the example above: the 15-second sign-in.
• Conclusion: the agent's closing assessment of the whole case.

The report's shape is defined by your project's report format. What to do with a failing report, triage, root cause, where the fix belongs: Analyzing failures.

Summary report

summary_report.md in the run folder contains one Markdown table across all tests in the run, with status and execution time per test.

# Run Summary

| ID | Name | Status | Duration |
|---|---|---|---|
| login_test | login_test | PASSED | 1m 12s |
| settings_test | settings_test | PASSED | 0m 48s |
| checkout_test | checkout_test | FAILED | 2m 03s |

**Total: 3 · Passed: 2 · Failed: 1 · Broken: 0**

Tests that a plan selected but the run never reached are listed as BROKEN so nothing goes missing silently. The Dashboard and Runs pages in AskUI Desktop derive their pass/fail counters from this table.

In CI, upload the whole workspace folder as a build artifact so the reports and screenshots stay attached to the run even when tests fail:

- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: askui-run
    path: agent_workspace/