Reporting

The Reporting tab decides how your tests are reported. Open Extending → Reporting. It has three settings:

• Create report: which tests get an execution report.
• Reporting model: which model writes the reports.
• Report format: the template every report follows (utils/format.md).

The first two are saved in utils/reporting.json in your project. Both files travel with the project, so a colleague's clone uses the same settings.

How a test is reported

The agent ends each test by recording a result: a status and a short reason. It picks the status by AskUI's status definitions. That recorded result decides whether the test passed. A test that records nothing is BROKEN.

The execution report is written after the test, by a separate agent. It reads the test's conversation and screenshots and writes the report in the shape of utils/format.md. The next test starts at once, and the report is written in the background. A report never changes the result of a test. What a report contains: Execution report.

Create report

A test plan can override this setting for its own runs. The order is: the plan's setting, else the project's, else Always.

Every report costs model tokens. With Only for failed or broken tests or Never, you can still write a report later. Right-click the test on the Runs page and pick Generate execution report: Write a report later.

Reporting model

The model that writes the reports.

• On the AskUI hub, pick from the list:
  • Same as your test runs: the default, and the recommended choice. A report covers one test, so the model of your runs is usually enough.
  • Best available: for more careful reports.
  • A named hosted model: any other model the hub serves.
• With your own model provider, the Same as for my test runs checkbox is on by default. Clear it to set up a separate provider for reports. The model must read screenshots.

How to connect your own provider: Model providers.

Status definitions

Every project uses AskUI's status definitions. They define when a test is PASSED, FAILED, WARNING, BROKEN, or SKIPPED. The testing agent gets them in its instructions and picks the status it records by them. Setups and teardowns use them too.

The first run of a project writes them to utils/status.md, and the agent reads them from there. The app does not offer the file for editing:

- **PASSED** — every step ran and the expected outcome was observed.
- **FAILED** — the system did not behave as the test expects: an expected outcome was not observed, or a step is not applicable to what the screen shows.
- **WARNING** — the expected outcome was observed, but something noteworthy went wrong on the way (e.g. a slow response, a minor visual deviation).
- **BROKEN** — the test could not be executed because of an error that is not the system under test (crash, infrastructure or tool failure, exception).
- **SKIPPED** — the test was intentionally not executed (e.g. its precondition was not met).

Because every project uses the same definitions, a PASSED means the same thing in every project. The summary report, the Dashboard, and the Management Report count these five statuses.

Report format

The Report format section is collapsed below the other settings. Click it to open the editor for utils/format.md. Changes save automatically, and the file applies to every report of the project.

If the project has no utils/format.md yet, click Create utils/format.md. It starts from the default format.

The format instructs the report writer, not the testing agent. Keep these parts:

• The step statuses (PASSED, FAILED, SKIPPED, WARNING, BROKEN): the default format asks the writer to judge each step like the status definitions.
• The overall status: the report states the status that the test recorded. The format cannot change the result of a test.
• The screenshot syntax: the writer gets the test's screenshots with file names such as screenshot_3.png. It embeds each one as ![Step 1](screenshot_3.png), so the image shows in the report.

The writer embeds exactly one screenshot per step, the one that shows the step's outcome, and a second only as a rare exception, for example a before-and-after comparison. This holds whatever the format says, and the default format says the same. The sections, the wording, and the level of evidence are yours to tailor: the report contract.

The settings file

utils/reporting.json holds the first two settings:

{
  "createReport": "failures",
  "modelId": "same-as-runs",
  "byomSameAsRuns": true
}

createReport sets when an execution report is written. A test plan can override it.

• always: after every test.
• failures: only for tests that ended FAILED or BROKEN.
• never: none during the run. You can still write them later from the Runs page.

modelId sets which hosted AskUI model writes the reports when your runs use AskUI's models. It has no effect when your runs use your own model (ModelBridge or AirGap).

• same-as-runs: the model the run used. This is the default.
• best: the best available hosted model.
• A hosted model id: that model.

byomSameAsRuns applies only to runs on your own model.

• true: reports use the same provider and model as the run. This is the default.
• false: reports use the separate provider you set up on the Reporting tab. That provider is stored in the app on your machine, not in the project, so the project file only records that it is separate.

A project without this file uses Always and the model of your runs.