AskUI in CI

The project your team authors in AskUI Desktop runs unchanged on a build server, same testware, headless execution, and the pipeline fails exactly when a test fails.

1: Get the CLI onto the runner

The CLI ships inside AskUI Desktop, so install AskUI Desktop on the runner machine once, as on any machine (Install the CLI):

• Windows: the installer puts askui on your PATH.
• macOS: open Settings → This machine → Command line and click Add to PATH.

A runner service that was already running needs a restart to see the new PATH. On macOS, a runner service does not read your shell profile, so if askui is still not found, add ~/.local/bin to the runner's own PATH setting.

Provide credentials as environment variables from the CI secret store: ASKUI_WORKSPACE_ID and ASKUI_TOKEN (create the token in the Hub). The same goes for every value the project references: a ${VAR} in a device profile, and every secret, resolve from the environment variable of that name.

2: Give it a device

The CLI runs on the project's device profiles, picked with --device by id or name, or the plan's device. Leave the runner-specific parts to the runner: write ${AGENTOS_HOST} or ${ANDROID_SERIAL} into the profile's host or serial, and set the variable in the pipeline (see Environment variables in a profile).

A browser runs on the bare runner; use a profile with Headless on when the runner has no display. Desktop targets need a test machine with AgentOS the runner can reach. The CLI connects to a device, it does not start an emulator, a simulator, or a VM, so start it in an earlier pipeline step.

Runner requirements

AskUI Desktop carries the CLI and the browser driver. Each device kind needs more on the runner:

Every device needs outbound HTTPS to the model: the AskUI hub, or the endpoint of --model-endpoint (see Network requirements). A project with custom tools also needs NuGet access for their #:package lines: nuget.org, or the feeds of the runner's NuGet.config.

3: Run a plan

askui run --plan smoke --output ./run-artifacts --reporter html,md,junit

In a pipeline the CLI never prompts, and prints one plain line per event. Gate the pipeline on the exit code:

Archive the output folder as the build artifact, reports and screenshots included; the CLI reference has every option.

4: Publish the test report

With --reporter including junit, the run writes <output>/junit.xml, and your CI system shows pass and fail per test. FAILED is a failure, BROKEN and a test the run never reached are errors, and a WARNING passes with its notes in the test output (JUnit mapping).

jobs:
  ui-tests:
    runs-on: [self-hosted, askui]
    permissions:
      contents: read
      checks: write
    env:
      ASKUI_WORKSPACE_ID: ${{ vars.ASKUI_WORKSPACE_ID }}
      ASKUI_TOKEN: ${{ secrets.ASKUI_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - run: askui run --plan smoke --output ./run-artifacts --reporter html,md,junit
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: askui-run
          path: ./run-artifacts
      - uses: dorny/test-reporter@v1
        if: always()
        with:
          name: AskUI tests
          path: run-artifacts/junit.xml
          reporter: java-junit

askui-tests:
  stage: test
  tags:
    - askui
  script:
    - askui run --plan smoke --output run-artifacts --reporter html,md,junit
  artifacts:
    when: always
    paths:
      - run-artifacts/
    reports:
      junit: run-artifacts/junit.xml
    expire_in: 7 days

More pipelines, Azure DevOps included: CI pipeline examples.

Which team runs what in CI, and who owns the results, is a working-model question: Sharing the project.