CI pipeline examples

Ready-to-use pipeline configurations for running AskUI tests in GitHub Actions, GitLab CI, and Azure DevOps.

These examples show how to run askui run in the most common CI environments. All examples assume:

The CLI is on the runner: AskUI Desktop is installed there, see AskUI in CI.
ASKUI_WORKSPACE_ID and ASKUI_TOKEN are stored as CI secrets.
Your test project is checked into the repository, with the device profile the run uses in its devices.json.

Each example writes JUnit XML next to the HTML and Markdown reports (--reporter html,md,junit), so the CI system shows pass and fail per test, and keeps the whole output folder as an artifact, also when tests fail.

AgentOS on Windows CI runners

Desktop automation on Windows requires AgentOS running as a service. Install it via the AskUI Desktop installer on your runner before running the pipeline. See CI/CD integration for setup details.

GitHub Actions

# .github/workflows/askui.yml
name: AskUI Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: [self-hosted, windows, askui]  # a runner with AskUI installed
    permissions:
      contents: read
      checks: write

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Run AskUI tests
        env:
          ASKUI_WORKSPACE_ID: ${{ secrets.ASKUI_WORKSPACE_ID }}
          ASKUI_TOKEN: ${{ secrets.ASKUI_TOKEN }}
        run: askui run --plan smoke --output run-artifacts --reporter html,md,junit

      - name: Upload run reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: askui-reports
          path: run-artifacts/

      - name: Publish test report
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: AskUI tests
          path: run-artifacts/junit.xml
          reporter: java-junit

GitLab CI

# .gitlab-ci.yml
askui-tests:
  stage: test
  tags:
    - windows  # use a self-hosted runner with AskUI installed
  variables:
    ASKUI_WORKSPACE_ID: $ASKUI_WORKSPACE_ID
    ASKUI_TOKEN: $ASKUI_TOKEN
  script:
    - askui run --plan smoke --output run-artifacts --reporter html,md,junit
  artifacts:
    when: always
    paths:
      - run-artifacts/
    reports:
      junit: run-artifacts/junit.xml
    expire_in: 7 days

Azure DevOps

# azure-pipelines.yml
trigger:
  - main

pool:
  name: askui  # a self-hosted agent pool with AskUI installed

steps:
  - checkout: self

  - script: askui run --plan smoke --output run-artifacts --reporter html,md,junit
    displayName: Run AskUI tests
    env:
      ASKUI_WORKSPACE_ID: $(ASKUI_WORKSPACE_ID)
      ASKUI_TOKEN: $(ASKUI_TOKEN)

  - task: PublishTestResults@2
    condition: always()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: run-artifacts/junit.xml
      testRunTitle: AskUI tests

  - task: PublishBuildArtifacts@1
    condition: always()
    inputs:
      pathToPublish: run-artifacts/
      artifactName: askui-reports

Reading the exit code

Every example fails the job on a non-zero exit code. The code tells you who has to act: 1 is a failed or broken test, 3 is a runner or project problem (credentials, device, MCP server, an unset ${VAR}), 4 means no test ran. All codes: Exit codes.

Connecting to a remote AgentOS

If your CI runner is not the machine being automated (e.g. a Windows runner automating a separate test VM), the device profile points at the remote AgentOS. Leave the host to the pipeline with a variable in the profile's host field:

devices.json (excerpt)

{
  "id": "win-vm",
  "name": "Windows test VM",
  "type": "agent-os",
  "host": "${AGENTOS_HOST}",
  "port": 26000
}

export AGENTOS_HOST=<vm-ip>
askui run --device win-vm

A remote host is attached to; the CLI starts a local AgentOS only for a profile on localhost. An unset AGENTOS_HOST stops the run with exit code 3. See Remote Windows VM for setting up the VM.

Storing credentials

Never hardcode ASKUI_WORKSPACE_ID or ASKUI_TOKEN in your pipeline files. Use your CI platform's secret store:

Platform	Where to add secrets
GitHub Actions	Repository Settings → Secrets and variables → Actions
GitLab CI	Settings → CI/CD → Variables
Azure DevOps	Pipeline → Edit → Variables

The project's secrets go into the same store: in a headless run, a secret named QA_PASSWORD resolves from the variable QA_PASSWORD.

Create a dedicated access token per pipeline so you can rotate or revoke individual pipelines without affecting others. See Workspaces.