Write Your First Automation

tip

What you will learn

• Initialize your first automation suite
• Configure your credentials (workspace id and access token)
• Understand the generated automation case
• Run your first automation
• Troubleshooting any issues
• Where to go next

Quickstart

Initialization

To create your first automation suite, enter

npx askui init

This will create

• a tsconfig.json: a json file specifying the root files and the compiler options required to compile the project,
• a folder called test which includes:
  • test/my-first-askui-test-suite.test.ts: an example test with askui,
  • a folder called helper which contains the jest.setup.ts file for setting up the test environment
  • a jest.config.ts configuration file

Configuration

Generate credentials through the user portal (usage is free!). Please see the documentation on signing up and generating credentials.

Then, go to your helper/jest.setup.ts and add the configuration for your workspace id and your access token to the UiControlClient.

 aui = await UiControlClient.build({
    credentials: {
      workspaceId: '<your workspace id>',
      token: '<your access token>',
    }
  });

Understand the Generated Code

Run

info

Before executing the automation, open test/my-first-askui-test-suite.test.ts on your main display. The code in this file is shown below. It contains the text on line 12 for which the AI will search.

Some users have reported instability running automation on macOS with external displays and/or virtual desktops (called Spaces). If you experience similar issues, please disconnect external displays and close virtual desktops, or see documentation on running automation in Docker.

test/my-first-askui-test-suite.test.ts

import { aui } from './helper/jest.setup';

describe('jest with askui', () => {
  it('should click on text', async () => {
    // Run this to see what askui annotates
    await aui.annotateInteractively();

    await aui.moveMouse(0, 0).exec();
    await aui
      .click()
      .text()
      .withText('Click on this text right here!')
      .exec();
  });
});

To execute the automation suite, enter

npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts 

You should now see the automation suite being executed inside the shell and the following things happening on your main display:

1. Line 6: Show an interactively annotated version of your main display with red bounding boxes around the annotated elements. When you press ESC on your keyboard the automation will resume.

What is Interactive Annotation?

The interactive annotation command requests the askui server to take a screenshot of the specified screen. Then, an AI model is used to annotate the image. After that, a full-screen window appears. Inside this window, boundary boxes enclosing the UI elements detected are going to appear. You can hover over the boxes to see the boxes' specifications, e.g., text detected, element type etc.

2. Line 8: Your mouse pointer moves to the upper left corner of your main display.
3. Line 12: Your mouse pointer comes back to click on the text Click on this text right here! in your automation file.

This is what it should look like on your display:

🎉 Congratulations! You just executed your first automation suite using askui.

Troubleshooting

For problems with the execution, take a look at our Troubleshooting page

You will find the following pages there:

• Linux Wayland window manager and libfuse2
• macOS missing permissions for UiController

Where to Go Next?

Writing powerful UI automations with askui needs some practice. We recommend you head over to the Selecting UI Elements page to learn more about how to select elements based on visual properties.

If you are looking for inspiration, the Tutorials section will provide you with food for thought. Why not Search for a Cat Image as a start?

Also our Discord-Community is there to help you out!