8 posts tagged with "AskUI SDK"

Introduction​

This release delivers version 24.9.1 of the AskUI Suite.

New Features​

• Support for accepting self-signed certificates on server/proxy side. See docs.
• ADE commandlet to generate annotations for training purposes: AskUI-NewAnnotationsForTraining
• ADE commandlet AskUI-StartOCRTeaching
• Snipping Workflow: Annotation Mode

Fixes​

• Fixed a compatibility issue that caused a crash on older macOS versions due to the usage of C++20 functionality.

Improvements​

• Windows installer is signed with the Microsoft provided certificate.
• Android devices can now be address by their device id:
  • AskUI-StartController command docs
  • AskUI-StopControllers command docs
• Windows and Linux ARM64 support.
• Error reporting commandlet collects software and hardware information about the system.
• Controller and Snipping Tool logs can be configured through a configuration file: See docs
• Our GUID is now RFC-4122 version 4 compliant.
• Local proxy settings are applied to npm.
• macOS and Linux installers can be used in silent mode. See docs

Supported Operating Systems​

• Windows 11
  • Version: 22H2
  • Architecture(s): AMD64 and ARM64
• MacOS
  • Version: 14.x
  • Architecture(s): ARM64 (Apple Silicon)
• Linux
  • Version: Ubuntu 22.04.2
  • Architecture(s): AMD64 and ARM64

Windows Installation​

AMD64 Download link.

ARM64 Download link.

macOS Installation​

# Download installer into /tmp folder
curl -o /tmp/AskUI-Suite-24.9.1-User-Installer-MacOS-ARM64-Full.run https://files.askui.com/releases/Installer/24.9.1/AskUI-Suite-24.9.1-User-Installer-MacOS-ARM64-Full.run

# Run the installer
bash /tmp/AskUI-Suite-24.9.1-User-Installer-MacOS-ARM64-Full.run

Linux Installation​

# Download installer into /tmp folder (AMD64)
curl -o /tmp/AskUI-Suite-24.9.1-User-Installer-Linux-x64-Full.run https://files.askui.com/releases/Installer/24.9.1/AskUI-Suite-24.9.1-User-Installer-Linux-x64-Full.run

# Run the installer
bash /tmp/AskUI-Suite-24.9.1-User-Installer-Linux-x64-Full.run

# Download installer into /tmp folder (ARM64)
curl -o /tmp/AskUI-Suite-24.9.1-User-Installer-Linux-x64-Full.run https://files.askui.com/releases/Installer/24.9.1/AskUI-Suite-24.9.1-User-Installer-Linux-ARM64-Full.run

# Run the installer
bash /tmp/AskUI-Suite-24.9.1-User-Installer-Linux-ARM64-Full.run

Versions​

Components and Packages​

• AskUI Controller
  • AskUI Remote Device Controller: 0.11.0.0
  • AskUI Legacy UI Controller: 0.14.7
  • AskUI Remote Device Snipping Tool: 0.2.0.0
• AskUI Runner: 0.1.4
• AskUI Development Environment (ADE): 0.9.1.0
• AskUI Development Kit (ADK): 0.20.7
• AskUI OCR Teaching: 0.4.0

3rd Party Versions​

• Node.JS: 21.0.0
• PowerShell 7.3.12
• PxProxy: 0.9.2

Introduction​

This release delivers some improvements to the AskUI initial setup and the error reporting.

• Video recording is disabled by default now.
• We added a .gitignore file to the initial project. This prevents checking in unwanted files like reports or .env.
• If an instruction fails the error log contains the code including the line number, so you can find the failing AskUI Instruction fast.

We also release the macOS and Linux Installer which brings the AskUI Suite to macOS and Linux. It includes the ADE and the new implementation of the AskUI Controller. The ADE makes managing all the AskUI components and AskUI projects way easier and streamlined. While the new AskUI Controller improves the execution speed and operating system compatibility. A better developer experience and faster execution for you!

New Features​

Video Recording Is Disabled by Default​

Video recording is disabled by default in helpers/askui-helper.ts. Enable it by uncommenting the commented-out lines.

...
beforeEach(async () => {
  /* Uncomment to enable video recording
  await aui.startVideoRecording();
  */
});

afterEach(async () => {
  /* Uncomment to enable video recording
  await aui.stopVideoRecording();
  const video = await aui.readVideoRecording();
  await AskUIAllureStepReporter.attachVideo(video);
  */
});
...

.gitignore for Reports-Directories and Configuration Files​

When you create a new project a sensible .gitignore file will be added. So you do not check in the following files accidentally:

allure-results/
report/
xray-report/

.env
.vscode

# Dangerous (may contain secrets)
.npmrc

# Dependencies
/node_modules

# Caches
.npm
.eslintcache

# Misc
.DS_Store

# Logs
npm-debug.log*
yarn-debug.log*
yarn-error.log*

Helpful Error Message When AskUI Instruction Fails​

The error messages Jest displayed when an AskUI Workflow/Instruction failed made it hard to find the offending line of code. Especially on larger workflows, it was a challenge. Now the error message contains the exact line and also the code that failed:

AskUI Installer Update 24.6.2​

This ships the AskUI Controller and the AskUI Suite for all major desktop operating systems.

Since the support for MacOS and Linux is new there might be undiscovered issues that will be tackled step by step.

Supported Operating Systems​

• Windows 11
  • Version: 22H2
  • Architecture(s): AMD64 and ARM64
• MacOS
  • Version: 14.x
  • Architecture(s): ARM64 (Apple Silicon)
• Linux
  • Version: Ubuntu 22.04.2
  • Architecture(s): AMD64

Windows Installation​

Download link.

macOS Installation​

# Download installer into /tmp folder
curl -o /tmp/AskUI-Suite-24.6.2-User-Installer-MacOS-ARM64-Full.run https://files.askui.com/releases/Installer/24.6.2/AskUI-Suite-24.6.2-User-Installer-MacOS-ARM64-Full.run

# Run the installer
bash /tmp/AskUI-Suite-24.6.2-User-Installer-MacOS-ARM64-Full.run

Linux Installation​

# Download installer into /tmp folder
curl -o /tmp/AskUI-Suite-24.6.2-User-Installer-Linux-x64-Full.run https://files.askui.com/releases/Installer/24.6.2/AskUI-Suite-24.6.2-User-Installer-Linux-x64-Full.run

# Run the installer
bash /tmp/AskUI-Suite-24.6.2-User-Installer-Linux-x64-Full.run

New Features​

Support for macOS and Linux.

Changes​

• The status indicator of the AskUI Controller was changed to match the users expectation.
  • Stats that are rendered:
    • Workflow in Progress
      • Color: Dark blue (Logo color)
      • Description: A workflow is in progress but the AskUI Controller waits for the next action.
    • Execution
      • Color: Green
      • Description: Indicates that for instance a mouse move or typing is in progress.
    • Idle
      • Color: White
      • Description: AskUI Controller is running but no workflow is in progress.

Fixes​

• N&A

Versions​

Components and Packages​

• AskUI Controller
  • AskUI Remote Device Controller: v0.9.0.0
  • AskUI Legacy UI Controller: 0.14.3
  • AskUI Remote Device Snipping Tool: v0.1.1.0
• AskUI Runner: 0.1.4
• AskUI Development Environment (ADE): v0.9.0.0
• AskUI Development Kit (ADK): 0.20.7

3rd Party Versions​

• Node.JS: 21.0.0
• PowerShell 7.3.12

Introduction​

This release adds a new parameter intersection_area to the relational selectors above(), below(), rightOf() and leftOf(). It allows you to specify more granular which elements get detected.

New Features​

Optional parameter intersection_area for relational selectors above(), below(), rightOf() and leftOf() in AskUI TypeScript SDK:

• above() API docs
• below() API docs
• leftOf() API docs
• rightOf() API docs

Fixes​

• Drag & Drop (Mouse down, mouse move, mouse up) crashes controller.
• TypeScript ADK retries requests to AskUI Inference that need fixing by a user. Resulted in long execution times because of unsuccessful retries.
• AskUI Controller crashes when trying to implement Drag & Drop behavior with mouseToggleDown() and mouseToggleUp().

Introduction​

This release includes two features we put a lot of effort into:

• AI Element Creation Workflow (Windows only)
• Convenience method expectAllExist which takes an array of descriptions for elements and returns if they are detected on the screen.

We also introduced a breaking change for the convenience method clickText() to make our API more consistent.

Breaking Changes​

• Method signature changed for clickText(). type becomes matching. API docs

// Before
await aui.clickText({text: 'Hello World', type: 'similar'});

// Now
await aui.clickText({text: 'Hello World', matching: 'similar'});

New Features​

ADE commandlets to create and manage AI Elements:

• AskUI-NewAICustomElement
• AskUI-ShowAIElement
• AskUI-RemoveAICustomElement
• AskUI-OpenAICustomElement
• Snipping Workflow

Convenience method added to AskUI TypeScript SDK:

• expectAllExist() API docs

Introduction​

We released the AskUI Installer version 24.5.1 which introduces new commandlets for error reporting and an overlay option for the AskUI Controller.

Also our Custom Element got a new implementation under the hood, so it works more consistently when changing the resolution of the screen. This change brings better stability to workflows that will be run on different machines. The update also expands the Custom Elements API of the AskUI SDK. You can now add a stopThreshold to optimize execution time.

Changes​

• Custom Elements API has a new parameter stopThreshold.

Improvements​

• Custom Elements implementation is more robust against screen resolution changes

AskUI Installer Update 24.5.1​

• New commandlets to manage error reporting.
• Status Indicator as an overlay of active screen.

Download link.

New Features​

ADE commandlets:

• AskUI-BuildErrorReport
• AskUI-CompressErrorReport
• AskUI-NewErrorReport

AskUI Controller:

• Overlay option

ADK:

• Custom Elements API

Fixes​

• Drag & Drop (Mouse down, mouse move, mouse up) crashes controller

Versions​

Components and Packages​

• AskUI Controller
  • AskUI Remote Device Controller: v0.3.0.0 Preview
  • AskUI Legacy UI Controller: 0.14.1
  • AskUI Remote Device Snipping Tool: v0.1.0.0 Preview
• AskUI Runner: 0.1.4
• AskUI Development Environment (ADE): Preview 3
• AskUI Development Kit (ADK): 0.18.0

3rd Party Versions​

• Node.JS: 21.0.0
• PowerShell 7.3.8

Introduction​

We had a bunch of releases today which will help you be more effective and also to integrate AskUI better in your current toolchain.

Our docs lacked a comprehensive source of common challenges about detecting elements and how to address them. Thus we created a Recommended Practices page and a separate Custom Elements page. Let us know if something is missing!

We were also able to release more Convenience methods in this version. They are more expressive and also shorter to write. Read about them below and also how to migrate to the new syntax for clickText() and clickButton().

Speaking of convenience: We also added an ESLint rule that checks for the correct usage of expect().

Last but not least we fixed a bug in our test environment implementation @askui/jest-allure-circus. It is now compatible with jest-cucumber. (Link to accompanying blog).

Breaking Changes​

• Method signature changed for clickText(). API docs

// Before
await aui.clickText('Hello World');

// Now
await aui.clickText({text: 'Hello World', type: 'similar'});

• Method signature changed for clickButton(). API docs

// Before
await aui.click().button().withText('Submit').exec();

// Now
await aui.clickButton({label: 'Submit'});

New Features​

New Convenience methods added to AskUI TypeScript SDK:

• clickCheckbox()
• clickSwitch()
• typeIntoTextfield()

We expanded our npm package @askui/eslint-plugin-askui with a new rule that checks for the correct usage of expect()

• Upgrade the plugin to 1.3.0
• Add the following rule: "@askui/askui/expect": "error"
• Check the README for a complete guide

Fixes​

• @askui/jest-allure-circus accepts the testEnvironmentOptions addCodeInReport now
  • Upgrade the dependency to version 1.0.23

// Usage
const config: Config.InitialOptions = {
  ...
  testEnvironment: '@askui/jest-allure-circus',
  testEnvironmentOptions: {
    addCodeInReport: false
  },
};
...

Introduction​

This week we introduced a new convenience method clickTextfieldNearestTo(label: string) which lets you click on a textfield with a label instead of a placeholder.

We also removed the convenience method dragTo as it was not working as intended.

There is a new option -ad --automations-description for initialising an AskUI project with npx askui@latest init. It lets you specify the name of the automations directory which was hardcoded to askui_example before.

Convenience Methods​

We had to remove dragTo() as it was not working as expected

We introduced the following method:

clickTextfieldNearestTo(label: string)​

Searches for an element of type textfield with a specific label nearest to it. If found, clicks it.

await aui.clickTextfieldNearestTo('E-Mail Address');

New Option -ad --automations-description for init​

When you run npx askui@latest init you always got a directory askui_example. This broke backwards compatibility with existing installations of AskUI.

Thus we introduced a new option when running askui init:

-ad, --automations-directory <value>  a name for the directory askui stores its configuration and workflows in. (default: "askui_example")

You can use it like this to store the AskUI automations in a directory askui-Workflows:

npx askui@latest init -ad askui-Workflows

Introduction​

This week we improved on our docs. A guide for setting up your terminal and extensions in Visual Studio Code on Windows will improve your experience while developing AskUI Workflows.

We also fixed a display bug at the Supported Keys page.

In version 0.14.0 of the AskUI SDK we introduced some convenience methods to free users from writing repetitive code. This speeds up the development of AskUI Workflows and improves readability. Make sure to read the convenience docs which give you a detailed description on how to use each method.

Visual Studio Code Setup on Windows​

Visual Studio Code is our recommended editor for developing AskUI workflows. Our docs for the AskUI Development Environment now have a section that helps you to setup Visual Studio Code for developing AskUI workflows efficiently:

• Terminal configuration
• Recommended extensions for Jest Runner, ESLint & Live View

Convenience Methods​

We introduced the following methods:

clickButton(label: string)​

Searches for an element of type button with a label and clicks it when found.

await aui.clickButton('Sign Up');

clickIcon(description: string)​

Clicks an icon based on a description.

await aui.clickIcon('Logo looking like an apple');

clickText(text: string)​

Searches for a text element and clicks it when found.

await aui.clickText('A big bear');

clickTexts(texts: string[])​

Searches for text elements and clicks them one after another when found.

await aui.clickTexts(['A big bear', 'Brown Fox', 'Walks into a bar']);

clickTextfield(placeholder: string)​

Searches for an element of type textfield with a specific placeholder text. If found, clicks it.

await aui.clickTextfield('E-Mail Address');

dragTo(element1: Executable, element2: Executable)​

Drags element1 to element2.

Both must be a moveMouse() or moveMouseTo() instruction as in the example below.

await aui.dragTo(
   aui.moveMouseTo().text('AskUI'),
   aui.moveMouseTo().text('UI Automation')
);

pressKeyNTimes(key: PC_AND_MODIFIER_KEY, times = 2)​

Press a key multiple times. At least two times.

await aui.pressKeyNTimes('right');
await aui.pressKeyNTimes('left', 3);

pressKeys(keys: PC_AND_MODIFIER_KEY[])​

Press an array of keys one after another.

await aui.pressKeys(['right', 'left', 'enter']);

waitUntil(AskUICommand: Executable, maxTry = 5, waitTime = 2000)​

Wait until an AskUICommand does not fail.

Use it to wait for an element to appear like this:

await aui.waitUntil(aui.expect().text('GitHub').exists());