Skip to main content

2024.04.29 | AskUI SDK 0.18.0

· 2 min read
Johannes Dienst
Developer Advocate

Introduction

We had a bunch of releases today which will help you be more effective and also to integrate AskUI better in your current toolchain.

Our docs lacked a comprehensive source of common challenges about detecting elements and how to address them. Thus we created a Recommended Practices page and a separate Custom Elements page. Let us know if something is missing!

We were also able to release more Convenience methods in this version. They are more expressive and also shorter to write. Read about them below and also how to migrate to the new syntax for clickText() and clickButton().

Speaking of convenience: We also added an ESLint rule that checks for the correct usage of expect().

Last but not least we fixed a bug in our test environment implementation @askui/jest-allure-circus. It is now compatible with jest-cucumber. (Link to accompanying blog).


Breaking Changes

  • Method signature changed for clickText(). API docs
// Before
await aui.clickText('Hello World');

// Now
await aui.clickText({text: 'Hello World', type: 'similar'});
  • Method signature changed for clickButton(). API docs

// Before
await aui.click().button().withText('Submit').exec();

// Now
await aui.clickButton({label: 'Submit'});

New Features

New Convenience methods added to AskUI TypeScript SDK:

We expanded our npm package @askui/eslint-plugin-askui with a new rule that checks for the correct usage of expect()


Fixes

  • @askui/jest-allure-circus accepts the testEnvironmentOptions addCodeInReport now
    • Upgrade the dependency to version 1.0.23
// Usage
const config: Config.InitialOptions = {
...
testEnvironment: '@askui/jest-allure-circus',
testEnvironmentOptions: {
addCodeInReport: false
},
};
...

2024.04.22 | AskUI Suite 24.4.1 - Inference Speedup

· 2 min read
Johannes Dienst
Developer Advocate

Introduction

We released the AskUI Installer version 24.4.1 which introduces new commandlets to start and stop the AskUI Controller, debug AskUI projects and processes.

Removed the icon classifier from the default inference. You can still target icons with icon().withText('<icon_name>') so the change is backwards compatible. But the name will not be displayed anymore. This discourages using the names of the icons as they are flaky across runs generally and often wrong.

Lastly we optimized our inference backend. This leads to faster workflow execution as it saves 300 - 700 milliseconds each time an instruction in an AskUI workflow invokes inference.


Changes

  • Removed the icon classifier from the default inference.

Improvements

  • Inference is faster 300-700 milliseconds

AskUI Installer Update 24.4.1

Introduces a number of new commandlets to manage your AskUI Installation: Download link.


New Features

This release focuses on new ADE commandlets.


Fixes

This release focuses on bug fixes.

  • High CPU usage of the RemoteDeviceController process, especially during idle.
  • All FFMPG processes are getting killed on AskUI Controller shutdown.
  • scroll() crashes AskUI Controller

Versions

Components and Packages

  • AskUI Controller
    • AskUI Controller: v0.2.1.0 Preview
    • AskUI Legacy UI Controller: 0.14.0
  • AskUI Runner: 0.1.4
  • AskUI Development Environment (ADE)
  • AskUI Development Kit (ADK)

3rd Party Versions

  • Node.JS: 21.0.0
  • PowerShell 7.3.8

2024.03.25 | AskUI SDK Automations Directory Option

· One min read
Johannes Dienst
Developer Advocate

Introduction

This week we introduced a new convenience method clickTextfieldNearestTo(label: string) which lets you click on a textfield with a label instead of a placeholder.

We also removed the convenience method dragTo as it was not working as intended.

There is a new option -ad --automations-description for initialising an AskUI project with npx askui@latest init. It lets you specify the name of the automations directory which was hardcoded to askui_example before.

Convenience Methods

We had to remove dragTo() as it was not working as expected


We introduced the following method:

clickTextfieldNearestTo(label: string)

Searches for an element of type textfield with a specific label nearest to it. If found, clicks it.

await aui.clickTextfieldNearestTo('E-Mail Address');

New Option -ad --automations-description for init

When you run npx askui@latest init you always got a directory askui_example. This broke backwards compatibility with existing installations of AskUI.

Thus we introduced a new option when running askui init:

-ad, --automations-directory <value>  a name for the directory askui stores its configuration and workflows in. (default: "askui_example")

You can use it like this to store the AskUI automations in a directory askui-Workflows:

npx askui@latest init -ad askui-Workflows

2024.03.04 | AskUI SDK Convenience Methods

· 2 min read
Johannes Dienst
Developer Advocate

Introduction

This week we improved on our docs. A guide for setting up your terminal and extensions in Visual Studio Code on Windows will improve your experience while developing AskUI Workflows.

We also fixed a display bug at the Supported Keys page.

In version 0.14.0 of the AskUI SDK we introduced some convenience methods to free users from writing repetitive code. This speeds up the development of AskUI Workflows and improves readability. Make sure to read the convenience docs which give you a detailed description on how to use each method.

Visual Studio Code Setup on Windows

Visual Studio Code is our recommended editor for developing AskUI workflows. Our docs for the AskUI Development Environment now have a section that helps you to setup Visual Studio Code for developing AskUI workflows efficiently:

  • Terminal configuration
  • Recommended extensions for Jest Runner, ESLint & Live View

Convenience Methods

We introduced the following methods:


clickButton(label: string)

Searches for an element of type button with a label and clicks it when found.

await aui.clickButton('Sign Up');

clickIcon(description: string)

Clicks an icon based on a description.

await aui.clickIcon('Logo looking like an apple');

clickText(text: string)

Searches for a text element and clicks it when found.

await aui.clickText('A big bear');

clickTexts(texts: string[])

Searches for text elements and clicks them one after another when found.

await aui.clickTexts(['A big bear', 'Brown Fox', 'Walks into a bar']);

clickTextfield(placeholder: string)

Searches for an element of type textfield with a specific placeholder text. If found, clicks it.

await aui.clickTextfield('E-Mail Address');

dragTo(element1: Executable, element2: Executable)

warning

This method is not working as intended and will be removed with the next release!

Drags element1 to element2.

Both must be a moveMouse() or moveMouseTo() instruction as in the example below.

await aui.dragTo(
aui.moveMouseTo().text('AskUI'),
aui.moveMouseTo().text('UI Automation')
);

pressKeyNTimes(key: PC_AND_MODIFIER_KEY, times = 2)

Press a key multiple times. At least two times.

await aui.pressKeyNTimes('right');
await aui.pressKeyNTimes('left', 3);

pressKeys(keys: PC_AND_MODIFIER_KEY[])

Press an array of keys one after another.

await aui.pressKeys(['right', 'left', 'enter']);

waitUntil(AskUICommand: Executable, maxTry = 5, waitTime = 2000)

Wait until an AskUICommand does not fail.

Use it to wait for an element to appear like this:

await aui.waitUntil(aui.expect().text('GitHub').exists());

2024.02.26 | AskUI Suite 24.2.2

· 2 min read
Johannes Dienst
Developer Advocate

Introduction

The main focus of the release AskUI Installer 24.2.2 was to polish the previous release 24.2.1 and therefore no major changes were introduced.

AskUI Installer Update 24.2.2

Introduces a number of new steps: Download link.

Installer for Windows

The installer contains a number of new and modified dialogues. Furthermore, it contains additional components.

More information can be found here.


New Features

This release focuses on bug fixes.


Improvements

This release focuses on bug fixes.

  • Improved the robustness of the inter-process connection between Legacy Controller and Remote Device Controller.
  • We only allow a single instance of the Legacy Controller for desktop scenarios (mobile platforms are untouched).

Fixes

This release focuses on bug fixes.

  • Fixed left and right arrow key mapping which were swapped.
  • Fixed issues with stuck mouse cursor and not finished actions.

Changes

  • Mouse movement uses the OS's cursor position set instead of delta movement on a 1000hz update rate. Making it more reliable.
  • Legacy Controller and Remote Device Controller are both shutdown with an error message if the connection is broken.

Known Issues

Mouse Movement

  • You may encounter issues when mouse pointer acceleration is enabled.
  • There might be issues with the mouse movement when it tries to move to invalid coordinates.

Versions

Components and Packages

  • AskUI Controller
    • AskUI Controller: v0.2.1.0 Preview
    • AskUI Legacy UI Controller: 0.14.0
  • AskUI Runner: 0.1.4
  • AskUI Development Environment (ADE)
  • AskUI Development Kit (ADK)

3rd Party Versions

  • Node.JS: 21.0.0
  • PowerShell 7.3.8

2024.02.07 | AskUI Suite & Installer

· 3 min read
Dominique Mader
Head of Software Engineering
Leon Meier
Head of Product

Introduction

AskUI offers a hassle-free installation experience for its products via a user-friendly installer. This installer facilitates the seamless installation of AskUI products with just a few clicks.

General

Installer for Windows

The installer contains a number of new and modified dialogues. Furthermore, it contains additional components.

More information can be found here.


New Features

We introduced a couple of new features.

In Preview


Known Issues

Installation Dependencies

  • AskUI Development Kit (former AskUI-Lib) is not included in the installer. It will be downloaded during the setup process.

Mouse Movement

  • You may encounter issues when mouse pointer acceleration is enabled.
  • There might be issues with the mouse movement when it tries to move to invalid coordinates.

Log Files

  • Log files disk space usage increased. They are stored in ~/.askui/ and can be deleted if that causes any issues.

Android

  • Recording on an Android device is not stable.
  • Recorded resolution might be lower than the device resolution.
  • Recorded video might be empty or single frame in cases no screen updates occurred during recording.
  • We don't support landscape mode on Android.
  • Devices with high DPI screen might not work as expected.
  • Video recording on MacOS is not supported.

Versions

Components and Packages

  • AskUI Controller
    • AskUI Controller: v0.2.0.0 Preview
    • AskUI Legacy UI Controller: 0.13.0
  • AskUI Runner: 0.1.4
  • AskUI Development Environment (ADE)
  • AskUI Development Kit (ADK)

3rd Party Versions

  • Node.JS: 21.0.0
  • PowerShell 7.3.8

To ensure a smooth onboarding experience, follow these steps:

  1. Download and Install:
    • Download and install the installer
    • Run the installer.
  2. Initialization:
    • Open the Command Prompt.
    • Type askui-shell.
  3. Configuration (First-Time Usage):
    • Configure AskUI settings:
      1. Type AskUI-SetSettings -WorkspaceId <askui_workspace_id> -Token <askui_token>.
      2. (Optional for Proxy Users) Type AskUI-SetSettings -HttpProxy <http_proxy_address> -HttpsProxy <https_proxy_address>.
  4. Verify Settings:
    • Type AskUI-ShowSettings to verify the configured settings.
  5. Create a New Project:
    • Create a new project by typing AskUI-NewProject.
  6. Start AskUI Controller:
    • Note: You can skip this if you selected Launch AskUI Controller at the installation wizard.
    • Start the AskUI Controller in background mode by typing AskUI-StartController -RunInBackground.
  7. Project Editing (Optional):
    • Open the project directory and starts editing the project.
info

You have to disable mouse acceleration for the AskUI Controller to work properly!

  1. Run the Project:
    • Run the project by typing AskUI-RunProject.

By following these steps, the user can efficiently set up AskUI Suite, configure essential settings, and seamlessly create and run projects.

2024.01.31 | OCR Model Enhancement

· One min read
Leon Meier
Head of Product

Introduction

We're excited to announce a speed enhancement in our OCR model, reducing processing times by 300 milliseconds.

Feature Update

OCR Model Speed Improvement

  • Faster text recognition, significantly improving workflow efficiency.
  • 1.62 sec down to 1.26 sec

Detailed Description

OCR Model Speed Enhancement

Our latest update has optimized the OCR model, achieving a notable speed increase while maintaining high accuracy.

Model: task='e2e_ocr' architecture='easy_ocr' version='1' interface='im2txtbox' use_case='faster' tags=[]
Average runtime: 1.2629041004180908
Standard deviation: 0.13027134152564313
Median: 1.2297358512878418

We are in the process of creating comprehensive documentation for our enhanced OCR model. We'd love to hear from you about what specific aspects or features you'd like to see covered in the documentation. Please share your thoughts and suggestions with us to ensure our documentation meets your needs and expectations.

Give Feedback >>>

2024.01.23

· 2 min read
Leon Meier
Head of Product

Introduction

We're introducing two significant enhancements to our platform: Single Sign-On (SSO) Integration for various identity providers and the AskUIXRayStepReporter for automation workflows in Xray for Jira. These features are designed to streamline the authentication process and improve test reporting, respectively.

Overview of Features

  1. Single Sign-On Integration

    • Documentation: SSO Setup Guide
    • This feature allows seamless integration with various identity providers including Azure AD, Google Workspace, and Okta, enhancing security and simplifying user management.
  2. AskUIXRayStepReporter

    • Documentation: AskUIXRayStepReporter Guide
    • This feature integrates with Xray for Jira to provide detailed reporting of automation workflows, including screenshots and test statuses.

Detailed Descriptions

Single Sign-On Integration

Our new SSO Integration feature supports a wide range of identity providers. It offers enhanced security, unified login experiences, and streamlined user account management, significantly reducing the overhead of managing separate login credentials.

AskUIXRayStepReporter

The AskUIXRayStepReporter is a powerful tool for reporting test automation results directly to Xray for Jira. It helps teams maintain comprehensive documentation of their test executions, complete with visual evidence and detailed status reports.


We are excited to see how these new features will enhance your experience with our platform, providing greater convenience, security, and efficiency in your automated testing and user management processes.

2023.12.13

· One min read
Leon Meier
Head of Product

Introduction

On December 13, we introduced three new features: the Visual Description Matching Method for selecting diverse visual elements, Targeted Annotations for precise debugging, and Precision Text Matching for flexible text matching.

Overview

  1. Visual Description Matching Method

    • Documentation: Matching Feature
    • This feature introduces the matching()method, allowing users to identify and interact with UI elements based solely on textual descriptions.
    • Enhances the flexibility of automated interactions with dynamic UIs.
  2. Targeted Annotations

    • Documentation: Annotate DetectedElements
    • Introduces an AnnotationRequest parameter in the annotate() method, allowing users to annotate specific elements retrieved from user interfaces by the get() method.
    • Simplifies debugging of UI element selection.
  3. Precision Text Matching

    • Documentation: Adjustable Similarity Score
    • Introduces an optional similarity_score parameter for the .withText() method.
    • Improves script resilience and minimizes mismatches in text-based automation.