Introduction​

We released the AskUI Installer version 24.4.1 which introduces new commandlets to start and stop the AskUI Controller, debug AskUI projects and processes.

Removed the icon classifier from the default inference. You can still target icons with icon().withText('<icon_name>') so the change is backwards compatible. But the name will not be displayed anymore. This discourages using the names of the icons as they are flaky across runs generally and often wrong.

Lastly we optimized our inference backend. This leads to faster workflow execution as it saves 300 - 700 milliseconds each time an instruction in an AskUI workflow invokes inference.

Changes​

• Removed the icon classifier from the default inference.

Improvements​

• Inference is faster 300-700 milliseconds

AskUI Installer Update 24.4.1​

Introduces a number of new commandlets to manage your AskUI Installation: Download link.

New Features​

This release focuses on new ADE commandlets.

• ADE commandlets to start and stop the AskUI Controller (and child processes):
  • AskUI-ShowControllers
  • AskUI-StopControllers
• AskUI-ShowSettings
  • New option: WithSecrets
• Debug Commands:
  • Project Management Debug Commands
  • Process Management Debug Commands

Fixes​

This release focuses on bug fixes.

• High CPU usage of the RemoteDeviceController process, especially during idle.
• All FFMPG processes are getting killed on AskUI Controller shutdown.
• scroll() crashes AskUI Controller

Versions​

Components and Packages​

• AskUI Controller
  • AskUI Controller: v0.2.1.0 Preview
  • AskUI Legacy UI Controller: 0.14.0
• AskUI Runner: 0.1.4
• AskUI Development Environment (ADE)
• AskUI Development Kit (ADK)

3rd Party Versions​

• Node.JS: 21.0.0
• PowerShell 7.3.8

Introduction​

This week we introduced a new convenience method clickTextfieldNearestTo(label: string) which lets you click on a textfield with a label instead of a placeholder.

We also removed the convenience method dragTo as it was not working as intended.

There is a new option -ad --automations-description for initialising an AskUI project with npx askui@latest init. It lets you specify the name of the automations directory which was hardcoded to askui_example before.

Convenience Methods​

We had to remove dragTo() as it was not working as expected

We introduced the following method:

clickTextfieldNearestTo(label: string)​

Searches for an element of type textfield with a specific label nearest to it. If found, clicks it.

await aui.clickTextfieldNearestTo('E-Mail Address');

New Option -ad --automations-description for init​

When you run npx askui@latest init you always got a directory askui_example. This broke backwards compatibility with existing installations of AskUI.

Thus we introduced a new option when running askui init:

-ad, --automations-directory <value>  a name for the directory askui stores its configuration and workflows in. (default: "askui_example")

You can use it like this to store the AskUI automations in a directory askui-Workflows:

npx askui@latest init -ad askui-Workflows

Introduction​

This week we improved on our docs. A guide for setting up your terminal and extensions in Visual Studio Code on Windows will improve your experience while developing AskUI Workflows.

We also fixed a display bug at the Supported Keys page.

In version 0.14.0 of the AskUI SDK we introduced some convenience methods to free users from writing repetitive code. This speeds up the development of AskUI Workflows and improves readability. Make sure to read the convenience docs which give you a detailed description on how to use each method.

Visual Studio Code Setup on Windows​

Visual Studio Code is our recommended editor for developing AskUI workflows. Our docs for the AskUI Development Environment now have a section that helps you to setup Visual Studio Code for developing AskUI workflows efficiently:

• Terminal configuration
• Recommended extensions for Jest Runner, ESLint & Live View

Convenience Methods​

We introduced the following methods:

clickButton(label: string)​

Searches for an element of type button with a label and clicks it when found.

await aui.clickButton('Sign Up');

clickIcon(description: string)​

Clicks an icon based on a description.

await aui.clickIcon('Logo looking like an apple');

clickText(text: string)​

Searches for a text element and clicks it when found.

await aui.clickText('A big bear');

clickTexts(texts: string[])​

Searches for text elements and clicks them one after another when found.

await aui.clickTexts(['A big bear', 'Brown Fox', 'Walks into a bar']);

clickTextfield(placeholder: string)​

Searches for an element of type textfield with a specific placeholder text. If found, clicks it.

await aui.clickTextfield('E-Mail Address');

dragTo(element1: Executable, element2: Executable)​

Drags element1 to element2.

Both must be a moveMouse() or moveMouseTo() instruction as in the example below.

await aui.dragTo(
   aui.moveMouseTo().text('AskUI'),
   aui.moveMouseTo().text('UI Automation')
);

pressKeyNTimes(key: PC_AND_MODIFIER_KEY, times = 2)​

Press a key multiple times. At least two times.

await aui.pressKeyNTimes('right');
await aui.pressKeyNTimes('left', 3);

pressKeys(keys: PC_AND_MODIFIER_KEY[])​

Press an array of keys one after another.

await aui.pressKeys(['right', 'left', 'enter']);

waitUntil(AskUICommand: Executable, maxTry = 5, waitTime = 2000)​

Wait until an AskUICommand does not fail.

Use it to wait for an element to appear like this:

await waitUntil(aui.expect().text('GitHub').exists());

Introduction​

The main focus of the release AskUI Installer 24.2.2 was to polish the previous release 24.2.1 and therefore no major changes were introduced.

AskUI Installer Update 24.2.2​

Introduces a number of new steps: Download link.

Installer for Windows​

The installer contains a number of new and modified dialogues. Furthermore, it contains additional components.

More information can be found here.

New Features​

This release focuses on bug fixes.

Improvements​

This release focuses on bug fixes.

• Improved the robustness of the inter-process connection between Legacy Controller and Remote Device Controller.
• We only allow a single instance of the Legacy Controller for desktop scenarios (mobile platforms are untouched).

Fixes​

This release focuses on bug fixes.

• Fixed left and right arrow key mapping which were swapped.
• Fixed issues with stuck mouse cursor and not finished actions.

Changes​

• Mouse movement uses the OS's cursor position set instead of delta movement on a 1000hz update rate. Making it more reliable.
• Legacy Controller and Remote Device Controller are both shutdown with an error message if the connection is broken.

Known Issues​

Mouse Movement​

• You may encounter issues when mouse pointer acceleration is enabled.
• There might be issues with the mouse movement when it tries to move to invalid coordinates.

Versions​

Components and Packages​

• AskUI Controller
  • AskUI Controller: v0.2.1.0 Preview
  • AskUI Legacy UI Controller: 0.14.0
• AskUI Runner: 0.1.4
• AskUI Development Environment (ADE)
• AskUI Development Kit (ADK)

3rd Party Versions​

• Node.JS: 21.0.0
• PowerShell 7.3.8

Introduction​

AskUI offers a hassle-free installation experience for its products via a user-friendly installer. This installer facilitates the seamless installation of AskUI products with just a few clicks.

General​

Installer for Windows​

The installer contains a number of new and modified dialogues. Furthermore, it contains additional components.

More information can be found here.

New Features​

We introduced a couple of new features.

In Preview​

• AskUI Development Environment (ADE)
  • Introduces environment management
  • Project management
  • Dependency installation
  • Shell
• AskUI Runner
• AskUI Controller

Known Issues​

Installation Dependencies​

• AskUI Development Kit (former AskUI-Lib) is not included in the installer. It will be downloaded during the setup process.

Mouse Movement​

• You may encounter issues when mouse pointer acceleration is enabled.
• There might be issues with the mouse movement when it tries to move to invalid coordinates.

Log Files​

• Log files disk space usage increased. They are stored in ~/.askui/ and can be deleted if that causes any issues.

Android​

• Recording on an Android device is not stable.
• Recorded resolution might be lower than the device resolution.
• Recorded video might be empty or single frame in cases no screen updates occurred during recording.
• We don't support landscape mode on Android.
• Devices with high DPI screen might not work as expected.
• Video recording on MacOS is not supported.

Versions​

Components and Packages​

• AskUI Controller
  • AskUI Controller: v0.2.0.0 Preview
  • AskUI Legacy UI Controller: 0.13.0
• AskUI Runner: 0.1.4
• AskUI Development Environment (ADE)
• AskUI Development Kit (ADK)

3rd Party Versions​

• Node.JS: 21.0.0
• PowerShell 7.3.8

Recommended First-Time User Journey​

To ensure a smooth onboarding experience, follow these steps:

1. Download and Install:
   • Download and install the installer
   • Run the installer.
2. Initialization:
   • Open the Command Prompt.
   • Type askui-shell.
3. Configuration (First-Time Usage):
   • Configure AskUI settings:
     1. Type AskUI-SetSettings -WorkspaceId <askui_workspace_id> -Token <askui_token>.
     2. (Optional for Proxy Users) Type AskUI-SetSettings -HttpProxy <http_proxy_address> -HttpsProxy <https_proxy_address>.
4. Verify Settings:
   • Type AskUI-ShowSettings to verify the configured settings.
5. Create a New Project:
   • Create a new project by typing AskUI-NewProject.
6. Start AskUI Controller:
   • Note: You can skip this if you selected Launch AskUI Controller at the installation wizard.
   • Start the AskUI Controller in background mode by typing AskUI-StartController -RunInBackground.
7. Project Editing (Optional):
   • Open the project directory and starts editing the project.

8. Run the Project:
   • Run the project by typing AskUI-RunProject.

By following these steps, the user can efficiently set up AskUI Suite, configure essential settings, and seamlessly create and run projects.

Introduction​

We're excited to announce a speed enhancement in our OCR model, reducing processing times by 300 milliseconds.

Feature Update​

OCR Model Speed Improvement

• Faster text recognition, significantly improving workflow efficiency.
• 1.62 sec down to 1.26 sec

Detailed Description​

OCR Model Speed Enhancement​

Our latest update has optimized the OCR model, achieving a notable speed increase while maintaining high accuracy.

Model: task='e2e_ocr' architecture='easy_ocr' version='1' interface='im2txtbox' use_case='faster' tags=[]
Average runtime: 1.2629041004180908
Standard deviation: 0.13027134152564313
Median: 1.2297358512878418

We are in the process of creating comprehensive documentation for our enhanced OCR model. We'd love to hear from you about what specific aspects or features you'd like to see covered in the documentation. Please share your thoughts and suggestions with us to ensure our documentation meets your needs and expectations.

Give Feedback >>>

Introduction​

We're introducing two significant enhancements to our platform: Single Sign-On (SSO) Integration for various identity providers and the AskUIXRayStepReporter for automation workflows in Xray for Jira. These features are designed to streamline the authentication process and improve test reporting, respectively.

Overview of Features​

1. Single Sign-On Integration

   • Documentation: SSO Setup Guide
   • This feature allows seamless integration with various identity providers including Azure AD, Google Workspace, and Okta, enhancing security and simplifying user management.

2. AskUIXRayStepReporter

   • Documentation: AskUIXRayStepReporter Guide
   • This feature integrates with Xray for Jira to provide detailed reporting of automation workflows, including screenshots and test statuses.

Detailed Descriptions​

Single Sign-On Integration​

Our new SSO Integration feature supports a wide range of identity providers. It offers enhanced security, unified login experiences, and streamlined user account management, significantly reducing the overhead of managing separate login credentials.

AskUIXRayStepReporter​

The AskUIXRayStepReporter is a powerful tool for reporting test automation results directly to Xray for Jira. It helps teams maintain comprehensive documentation of their test executions, complete with visual evidence and detailed status reports.

We are excited to see how these new features will enhance your experience with our platform, providing greater convenience, security, and efficiency in your automated testing and user management processes.

Introduction​

On December 13, we introduced three new features: the Visual Description Matching Method for selecting diverse visual elements, Targeted Annotations for precise debugging, and Precision Text Matching for flexible text matching.

Overview​

1. Visual Description Matching Method

   • Documentation: Matching Feature
   • This feature introduces the matching()method, allowing users to identify and interact with UI elements based solely on textual descriptions.
   • Enhances the flexibility of automated interactions with dynamic UIs.

2. Targeted Annotations

   • Documentation: Annotate DetectedElements
   • Introduces an AnnotationRequest parameter in the annotate() method, allowing users to annotate specific elements retrieved from user interfaces by the get() method.
   • Simplifies debugging of UI element selection.

3. Precision Text Matching

   • Documentation: Adjustable Similarity Score
   • Introduces an optional similarity_score parameter for the .withText() method.
   • Improves script resilience and minimizes mismatches in text-based automation.