Skip to main content
Version: 0.18.0

Table of Contents

tip

Use your browser search: CMD/CTRL + f

Actions​

Name
click() production

Mouse left-clicks/taps on the filtered element by moving the mouse cursor to the filtered element first.

If you need a simple mouseleftclick/tap only, use mouseLeftClick.

Example:

await aui.click().button().withText('Submit').exec()
execOnShell() production

Executes a shell command on the device your AskUI Controller is connected to.

Example:

// Open the lastpass app
await aui.execOnShell('monkey -p com.lastpass.authenticator 1').exec();

// Open Google Chrome on Windows
await aui.execOnShell("start chrome").exec()

;// Open Google Chrome on macOS
await aui.execOnShell("open -a 'Google Chrome'").exec();

// Open Google Chrome on Linux
await aui.execOnShell("chrome").exec();
  • @param {string} shell_command - A shell command which is executed.
expect() production

Expects a condition, e.g., exists() or notExits().

Use the structure expect().<your filter>.(exists()|notExists()) as shown in the examples below.

Examples:

await aui.expect().text('Login').exists().exec()
await aui.expect().text('Login').notExists().exec()
mouseDoubleLeftClick() production

Double-clicks with left mouse key.

If you need to move the mouse first, use moveMouseTo().

Examples:

// Optional: Move mouse to an element first
await aui.moveMouseTo().button().withText('Login').exec();

await aui.mouseDoubleLeftClick().exec();
mouseDoubleMiddleClick() production

Double-clicks with middle mouse key.

If you need to move the mouse first, use moveMouseTo().

Examples:

// Optional: Move mouse to an element first
await aui.moveMouseTo().button().withText('Login').exec();

await aui.mouseDoubleMiddleClick().exec();
mouseDoubleRightClick() production

Double-clicks with right mouse key.

If you need to move the mouse first, use moveMouseTo().

Examples:

// Optional: Move mouse to an element first
await aui.moveMouseTo().button().withText('Login').exec();

await aui.mouseDoubleRightClick().exec();
mouseLeftClick() production

Clicks with left mouse key.

If you need to move the mouse first, use moveMouseTo().

Examples:

// Optional: Move mouse to an element first
await aui.moveMouseTo().button().withText('Login').exec();

await aui.mouseLeftClick().exec();
mouseMiddleClick() production

Clicks with middle mouse key.

If you need to move the mouse first, use moveMouseTo().

Examples:

// Optional: Move mouse to an element first
await aui.moveMouseTo().button().withText('Login').exec();

await aui.mouseMiddleClick().exec();
mouseRightClick() production

Clicks with right mouse key.

If you need to move the mouse first, use moveMouseTo().

Examples:

// Optional: Move mouse to an element first
await aui.moveMouseTo().button().withText('Login').exec();

await aui.mouseRightClick().exec();
mouseToggleDown() production

Toggles mouse down (Left mouse key/tap).

Example:

await aui.mouseToggleDown().exec();
mouseToggleUp() production

Toggles mouse up (Left mouse key/tap).

Example:

await aui.mouseToggleUp().exec();
moveMouse() production

Moves the mouse to the absolute x and y coordinates.

If you want to move your mouse cursor to an element, use moveMouseTo().

Example:

await aui.moveMouse(500, 500).exec();
  • @param {number} x_coordinate - A (positive/negative) x coordinate.
  • @param {number} y_coordinate - A (positive/negative) y coordinate.
moveMouseRelatively() production

Moves the mouse from the current position (relative) in x and y direction.

Example:

await aui.moveMouseRelatively(20, 20).exec();
  • @param {number} x_offset - A (positive/negative) x direction.
  • @param {number} y_offset - A (positive/negative) y direction.
moveMouseRelativelyTo() production

Moves the mouse relatively to an element in the direction. This can be used when the mouse should not hover over an element anymore.

Examples:

// Move mouse 30 pixels below button
await aui.moveMouseRelativelyTo(0, 30).button().withText('Submit').exec()

  • @param {number} x_offset - A (positive/negative) x direction.
  • @param {number} y_offset - A (positive/negative) y direction.
moveMouseTo() production

Move mouse over the filtered element.

Example:

await aui.moveMouseTo().button().withText('Submit').exec()
pressAndroidKey() production

Press one Android key like DEL

  • @param {ANDROID_KEY} key - A Android key
pressAndroidThirdKey() production

Press three Android keys like CTRL+ALT+DEL

  • @param {ANDROID_KEY} first_key - A Android key
  • @param {ANDROID_KEY} second_key - A Android key
  • @param {ANDROID_KEY} third_key - A Android key
pressAndroidTwoKey() production

Press two Android keys like ALT+F4

  • @param {ANDROID_KEY} first_key - A Android key
  • @param {ANDROID_KEY} second_key - A Android key
pressKey() production

Press one keys like DEL

Operating system specific mappings:

  1. Windows: command-key maps to windows-key
  • @param {PC_AND_MODIFIER_KEY} key - A key
pressThreeKeys() production

Press three keys like CTRL+ALT+DEL

Operating system specific mappings:

  1. Windows: command-key maps to windows-key
  • @param {MODIFIER_KEY} first_key - A modifier key
  • @param {MODIFIER_KEY} second_key - A modifier key
  • @param {PC_KEY} third_key - A key
pressTwoKeys() production

Press two keys like ALT+F4

Operating system specific mappings:

  1. Windows: command-key maps to windows-key
  • @param {MODIFIER_KEY} first_key - A modifier key
  • @param {PC_KEY} second_key - A key
scroll() production

Scrolls based on the current mouse position in the x and y direction.

Important: Mouse must be positioned in a scrollable area.

macOS: May not work as expected!

Example:

// Scroll 10 up in y direction
await aui.scroll(0, 10).exec()
  • @param {number} x_offset - A (positive/negative) x direction.
  • @param {number} y_offset - A (positive/negative) y direction.
scrollInside() production

Moves mouse to the filtered element and scrolls in the x and y direction.

macOS: May not work as expected!

Example:

await aui.scroll(0, 10).textarea().exec()
  • @param {number} x_offset - A (positive/negative) x direction.
  • @param {number} y_offset - A (positive/negative) y direction.
swipe() production

Swipe an element in the x and y direction. Holds the left mouse button down on Windows, Linux and macOS and drags the element. On touch devices it taps the element and then drags it.

Example:

// Swipe the element 500 to the right
await aui.swipe(500, 0).image().exec()

  • @param {number} x_offset - A x direction. positive and negative values are accepted
  • @param {number} y_offset - A y direction. positive and negative values are accepted
type() production

Types a text at the current position. If you need to focus the element first, use typeIn()

Note: In the current version it copies the text and pastes it.

Examples:

await aui.type('Type some text').exec()

// mask the text so it is not send to the askui-inference server
await aui.type('Type some text', { isSecret: true, secretMask: '**' }).exec()
  • @param {string} text - A text to type
typeIn() production

Puts the focus on the filtered element and types in the text.

Note: In the current version it copies the text and pastes it.

Examples:

await aui.typeIn('Type some text').textfield().exec()

// mask the text so it is not send to the askui-inference server
await aui.typeIn('Type some text', \{ isSecret: true, secretMask: '**' }).textfield().exec()

  • @param {string} text - A text to type

Element-descriptions​

Name
button() production

Filters for a UI element 'button'.

Examples:

await aui.moveMouseTo().button().exec()
checkbox() production

Filters for a UI element 'checkbox'.

container() production

Filters for a UI element 'container'.

containsText() production

Filters for text containing the text provided as an argument.

Examples:

'This is a text' === containsText('text') => true
'This is a text' === containsText('other text') => false
'This is a text' === containsText('other') => false

  • @param {string} text - A text to be matched.
customElement() production

Filters for a 'custom element', that is a UI element which is defined by providing an image and other parameters such as degree of rotation. It allows filtering for a UI element that is not recognized by our machine learning models by default. It can also be used for pixel assertions of elements using classical template matching.

Example

await aui
    .click()
    .customElement({
        customImage: './logo.png', // required
        name: 'myLogo', // optional
        threshold: 0.9, // optional, defaults to 0.9
        rotationDegreePerStep: 0, // optional, defaults to 0
        imageCompareFormat: 'grayscale', // optional, defaults to 'grayscale'
        // mask:{x:0, y:0}[] // optional, a polygon to match only a certain area of the custom element
    })
    .exec();

Arguments

  • customImage (string, required):
    • A cropped image in the form of a base64 string or file path.
  • name (string, optional):
    • A unique name that can be used for filtering for the custom element. If not given, any text inside the custom image will be detected via OCR.
  • threshold (number, optional):
    • A threshold for how much a UI element needs to be similar to the custom element as defined. Takes values between 0.0 (== all elements are recognized as the custom element which is probably not what you want) and 1.0 (== elements need to look exactly like the customImage which is unlikely to be achieved as even minor differences count). Defaults to 0.9.
  • rotationDegreePerStep (number, optional):
    • Step size in rotation degree. Rotates the custom image by this step size until 360° is exceeded. The range is from 0 to 360. Defaults to 0.
  • imageCompareFormat ('RGB' | 'grayscale', optional):
    • The color compare style. 'greyscale' compares the brightness of each pixel whereas 'RGB' compares all three color. Defaults to 'grayscale'. of the given custom image.
  • @param {CustomElementJson} customElement - The custom element to filter for.
element() production

Filters for any UI element on the screen.

Examples:

await aui.moveMouseTo().element().exec()
icon() production

Filters for a UI element 'icon'.

You can combine it with the element-description 'withText()' to look for a specific icon.

Examples:

icon().withText('plus')

Note: This is an alpha feature. The prediction of the icon name is sometimes unstable. Use custom elements as an alternative.

image() production

Filters for a UI element 'image'.

Examples:

// Works if there is only one image visible on the screen
await aui.click().image().exec();

// Works if you have an image with
// a caption text below
await aui.click().image().above().text('The caption').exec();
matching() experimental

Filters elements based on a textual description.

What Should I Write as Matching Text

The text description inside the matching() should describe the element visually. It understands color, some famous company/product names, general descriptions.

Important: Matching only returns the best matching element when you use it with get()

A bit of playing around to find a matching description is sometimes needed: E.g., puzzle piece can fail while an icon showing a puzzle piece might work. Generally, the more detail the better.

We also recommend to not restrict the type of element by using the general selector element() as shown in the examples below.

Examples:

// Select the black sneaker from a bunch of sneakers
await aui.click().element().matching('a black sneaker shoe').exec();

// Select an image that has text in it
await aui.click().element().matching('has Burger King in it').exec();
await aui.click().element().matching('has adidas in it').exec();

// Target a logo/image by describing it
await aui.click().element().matching('a mask on purple background and a firefox logo').exec();
await aui.click().element().matching('logo looking like an apple with one bite bitten off').exec();
await aui.click().element().matching('logo looking like a seashell').exec();
  • @param {string} text - A description of the target element.
otherElement() production

Filters for a UI element 'other element'.

special() production

Filters special elements defined over a specifically trained custom element descriptor.

Custom element descriptors are trained on your elements that were not detected with our default models. Please contact us for further details on how to do this. We are working on a solution to provide this in our User Portal.

In the example below circle refers to a circle shaped icon with specific properties.

Examples:

await aui.moveMouseTo().element().special("circle").exec()
  • @param {string} text - A text to be matched.
switch() production

Filters for a UI element 'switch'.

table() production

Filters for a UI element 'table'.

text() production

Filters for an UI element 'text'.

Takes an optional parameter to filter for a specific text. See the examples below.

See also the filters withTextRegex() and withExactText()

Examples:

await aui.click().text().exec();
await aui.click().text('Username').exec();
await aui.click().text().withTextRegex('\b[Ss]\w+').exec();
  • @param {string} [text] - A text to be matched.
textfield() production

Filters for a UI element 'textfield'.

Examples:

// Works if there is only one textfield visible on the screen
await aui.typeIn('Oh yeah').textfield().exec();

// Works if you have a labeled textfield
// Label is above the textfield
await aui.typeIn('Oh yeah').textfield().below().text('E-Mail Address').exec();
withExactText() production

Filters for equal text.

Note: This should be only used in cases where the similarity comparison of {@link FluentFilters.withText()} allows not for specific enough filtering (too many elements).

Examples:

'text' === withExactText('text') => true
'test' === withExactText('text') => false
'other' === withExactText('text') => false

await aui.moveMouseTo().text().withExactText('Password').exec()
  • @param {string} text - A text to be matched.
withText() production

Filters for similar -- meaning >70% similar -- text.

Takes an optional parameter to specify the similarity. Usually you need the optional parameter for long texts you want to match precisely.

We use RapidFuzz which calculates the similarity like this:

1 - (distance / (lengthString1 + lengthString2))

Examples:

'text' === withText('text') => true
'test' === withText('text') => true
'Test' === withText('text') => true
'Text' === withText('text') => true
'TEXT' === withText('text') => true
'texst' === withText('text') => true
'texts' === withText('text') => true

'atebxtc' === withText('text') => false
'other' === withText('text') => false

// optional parameter: similarity_score
'978-0-201-00650-6' == withText("978-0-201-00", 90) => false with 82.76 < 90 similarity
'978-0-201-00650-6' == withText("978-0-201-00650", 90) => true with 93.75 > 90 similarity

  • @param {string} text - A text to be matched.
withTextRegex() production

Filters for texts, which match the regex pattern.

Examples:

'The rain in Spain' === withTextRegex('\b[Ss]\w+') => true
'The rain in Portugal' === withTextRegex('\b[Ss]\w+') => false
'The rain in switzerland' === withTextRegex('\b[Ss]\w+') => true

await aui.get().text().withTextRegex('\b[Ss]\w+').exec()
  • @param {string} regex_pattern - A regex pattern

Relations​

Name
above() production

Filters for an element above another element.

Takes an optional parameter index to select the nth element (starting with 0)

Examples:

--------------
|   text1    |
--------------
--------------
|   text0    |
--------------
--------------
|   button   |
--------------

// Returns text0 because text0 is the first element above button
...text().above().button()
...text().above(0).button()
// Returns text1 because text1 is the second element above button
...text().above(1).button()
// Returns no element because button is below text
...button().above().text()

  • @param {number} [optionalIndex=0] - element index
and() production

Logic and operator

Examples:

example scene: 
 ---------------   ----------------
 |  icon user  |   |  icon search |
 ---------------   ----------------
const icons = await aui.get().icon().exec();
console.log(icons);

Using only the element-description icon, the get will return both icons

console output: [
  DetectedElement {
     name: 'ICON',
     text: 'user',
     bndbox: BoundingBox {
        xmin: 1000,
        ymin: 1010,
        xmax: 1020,
        ymax: 1030
     }
  },
  DetectedElement {
     name: 'ICON',
     text: 'search',
     bndbox: BoundingBox {
        xmin: 900,
        ymin: 910,
        xmax: 920,
        ymax: 930
     }
  }
 ]

You can combine element-descriptions with the and() relation and specify exactly which icon you want.

const icons = await aui.get().icon().and().withText('user').exec()
console.log(icons)

The get returns only the user icon although both elements are icons.

 console output: [
  DetectedElement {
     name: 'ICON',
     text: 'user',
     bndbox: BoundingBox {
        xmin: 900,
        ymin: 910,
        xmax: 920,
        ymax: 930
     }
  }
 ]
below() production

Filters for an element below another element.

Takes an optional parameter index to select the nth element (starting with 0)

Examples:

--------------
|    text    |
--------------
--------------
|   button0  |
--------------
--------------
|   button1  |
--------------

// Returns button0 because button0 is the first button below text
...button().below().text()
...button().below(0).text()
// Returns button1 because button1  is the second button below text
...button().below(1).text()
// Returns no element because text is above button
...text().below().button()

  • @param {number} [optionalIndex=0] - element index
contains() production

Filters for an element containing another element.

Example:

--------------------
|     outerEl      |
|  --------------  |
|  |  innerEl   |  |
|  --------------  |
|                  |
--------------------

// Returns outerEl because outerEl contains innerEl
...outerEl().contains().innerEl()
//  Returns no element because innerEl contains no outerEl
...innerEl().contains().outerEl()

in() production

Filters for an element inside another element.

Examples:

--------------------
|     outerEl      |
|  --------------  |
|  |  innerEl   |  |
|  --------------  |
|                  |
--------------------

// Returns innerEl because innerEl is inside outerEl
...innerEl().in().outerEl()
// Returns nothing because innerEl is not inside outerEl
...outerEl().in().innerEl()

leftOf() production

Filters for an element left of another element.

Takes an optional parameter index to select the nth element (starting with 0)

Examples:

--------------  --------------  --------------
|  leftEl1   |  |  leftEl0   |  |  rightEl   |
--------------  --------------  --------------

// Returns leftEl0 because leftEl0 is the first element left of rightEl
...leftEl().leftOf().rightEl()
...leftEl().leftOf(0).rightEl()
// Returns leftEl1 because leftEl1 is the second element left of rightEl
...leftEl().leftOf(1).rightEl()
// Returns no element because rightEl is left of leftEl
...rightEl().leftOf().leftEl()

  • @param {number} [optionalIndex=0] - element index
nearestTo() production

Filters for an element nearest to another element.

Examples:

--------------
|  button 1  |
--------------
--------------
|   text     |
--------------
              
              
              
--------------
|  button 2  |
--------------

// Returns button 1 because button 1 is nearer to the text than button 2
...button().nearestTo().text()

or() production

Logic or operator

Examples:

scene 1
--------------  ---------------
|  button    |  |  icon       |
--------------  ---------------

scene 2
--------------  ---------------
|  button    |  |  text       |
--------------  ---------------

In case, that your reference element can have multiple values, in the following example, the element right of the button can be either icon or text. You can use the or() relation, so your instruction is valid for both scenes

const button = await aui.get().button().rightOf().icon().or().text().exec();
console.log(button);

Returns the same button for both cases

 console output: [
  DetectedElement {
     name: 'BUTTON',
     text: 'button',
     bndbox: BoundingBox {
        xmin: 900,
        ymin: 910,
        xmax: 920,
        ymax: 930
     }
  }
 ]
rightOf() production

Filters for an element right of another element.

Takes an optional parameter index to select the nth element (starting with 0)

Examples:

--------------  --------------  --------------
|  leftEl    |  |  rightEl0  |  |  rightEl1  |
--------------  --------------  --------------

// Returns rightEl0 because rightEl0 is the first element right of leftEl
...rightEl().rightOf().leftEl()
...rightEl().rightOf(0).leftEl()
// Returns rightEl1 because rightEl1 is the second element right of leftEl
...rightEl().rightOf(1).leftEl()
// Returns no element because leftEl is left of rightEl
...leftEl().rightOf().rightEl()

  • @param {number} [optionalIndex=0] - element index

Checks​

Name
exists() production

Expects that filtered element exists.

Always use together with expect().

Note Throws an error and stops the execution when the element is not found. You can catch the error and decide what to do as in the examples below.

Examples:

// Stops execution at this point when the element does not exist.
await aui.expect().text('Login').exists().exec()

// This will catch the error and log a message
// But the execution will continue afterwards
try {
    await aui.expect().text('Login').exists().exec()
} catch (error) {
    console.log('Too bad we could not find the element!');
}
notExists() production

Expects that filtered element not exists.

Always use together with expect().

Note Throws an error and stops the execution when the element is found. You can catch the error and decide what to do as in the examples below.

Examples:

// Stops execution at this point when the element does exist.
await aui.expect().text('Login').notExists().exec()

// This will catch the error and log a message
// But the execution will continue afterwards
try {
    await aui.expect().text('Login').notExists().exec()
} catch (error) {
    console.log('Too bad we could find the element!');
}

Getters​

Name
get() production

Returns an array with all filtered elements. A detected element has the following properties:

  • name of the element
  • text content of element
  • bndbox: location of element described with coordinates of a bounding box

Examples:

// ************************************ //
// Log the DetectedElements completely  //
// ************************************ //
const text = await aui.get().text('Sign').exec();
console.log(text);

// Console output
[
  DetectedElement {
    name: 'TEXT',
    text: 'Sign In',
    bndbox: BoundingBox {
      xmin: 1128.2720982142857,
      ymin: 160.21332310267857,
      xmax: 1178.8204241071428,
      ymax: 180.83512834821428
    }
  }
]
// *************************************************** //
// Examples on how to work with the returned elements  //
// *************************************************** //
const texts = await aui.get().text().below().textfield().exec();

// We can get a lot of elements this way
console.log(texts);

// Console output
[
  DetectedElement {
    name: 'TEXT',
    text: 'Sign In',
    bndbox: BoundingBox {
      xmin: 1128.2720982142857,
      ymin: 160.21332310267857,
      xmax: 1178.8204241071428,
      ymax: 180.83512834821428
    },
  DetectedElement {
    name: 'TEXT',
    text: 'Login',
    bndbox: BoundingBox {
      xmin: 250.8204241071428,
      ymin: 300.21332310267857,
      xmax: 450.6304241071428,
      ymax: 950.47812834821428
    },
    ... 10 more items
  }
]

// Extract the FIRST element
// Arrays start with index 0!
const firstTextElement = texts[0];
const textOfFirstElement = firstElement.text;

console.log(textOfFirstElement);

// Console output
Sign In

// Log the text of the SECOND element
// with shorter code
const texts = await aui.get().text().below().textfield().exec();
console.log(texts[1].text)

// Console output
Login
getAll() production

Returns an array with all detected elements. A detected element has the following properties:

  • name of the element
  • text content of element
  • bndbox: location of element described with coordinates of a bounding box

Examples:

const detectedElements = await aui.getAll().exec();
console.log(detectedElements);
 console output: [
  DetectedElement {
     name: 'TEXT',
     text: 'Sign In',
     bndbox: BoundingBox {
        xmin: 1128.2720982142857,
        ymin: 160.21332310267857,
        xmax: 1178.8204241071428,
        ymax: 180.83512834821428
     },
   }
  DetectedElement {
     name: 'ICON',
     text: 'search',
     bndbox: BoundingBox {
        xmin: 250.8204241071428,
        ymin: 300.21332310267857,
        xmax: 450.6304241071428,
        ymax: 950.47812834821428
     },
     ... 381 more items
   }
 ]