Table of Contents

tip

Use your browser search: CMD/CTRL + f

Actions

Name
click() production Mouse left-clicks/taps on the filtered element by moving the mouse cursor to the filtered element first. If you need a simple mouseleftclick/tap only, use mouseLeftClick. Example: await aui.click().button().withText('Google Search').exec();
execOnShell() production Executes a shell command on the device your AskUI Controller is connected to. Example: // Open the lastpass app await aui.execOnShell('monkey -p com.lastpass.authenticator 1').exec(); // Open Google Chrome on Windows await aui.execOnShell("start chrome").exec() ;// Open Google Chrome on macOS await aui.execOnShell("open -a 'Google Chrome'").exec(); // Open Google Chrome on Linux await aui.execOnShell("chrome").exec(); @param {string} shell_command - A shell command which is executed.
expect() production Expects a condition, for example, exists() or notExits(). Use the structure expect().<your filter>.(exists()|notExists()) as shown in the examples below. Examples: await aui.expect().text('Login').exists().exec() await aui.expect().text('Login').notExists().exec()
mouseDoubleLeftClick() production Double-clicks with left mouse key. If you need to move the mouse first, use moveMouseTo(). Examples: // Optional: Move mouse to an element first await aui.moveMouseTo().button().withText('Login').exec(); await aui.mouseDoubleLeftClick().exec();
mouseDoubleMiddleClick() production Double-clicks with middle mouse key. If you need to move the mouse first, use moveMouseTo(). Examples: // Optional: Move mouse to an element first await aui.moveMouseTo().button().withText('Login').exec(); await aui.mouseDoubleMiddleClick().exec();
mouseDoubleRightClick() production Double-clicks with right mouse key. If you need to move the mouse first, use moveMouseTo(). Examples: // Optional: Move mouse to an element first await aui.moveMouseTo().button().withText('Login').exec(); await aui.mouseDoubleRightClick().exec();
mouseLeftClick() production Clicks with left mouse key. If you need to move the mouse first, use moveMouseTo(). Examples: // Optional: Move mouse to an element first await aui.moveMouseTo().button().withText('Login').exec(); await aui.mouseLeftClick().exec();
mouseMiddleClick() production Clicks with middle mouse key. If you need to move the mouse first, use moveMouseTo(). Examples: // Optional: Move mouse to an element first await aui.moveMouseTo().button().withText('Login').exec(); await aui.mouseMiddleClick().exec();
mouseRightClick() production Clicks with right mouse key. If you need to move the mouse first, use moveMouseTo(). Examples: // Optional: Move mouse to an element first await aui.moveMouseTo().button().withText('Login').exec(); await aui.mouseRightClick().exec();
mouseToggleDown() production Toggles mouse down (Left mouse key/tap). This is the equivalent to mouse-left-press-and-hold. It holds the mouse button until the mouseToogleUp() is called. Often combined with mouseToggleUP to automate drag-and-drop. Example: await aui.mouseToggleDown().exec(); await aui.moveMouseRelatively(-400,0).exec(); await aui.mouseToggleUp().exec();
mouseToggleUp() production Toggles mouse up (Left mouse key/tap). This is the equivalent to releasing the pressing mouse left button. Often combined with mouseToggleDown() to automate drag-and-drop. Example: await aui.mouseToggleDown().exec(); await aui.moveMouseRelatively(-400,0).exec(); await aui.mouseToggleUp().exec();
moveMouse() production Moves the mouse to the absolute x and y coordinates. If you want to move your mouse cursor to an element, use moveMouseTo(). Example: await aui.moveMouse(500, 500).exec(); @param {number} x_coordinate - A (positive/negative) x coordinate. @param {number} y_coordinate - A (positive/negative) y coordinate.
moveMouseRelatively() production Moves the mouse from the current position (relative) in x and y direction. Example: await aui.moveMouseRelatively(0, 50).exec(); @param {number} x_offset - A (positive/negative) x direction. @param {number} y_offset - A (positive/negative) y direction.
moveMouseRelativelyTo() production Moves the mouse relatively to an element in the direction. This can be used when the mouse should not hover over an element anymore. Examples: // Move mouse 30 pixels below button await aui.moveMouseRelativelyTo(0, 30).button().withText('Submit').exec() @param {number} x_offset - A (positive/negative) x direction. @param {number} y_offset - A (positive/negative) y direction.
moveMouseTo() production Move mouse over the filtered element. Example: await aui.moveMouseTo().text().withText('Grinning_Face').exec()
pressAndroidKey() production Press one Android key like del See API docs for available keys. Examples: await aui.pressAndroidKey('notification').exec(); @param {ANDROID_KEY} key - A Android key
pressAndroidThirdKey() production Press three Android keys like CTRL+ALT+DEL @param {ANDROID_KEY} first_key - A Android key @param {ANDROID_KEY} second_key - A Android key @param {ANDROID_KEY} third_key - A Android key
pressAndroidTwoKey() production Press two Android keys like volume_down+power See API docs for available keys. Examples: await aui.pressAndroidTwoKey('volume_down', 'power').exec(); @param {ANDROID_KEY} first_key - A Android key @param {ANDROID_KEY} second_key - A Android key
pressKey() production Press one keys like DEL Operating system specific mappings: Windows: command-key maps to windows-key Examples: await aui.pressKey('tab').exec(); @param {PC_AND_MODIFIER_KEY} key - A key
pressThreeKeys() production Press three keys like CTRL+ALT+DEL Operating system specific mappings: Windows: command-key maps to windows-key Examples: await aui.pressThreeKeys('control', 'command' 'space').exec(); @param {MODIFIER_KEY} first_key - A modifier key @param {MODIFIER_KEY} second_key - A modifier key @param {PC_KEY} third_key - A key
pressTwoKeys() production Press two keys like ALT+F4 Operating system specific mappings: Windows: command-key maps to windows-key @param {MODIFIER_KEY} first_key - A modifier key @param {PC_KEY} second_key - A key
scroll() production Scrolls based on the current mouse position in the x and y direction. Important: Mouse must be positioned in a scrollable area. macOS: May not work as expected! Example: // Scroll 500 pixels down in y direction await aui.scroll(0, -500).exec() @param {number} x_offset - A (positive/negative) x direction. @param {number} y_offset - A (positive/negative) y direction.
scrollInside() production Moves mouse to the filtered element and scrolls in the x and y direction. macOS: May not work as expected! Example: await aui.scrollInside(0,-500).text().withText('Bottom sheet').exec(); @param {number} x_offset - A (positive/negative) x direction. @param {number} y_offset - A (positive/negative) y direction.
swipe() production Swipe an element in the x and y direction. Holds the left mouse button down on Windows, Linux and macOS and drags the element. On touch devices it taps the element and then drags it. Example: // Swipe the element 500 to the right await aui.swipe(500, 0).image().exec() @param {number} x_offset - A x direction. positive and negative values are accepted @param {number} y_offset - A y direction. positive and negative values are accepted
type() production Types a text at the current position. If you need to focus the element first, use typeIn() Note: In the current version it copies the text and pastes it. Examples: await aui.type('askui@askui.com').exec() // mask the text so it is not send to the askui-inference server await aui.type('Type some text', \{ isSecret: true, secretMask: '**' }).exec() @param {string} text - A text to type
typeIn() production Puts the focus on the filtered element and types in the text. Note: In the current version it copies the text and pastes it. Examples: await aui.typeIn('Type some text').textfield().exec() // mask the text so it is not send to the askui-inference server await aui.typeIn('Type some text', \{ isSecret: true, secretMask: '**' }).textfield().exec() @param {string} text - A text to type

Element-descriptions

Name
aiElement() production Detects an AI Element created with the snipping workflow. Examples: await aui.click().aiElement('askui-logo').exec(); @param {string} aiElementName - Name of the AI Element.
button() production Filters for a UI element 'button'. Examples: await aui.click().button().contains().text().withText('Google Search').exec()
checkbox() production Filters for a UI element 'checkbox'.
container() production Filters for a UI element 'container'.
containsText() production Filters for text containing the text provided as an argument. Examples: 'This is a text' === containsText('text') => true 'This is a text' === containsText('other text') => false 'This is a text' === containsText('other') => false @param {string} text - A text to be matched.
customElement() production Filters for a 'custom element', that is a UI element that is defined by providing an image and other parameters such as degree of rotation. It allows filtering for a UI element based on an image instead of using text or element descriptions like button().withText('Submit') in await aui.click().button().withText('Submit').exec(). See the tutorial - Custom Element for more details. Example await aui .click() .customElement({ customImage: './logo.png', // required name: 'myLogo', // optional threshold: 0.9, // optional, defaults to 0.9 stopThreshold: 0.9, // optional, defaults to 0.9 rotationDegreePerStep: 0, // optional, defaults to 0 imageCompareFormat: 'grayscale', // optional, defaults to 'grayscale' // mask:{x:0, y:0}[] // optional, a polygon to match only a certain area of the custom element }) .exec(); Arguments customImage (string, required): A cropped image in the form of a base64 string or file path. name (string, optional): A unique name that can be used for filtering for the custom element. If not given, any text inside the custom image will be detected via OCR. threshold (number, optional): A threshold for how much a UI element needs to be similar to the custom element as defined by the image. Takes values between 0.0 (== all elements are recognized as the custom element which is probably not what you want) and 1.0 (== elements need to look exactly like the customImage which is unlikely to be achieved as even minor differences count). Defaults to 0.9. stopThreshold (number, optional): A threshold for when to stop searching for UI elements similar to the custom element. As soon as UI elements have been found that are at least as similar as the stopThreshold, the search is going to stop. After that elements are filtered using the threshold. Because of that the stopThreshold should be greater than or equal to threshold. It is primarily to be used as a speed improvement (by lowering the value). Takes values between 0.0 and 1.0. Defaults to 0.9. rotationDegreePerStep (number, optional): Step size in rotation degree. Rotates the custom image by this step size until 360° is exceeded. The range is from 0 to 360. Defaults to 0. imageCompareFormat ('RGB' | 'grayscale' | 'edges', optional): The color compare style. 'edges' compares only edges, 'greyscale' compares the brightness of each pixel whereas 'RGB' compares all three colors (red, green, blue). Defaults to 'grayscale'. @param {CustomElementJson} customElement - The custom element to filter for.
element() production Filters for any UI element on the screen. Examples: await aui.moveMouseTo().element().exec()
icon() production Filters for a UI element 'icon'. You can combine it with the element-description 'withText()' to look for a specific icon. Examples: icon().withText('plus') Note: This is an alpha feature. The prediction of the icon name is sometimes unstable. Use custom elements as an alternative.
image() production Filters for a UI element 'image'. Examples: // Works if there is only one image visible on the screen await aui.click().image().exec(); // Works if you have an image with // a text below await aui.click().image().above().text().withText('Automating WebGL').exec();
matching() experimental Filters elements based on a textual description. What Should I Write as Matching Text The text description inside the matching() should describe the element visually. It understands color, some famous company/product names, general descriptions. Important: Matching only returns the best matching element when you use it with get() A bit of playing around to find a matching description is sometimes needed: For example, puzzle piece can fail while an icon showing a puzzle piece might work. Generally, the more detail the better. We also recommend to not restrict the type of element by using the general selector element() as shown in the examples below. Examples: // Select the black sneaker from a bunch of sneakers await aui.click().element().matching('a black sneaker shoe').exec(); // Select an image that has text in it await aui.click().element().matching('has Burger King in it').exec(); await aui.click().element().matching('has adidas in it').exec(); // Target a logo/image by describing it await aui.click().element().matching('a mask on purple background and a firefox logo').exec(); await aui.click().element().matching('logo looking like an apple with one bite bitten off').exec(); await aui.click().element().matching('logo looking like a seashell').exec(); @param {string} text - A description of the target element.
otherElement() production Filters for a UI element 'other element'.
special() production Filters special elements defined over a specifically trained custom element descriptor. Custom element descriptors are trained on your elements that were not detected with our default models. Please contact us for further details on how to do this. We are working on a solution to provide this in our User Portal. In the example below circle refers to a circle shaped icon with specific properties. Examples: await aui.moveMouseTo().element().special("circle").exec() @param {string} text - A text to be matched.
switch() production Filters for a UI element 'switch'.
table() production Filters for a UI element 'table'.
text() production Filters for an UI element 'text'. Takes an optional parameter to filter for a specific text. See the examples below. See also the filters withTextRegex() and withExactText() Examples: await aui.click().text().exec(); await aui.click().text('Username').exec(); // Matching with an exact text await aui.click().text().withExactText('Username').exec(); // Matching with a regex await aui.click().text().withTextRegex('\b[Ss]\w+').exec(); @param {string} [text] - A text to be matched.
textfield() production Filters for a UI element 'textfield'. Examples: // Works if there is only one textfield visible on the screen await aui.typeIn('Oh yeah').textfield().exec(); // Works if you have a labeled textfield // Label is above the textfield await aui.typeIn('Oh yeah').textfield().below().text().withText('E-Mail Address').exec();
withExactText() production Filters for equal text. Note: This should be only used in cases where the similarity comparison of {@link FluentFilters.withText()} allows not for specific enough filtering (too many elements). Examples: 'text' === withExactText('text') => true 'test' === withExactText('text') => false 'other' === withExactText('text') => false await aui.moveMouseTo().text().withExactText('Password').exec() @param {string} text - A text to be matched.
withText() production Filters for similar -- meaning >70% similar -- text. Takes an optional parameter to specify the similarity. Usually you need the optional parameter for long texts you want to match precisely. We use RapidFuzz which calculates the similarity like this: 1 - (distance / (lengthString1 + lengthString2)) Examples: 'text' === withText('text') => true 'test' === withText('text') => true 'Test' === withText('text') => true 'Text' === withText('text') => true 'TEXT' === withText('text') => true 'texst' === withText('text') => true 'texts' === withText('text') => true // usually false 'atebxtc' === withText('text') => false 'other' === withText('text') => false // optional parameter: similarity_score '978-0-201-00650-6' == withText("978-0-201-00", 90) => false with 82.76 < 90 similarity '978-0-201-00650-6' == withText("978-0-201-00650", 90) => true with 93.75 > 90 similarity @param {string} text - A text to be matched. @param {number} [similarityScore=70] - Similarity score minimum value, it should be between 0 and 100.
withTextRegex() production Filters for texts, which match the regex pattern. Examples: 'The rain in Spain' === withTextRegex('\b[Ss]\w+') => true 'The rain in Portugal' === withTextRegex('\b[Ss]\w+') => false 'The rain in switzerland' === withTextRegex('\b[Ss]\w+') => true await aui.get().text().withTextRegex('\b[Ss]\w+').exec() @param {string} regex_pattern - A regex pattern

Relations

Name
above() production Filters for an element above another element. Takes an optional parameter index to select the nth element (defaults to 0). Takes an optional parameter intersection_area to specify which elements above of the other element are filtered for based on their horizontal position (y-coordinates of bounding box): "element_center_line" - considered above of the other element if element's bounding box intersects with a vertical line passing through the center of the other element "element_edge_area" - considered above of the other element if element's bounding box intersects with an area between the left and the right edge of the other element "display_edge_area" - considered above of the other element no matter where it is placed horizontally on the screen (y-axis) Examples: ------------ | text3 | ------------ ------------ | text2 | ------------ ------------ | text1 | ------------ ------------ | text0 | ------------ ------------ | button | ------------ // General explanation for element_center_line // This will find text0 and text3 ...text().above(..., 'element_center_line').button() // General explanation for element_edge_area // This will find text0, text1 and text3 ...text().above(..., 'element_edge_area').button() // General explanation and display_edge_area // This will find text0, text1, text2 and text3 ...text().above(..., 'display_edge_area').button() // More examples: // Returns text0 because it is the first element above button ...text().above().button() ...text().above(0).button() ...text().above(0, 'element_edge_area').button() // Returns text3 because it is the second text touched by the // vertical line from the center of button // Notice: text1 is not touched! ...text().above(1, 'element_center_line').button() // Returns text3 because it is the third text touched by the // vertical area above the x-axis of button // Notice: text2 is not touched! ...text().above(2, 'element_edge_area').button() // Returns text2 because it is the third element above button ...text().above(2, 'display_edge_area').button() // Returns no element because button is below the texts ...button().above().text() @param {number} [index=0] - Index of element to filter for going into the direction specified. Defaults to 0 which is the first element (zero-indexed) found in that direction. @param {INTERSECTION_AREA} [intersection_area="element_edge_area"] - Intersecting with either "element_center_line", "element_edge_area" or "display_edge_area". Defaults to "element_edge_area".
and() production Logic and operator Examples: example scene: --------------- ---------------- | icon user | | icon search | --------------- ---------------- const icons = await aui.get().icon().exec(); console.log(icons); Using only the element-description icon, the get will return both icons console output: [ DetectedElement { name: 'ICON', text: 'user', bndbox: BoundingBox { xmin: 1000, ymin: 1010, xmax: 1020, ymax: 1030 } }, DetectedElement { name: 'ICON', text: 'search', bndbox: BoundingBox { xmin: 900, ymin: 910, xmax: 920, ymax: 930 } } ] You can combine element-descriptions with the and() relation and specify exactly which icon you want. const icons = await aui.get().icon().and().withText('user').exec() console.log(icons) The get returns only the user icon although both elements are icons. console output: [ DetectedElement { name: 'ICON', text: 'user', bndbox: BoundingBox { xmin: 900, ymin: 910, xmax: 920, ymax: 930 } } ]
below() production Filters for an element below another element. Takes an optional parameter index to select the nth element (defaults to 0). Takes an optional parameter intersection_area to specify which elements below of the other element are filtered for based on their horizontal position (y-coordinates of bounding box): "element_center_line" - considered below of the other element if element's bounding box intersects with a vertical line passing through the center of the other element "element_edge_area" - considered below of the other element if element's bounding box intersects with an area between the left and the right edge of the other element "display_edge_area" - considered below of the other element no matter where it is placed horizontally on the screen (y-axis) Examples: ------------ | text | ------------ ------------ | button0 | ------------ ----------- | button1 | ----------- ----------- | button2 | ----------- ------------ | button3 | ------------ // General explanation for element_center_line // This will find button0 and button3 ...button().below(..., 'element_center_line').text() // General explanation for element_edge_area // This will find button0, button1 and button3 ...button().below(..., 'element_edge_area').text() // General explanation and display_edge_area // This will find button0, button1, button2 and button3 ...button().below(..., 'display_edge_area').text() // More examples: // Returns button0 because button0 is the first button below text ...button().below().text() ...button().below(0).text() ...button().below(0, 'element_edge_area').text() // Returns button3 because it is the second button touched by the // vertical line from the center of text // Notice: button1 is not touched ...button().below(1, 'element_center_line').text() // Returns button3 because it is the third button touched by the // vertical area below the x-axis of text // Notice: button2 is not touched! ...button().below(2, 'element_edge_area').text() // Returns button2 because it is the third element below text ...button().below(2, 'display_edge_area').text() // Returns no element because text is above the buttons ...text().below().button() @param {number} [index=0] - Index of element to filter for going into the direction specified. Defaults to 0 which is the first element (zero-indexed) found in that direction. @param {INTERSECTION_AREA} [intersection_area="element_edge_area"] - Intersecting with either "element_center_line", "element_edge_area" or "display_edge_area". Defaults to "element_edge_area".
contains() production Filters for an element containing another element. Example: -------------------- | outerEl | | -------------- | | | innerEl | | | -------------- | | | -------------------- // Returns outerEl because outerEl contains innerEl ...outerEl().contains().innerEl() // Returns no element because innerEl contains no outerEl ...innerEl().contains().outerEl()
in() production Filters for an element inside another element. Examples: -------------------- | outerEl | | -------------- | | | innerEl | | | -------------- | | | -------------------- // Returns innerEl because innerEl is inside outerEl ...innerEl().in().outerEl() // Returns nothing because innerEl is not inside outerEl ...outerEl().in().innerEl()
leftOf() production Filters for an element left of another element. Takes an optional parameter index to select the nth element (defaults to 0). Takes an optional parameter intersection_area to specify which elements left of the other element are filtered for based on their vertical position (y-coordinates of bounding box): "element_center_line" - considered left of the other element if element's bounding box intersects with a horizontal line passing through the center of the other element "element_edge_area" - considered left of the other element if element's bounding box intersects with an area between the top and the bottom edge of the other element "display_edge_area" - considered left of the other element no matter where it is placed vertically on the screen (y-axis) Examples: --------- --------- ---------- | text3 | | text0 | | button | --------- --------- --------- ---------- --------- | text1 | | text2 | --------- --------- // General explanation for element_center_line // This will find text0 and text3 ...text().leftOf(..., 'element_center_line').button() // General explanation for element_edge_area // This will find text0, text1 and text3 ...text().leftOf(..., 'element_edge_area').button() // General explanation and display_edge_area // This will find text0, text1, text2 and text3 ...text().leftOf(..., 'display_edge_area').button() // More examples: // Returns text0 because it is the first element leftOf button ...text().leftOf().button() ...text().leftOf(0).button() ...text().leftOf(0, 'element_edge_area').button() // Returns text3 because it is the second text touched by the // horizontal line from the center of button // Notice: text1 is not touched! ...text().leftOf(1, 'element_center_line').button() // Returns text3 because it is the third text touched by the // vertical area leftOf the y-axis of button // Notice: text2 is not touched! ...text().leftOf(2, 'element_edge_area').button() // Returns text2 because it is the third element leftOf button ...text().leftOf(2, 'display_edge_area').button() // Returns no element because button is rightOf the texts ...button().leftOf().text() @param {number} [index=0] - Index of element to filter for going into the direction specified. Defaults to 0 which is the first element (zero-indexed) found in that direction. @param {INTERSECTION_AREA} [intersection_area="element_edge_area"] - Intersecting with either "element_center_line", "element_edge_area" or "display_edge_area". Defaults to "element_edge_area".
nearestTo() production Filters for an element nearest to another element. Examples: -------------- | button 1 | -------------- -------------- | text | -------------- -------------- | button 2 | -------------- // Returns button 1 because button 1 is nearer to the text than button 2 ...button().nearestTo().text()
or() production Logic or operator Examples: scene 1 -------------- --------------- | button | | icon | -------------- --------------- scene 2 -------------- --------------- | button | | text | -------------- --------------- In case, that your reference element can have multiple values, in the following example, the element right of the button can be either icon or text. You can use the or() relation, so your instruction is valid for both scenes const button = await aui.get().button().rightOf().icon().or().text().exec(); console.log(button); Returns the same button for both cases console output: [ DetectedElement { name: 'BUTTON', text: 'button', bndbox: BoundingBox { xmin: 900, ymin: 910, xmax: 920, ymax: 930 } } ]
rightOf() production Filters for an element right of another element. Takes an optional parameter index to select the nth element (defaults to 0). Takes an optional parameter intersection_area to specify which elements right of the other element are filtered for based on their vertical position (y-coordinates of bounding box): "element_center_line" - considered right of the other element if element's bounding box intersects with a horizontal line passing through the center of the other element "element_edge_area" - considered right of the other element if element's bounding box intersects with an area between the top and the bottom edge of the other element "display_edge_area" - considered right of the other element no matter where it is placed vertically on the screen (y-axis) Examples: ---------- --------- --------- | button | | text0 | | text3 | ---------- --------- --------- --------- | text1 | --------- --------- | text2 | --------- // General explanation for element_center_line // This will find text0 and text3 ...text().rightOf(..., 'element_center_line').button() // General explanation for element_edge_area // This will find text0, text1 and text3 ...text().rightOf(..., 'element_edge_area').button() // General explanation and display_edge_area // This will find text0, text1, text2 and text3 ...text().rightOf(..., 'display_edge_area').button() // More examples: // Returns text0 because it is the first element rightOf button ...text().rightOf().button() ...text().rightOf(0).button() ...text().rightOf(0, 'element_edge_area').button() // Returns text3 because it is the second text touched by the // horizontal line from the center of button // Notice: text1 is not touched! ...text().rightOf(1, 'element_center_line').button() // Returns text3 because it is the third text touched by the // vertical area rightOf the y-axis of button // Notice: text2 is not touched! ...text().rightOf(2, 'element_edge_area').button() // Returns text2 because it is the third element rightOf button ...text().rightOf(2, 'display_edge_area').button() // Returns no element because button is rightOf the texts ...button().rightOf().text() @param {number} [index=0] - Index of element to filter for going into the direction specified. Defaults to 0 which is the first element (zero-indexed) found in that direction. @param {INTERSECTION_AREA} [intersection_area="element_edge_area"] - Intersecting with either "element_center_line", "element_edge_area" or "display_edge_area". Defaults to "element_edge_area".

Checks

Name
exists() production Expects that filtered element exists. Always use together with expect(). Note Throws an error and stops the execution when the element is not found. You can catch the error and decide what to do as in the examples below. Examples: // Stops execution at this point when the element does not exist. await aui.expect().text('Login').exists().exec() // This will catch the error and log a message // But the execution will continue afterwards try { await aui.expect().text('Login').exists().exec() } catch (error) { console.log('Too bad we could not find the element!'); }
notExists() production Expects that filtered element not exists. Always use together with expect(). Note Throws an error and stops the execution when the element is found. You can catch the error and decide what to do as in the examples below. Examples: // Stops execution at this point when the element does exist. await aui.expect().text('Login').notExists().exec() // This will catch the error and log a message // But the execution will continue afterwards try { await aui.expect().text('Login').notExists().exec() } catch (error) { console.log('Too bad we could find the element!'); }

Getters

Name
get() production Returns an array with all filtered elements. A detected element has the following properties: name of the element text content of element bndbox: location of element described with coordinates of a bounding box Examples: // ************************************ // // Log the DetectedElements completely // // ************************************ // const text = await aui.get().text('Sign').exec(); console.log(text); // Console output [ DetectedElement { name: 'TEXT', text: 'Sign In', bndbox: BoundingBox { xmin: 1128.2720982142857, ymin: 160.21332310267857, xmax: 1178.8204241071428, ymax: 180.83512834821428 } } ] // *************************************************** // // Examples on how to work with the returned elements // // *************************************************** // const texts = await aui.get().text().below().textfield().exec(); // We can get a lot of elements this way console.log(texts); // Console output [ DetectedElement { name: 'TEXT', text: 'Sign In', bndbox: BoundingBox { xmin: 1128.2720982142857, ymin: 160.21332310267857, xmax: 1178.8204241071428, ymax: 180.83512834821428 }, }, DetectedElement { name: 'TEXT', text: 'Login', bndbox: BoundingBox { xmin: 250.8204241071428, ymin: 300.21332310267857, xmax: 450.6304241071428, ymax: 950.47812834821428 }, }, ... 10 more items ] // Extract the FIRST element // Arrays start with index 0! const firstTextElement = texts[0]; const textOfFirstElement = firstElement.text; console.log(textOfFirstElement); // Console output Sign In // Log the text of the SECOND element // with shorter code const texts = await aui.get().text().below().textfield().exec(); console.log(texts[1].text) // Console output Login
getAll() production Returns an array with all detected elements. A detected element has the following properties: name of the element text content of element bndbox: location of element described with coordinates of a bounding box Examples: const detectedElements = await aui.getAll().exec(); console.log(detectedElements); console output: [ DetectedElement { name: 'TEXT', text: 'Sign In', bndbox: BoundingBox { xmin: 1128.2720982142857, ymin: 160.21332310267857, xmax: 1178.8204241071428, ymax: 180.83512834821428 }, }, DetectedElement { name: 'ICON', text: 'search', bndbox: BoundingBox { xmin: 250.8204241071428, ymin: 300.21332310267857, xmax: 450.6304241071428, ymax: 950.47812834821428 }, }, ... 381 more items ]