Skip to main content
Version: 0.18.0

customElement

production

Filters for a 'custom element', that is a UI element which is defined by providing an image and other parameters such as degree of rotation. It allows filtering for a UI element that is not recognized by our machine learning models by default. It can also be used for pixel assertions of elements using classical template matching.

See the tutorial - Custom Element for more detail.

Example

await aui
.click()
.customElement({
customImage: './logo.png', // required
name: 'myLogo', // optional
threshold: 0.9, // optional, defaults to 0.9
rotationDegreePerStep: 0, // optional, defaults to 0
imageCompareFormat: 'grayscale', // optional, defaults to 'grayscale'
// mask:{x:0, y:0}[] // optional, a polygon to match only a certain area of the custom element
})
.exec();

Arguments

  • customImage (string, required):
    • A cropped image in the form of a base64 string or file path.
  • name (string, optional):
    • A unique name that can be used for filtering for the custom element. If not given, any text inside the custom image will be detected via OCR.
  • threshold (number, optional):
    • A threshold for how much a UI element needs to be similar to the custom element as defined. Takes values between 0.0 (== all elements are recognized as the custom element which is probably not what you want) and 1.0 (== elements need to look exactly like the customImage which is unlikely to be achieved as even minor differences count). Defaults to 0.9.
  • rotationDegreePerStep (number, optional):
    • Step size in rotation degree. Rotates the custom image by this step size until 360° is exceeded. The range is from 0 to 360. Defaults to 0.
  • imageCompareFormat ('RGB' | 'grayscale', optional):
    • The color compare style. 'greyscale' compares the brightness of each pixel whereas 'RGB' compares all three color. Defaults to 'grayscale'.

    • @param {CustomElementJson} customElement - The custom element to filter for.