customElement
Filters for a 'custom element', that is a UI element which is defined by providing an image and other parameters such as degree of rotation. It allows filtering for a UI element that is not recognized by our machine learning models by default. It can also be used for pixel assertions of elements using classical template matching.
See the tutorial - Custom Element for more detail.
Example
await aui
.click()
.customElement({
customImage: './logo.png', // required
name: 'myLogo', // optional
threshold: 0.9, // optional, defaults to 0.9
rotationDegreePerStep: 0, // optional, defaults to 0
imageCompareFormat: 'grayscale', // optional, defaults to 'grayscale'
// mask:{x:0, y:0}[] // optional, a polygon to match only a certain area of the custom element
})
.exec();
Arguments
- customImage (
string
, required):- A cropped image in the form of a base64 string or file path.
- name (
string
, optional):- A unique name that can be used for filtering for the custom element. If not given, any text inside the custom image will be detected via OCR.
- threshold (
number
, optional):- A threshold for how much a UI element needs to be similar to the custom element as defined. Takes values between
0.0
(== all elements are recognized as the custom element which is probably not what you want) and1.0
(== elements need to look exactly like thecustomImage
which is unlikely to be achieved as even minor differences count). Defaults to0.9
.
- A threshold for how much a UI element needs to be similar to the custom element as defined. Takes values between
- rotationDegreePerStep (
number
, optional):- Step size in rotation degree. Rotates the custom image by this step size until 360° is exceeded. The range is from
0
to360
. Defaults to0
.
- Step size in rotation degree. Rotates the custom image by this step size until 360° is exceeded. The range is from
- imageCompareFormat (
'RGB' | 'grayscale'
, optional):-
The color compare style. 'greyscale' compares the brightness of each pixel whereas 'RGB' compares all three color. Defaults to 'grayscale'.
- @param {CustomElementJson} customElement - The custom element to filter for.
-