name
assigned to the AI element during snipping to retrieve the data used for locating the ui element(s).
Arguments:
name
str - Name of the AI elementthreshold
float, optional - A threshold for how similar UI elements need to be to the image to be considered a match.
Takes values between 0.0
(= all elements are recognized) and 1.0
(= elements need to look exactly
like defined). Defaults to 0.5
. Important: The threshold impacts the prediction quality.stop_threshold
float | None, optional - A threshold for when to stop searching for UI elements similar to the image. As soon
as UI elements have been found that are at least as similar as the stop_threshold
, the search stops.
Should be greater than or equal to threshold
. Takes values between 0.0
and 1.0
. Defaults to value of
threshold
if not provided. Important: The stop_threshold
impacts the prediction speed.mask
list[tuple[float, float]] | None, optional - A polygon to match only a certain area of the image. Must have at least 3 points.
Defaults to None
.rotation_degree_per_step
int, optional - A step size in rotation degree. Rotates the image by rotation_degree_per_step
until 360° is exceeded. Range is between 0°
- 360°
. Defaults to 0°
. Important: This increases the
prediction time quite a bit. So only use it when absolutely necessary.name
str | None, optional - Name for the image. Defaults to random name.image_compare_format
Literal[“RGB”, “grayscale”, “edges”], optional - A color compare style. Defaults to 'grayscale'
. Important:
The image_compare_format
impacts the prediction time as well as quality. As a rule of thumb,
'edges'
is likely to be faster than 'grayscale'
and 'grayscale'
is likely to be faster than 'RGB'
.
For quality it is most often the other way around.class_name
Literal[“switch”, “text”, “textfield”] | None, optional - The class of the ui element, e.g., 'text'
or 'textfield'
. Defaults to None
.image
Union[PIL.Image.Image, pathlib.Path, str] - The image to match against (PIL Image, path, or string)threshold
float, optional - A threshold for how similar UI elements need to be to the image to be considered a match.
Takes values between 0.0
(= all elements are recognized) and 1.0
(= elements need to look exactly
like defined). Defaults to 0.5
. Important: The threshold impacts the prediction quality.stop_threshold
float | None, optional - A threshold for when to stop searching for UI elements similar to the image. As soon
as UI elements have been found that are at least as similar as the stop_threshold
, the search stops.
Should be greater than or equal to threshold
. Takes values between 0.0
and 1.0
. Defaults to value of
threshold
if not provided. Important: The stop_threshold
impacts the prediction speed.mask
list[tuple[float, float]] | None, optional - A polygon to match only a certain area of the image. Must have at least 3 points.
Defaults to None
.rotation_degree_per_step
int, optional - A step size in rotation degree. Rotates the image by rotation_degree_per_step
until 360° is exceeded. Range is between 0°
- 360°
. Defaults to 0°
. Important: This increases the
prediction time quite a bit. So only use it when absolutely necessary.name
str | None, optional - Name for the image. Defaults to random name.image_compare_format
Literal[“RGB”, “grayscale”, “edges”], optional - A color compare style. Defaults to 'grayscale'
. Important:
The image_compare_format
impacts the prediction time as well as quality. As a rule of thumb,
'edges'
is likely to be faster than 'grayscale'
and 'grayscale'
is likely to be faster than 'RGB'
.
For quality it is most often the other way around.Prompt
, Text
, Image
, etc.
prompt
str - A textual prompt / description of a ui element, e.g., "green sign up button"
"center"
: A is considered to be above, below, right or left of B if it is above, below, right or left of A’s center (in a straight vertical or horizontal line)."boundary"
: A is considered to be above, below, right or left of B if it is above, below, right or left of (any point of the bounding box of) A (in a straight vertical or horizontal line)."any"
: A is considered to be above, below, right or left of B if it is above, below, right or left of B no matter if it can be reached in a straight vertical or horizontal line from (a point of the bounding box of) A.Prompt
, Text
, Image
, etc.
other_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) above the other element(s)
(located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element above the other element(s).
Elements’ (relative) position is determined by the bottom border
(y-coordinate) of their bounding box.
We don’t guarantee the order of elements with the same bottom border
(y-coordinate). Defaults to 0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be above the
other element(s) (located by other_locator). Defaults to "boundary"
.Self
- The locator with the relation addedother_locator
Relatable - The locator to combine withSelf
- The locator with the relation addedother_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) below the other
element(s) (located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element below the other
element(s). Elements’ (relative) position is determined by the top
border (y-coordinate) of their bounding box.
We don’t guarantee the order of elements with the same top border
(y-coordinate). Defaults to 0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be
below the other element(s) (located by other_locator). Defaults to "boundary"
.Self
- The locator with the relation addedother_locator
Relatable - The locator to check if it’s containedSelf
- The locator with the relation addedother_locator
Relatable - The locator to check if it contains this elementSelf
- The locator with the relation addedother_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) left of the other
element(s) (located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element left of the other
element(s). Elements’ (relative) position is determined by the right
border (x-coordinate) of their bounding box.
We don’t guarantee the order of elements with the same right border
(x-coordinate). Defaults to 0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be
left of the other element(s) (located by other_locator). Defaults to "center"
.Self
- The locator with the relation addedother_locator
Relatable - The locator to compare distance againstSelf
- The locator with the relation addedother_locator
Relatable - The locator to combine withSelf
- The locator with the relation added0
), second (1
), third (2
) etc. element
above, below, right or left of the other element B. A’s position relative
to other elements above, below, right or left of B
(which determines its index) is determined by the relative position of its
lowest (above), highest (below), leftmost (right) or rightmost (left) point(s)
(edge of its bounding box).
Important: Which elements are counted (“indexed”) depends on the locator used, e.g.,
when using Text
only text matched is counted, and the reference_point
.
Examples:
reference_point
"center"
, A is the first (index=0
) element above B."boundary"
, A is the second (index=1
) element above B."any"
, A is the third (index=2
) element above B.text
str | None, optional - The text content of the ui element, e.g., 'Sign up'
. Defaults to None
.
If None
, the locator will match any text element.match_type
TextMatchType, optional - The type of match to use. Defaults to "similar"
.similarity_threshold
int, optional - A threshold for how similar the actual text content of the ui element
needs to be to the specified text to be considered a match when match_type
is "similar"
.
Takes values between 0
and 100
(inclusive, higher is more similar).
Defaults to 70
."similar"
uses a similarity threshold to determine if the text is a match."exact"
requires the text to be exactly the same (this is not the same as "similar"
with a similarity_threshold
of 100
as a similarity_threshold
of 100
can still
allow for small differences in very long texts)."contains"
requires the text to contain (exactly) the specified text."regex"
uses a regular expression to match the text.other_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) right of the other
element(s) (located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element right of the other
element(s). Elements’ (relative) position is determined by the left
border (x-coordinate) of their bounding box.
We don’t guarantee the order of elements with the same left border
(x-coordinate). Defaults to 0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be
right of the other element(s) (located by other_locator). Defaults to "center"
.Self
- The locator with the relation added