askui.locators.AiElement
name
assigned to the AI element during snipping to retrieve the data used for locating the ui element(s).
Arguments:
name
str - Name of the AI elementthreshold
float, optional - A threshold for how similar UI elements need to be to the image to be considered a match. Takes values between0.0
(= all elements are recognized) and1.0
(= elements need to look exactly like defined). Defaults to0.5
. Important: The threshold impacts the prediction quality.stop_threshold
float | None, optional - A threshold for when to stop searching for UI elements similar to the image. As soon as UI elements have been found that are at least as similar as thestop_threshold
, the search stops. Should be greater than or equal tothreshold
. Takes values between0.0
and1.0
. Defaults to value ofthreshold
if not provided. Important: Thestop_threshold
impacts the prediction speed.mask
list[tuple[float, float]] | None, optional - A polygon to match only a certain area of the image. Must have at least 3 points. Defaults toNone
.rotation_degree_per_step
int, optional - A step size in rotation degree. Rotates the image by rotation_degree_per_step until 360° is exceeded. Range is between0°
-360°
. Defaults to0°
. Important: This increases the prediction time quite a bit. So only use it when absolutely necessary.name
str | None, optional - Name for the image. Defaults to random name.image_compare_format
Literal[“RGB”, “grayscale”, “edges”], optional - A color compare style. Defaults to'grayscale'
. Important: Theimage_compare_format
impacts the prediction time as well as quality. As a rule of thumb,'edges'
is likely to be faster than'grayscale'
and'grayscale'
is likely to be faster than'RGB'
. For quality it is most often the other way around.
askui.locators.CircularDependencyError
askui.locators.Element
class_name
Literal[“switch”, “text”, “textfield”] | None, optional - The class of the ui element, e.g.,'text'
or'textfield'
. Defaults toNone
.
askui.locators.Image
image
Union[PIL.Image.Image, pathlib.Path, str] - The image to match against (PIL Image, path, or string)threshold
float, optional - A threshold for how similar UI elements need to be to the image to be considered a match. Takes values between0.0
(= all elements are recognized) and1.0
(= elements need to look exactly like defined). Defaults to0.5
. Important: The threshold impacts the prediction quality.stop_threshold
float | None, optional - A threshold for when to stop searching for UI elements similar to the image. As soon as UI elements have been found that are at least as similar as thestop_threshold
, the search stops. Should be greater than or equal tothreshold
. Takes values between0.0
and1.0
. Defaults to value ofthreshold
if not provided. Important: Thestop_threshold
impacts the prediction speed.mask
list[tuple[float, float]] | None, optional - A polygon to match only a certain area of the image. Must have at least 3 points. Defaults toNone
.rotation_degree_per_step
int, optional - A step size in rotation degree. Rotates the image byrotation_degree_per_step
until 360° is exceeded. Range is between0°
-360°
. Defaults to0°
. Important: This increases the prediction time quite a bit. So only use it when absolutely necessary.name
str | None, optional - Name for the image. Defaults to random name.image_compare_format
Literal[“RGB”, “grayscale”, “edges”], optional - A color compare style. Defaults to'grayscale'
. Important: Theimage_compare_format
impacts the prediction time as well as quality. As a rule of thumb,'edges'
is likely to be faster than'grayscale'
and'grayscale'
is likely to be faster than'RGB'
. For quality it is most often the other way around.
askui.locators.Locator
Prompt
, Text
, Image
, etc.
askui.locators.Prompt
prompt
str - A textual prompt / description of a ui element, e.g.,"green sign up button"
askui.locators.ReferencePoint
"center"
: A is considered to be above, below, right or left of B if it is above, below, right or left of A’s center (in a straight vertical or horizontal line).
"boundary"
: A is considered to be above, below, right or left of B if it is above, below, right or left of (any point of the bounding box of) A (in a straight vertical or horizontal line).
"any"
: A is considered to be above, below, right or left of B if it is above, below, right or left of B no matter if it can be reached in a straight vertical or horizontal line from (a point of the bounding box of) A.
askui.locators.Relatable
Prompt
, Text
, Image
, etc.
above_of
- if most of A (or, more specifically, A’s bounding box) is above B (or, more specifically, the top border of B’s bounding box) and
- if the bottom border of A (or, more specifically, A’s bounding box) is above the bottom border of B (or, more specifically, B’s bounding box).
other_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) above the other element(s) (located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element above the other element(s). Elements’ (relative) position is determined by the bottom border (y-coordinate) of their bounding box. We don’t guarantee the order of elements with the same bottom border (y-coordinate). Defaults to0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be above the other element(s) (located by other_locator). Defaults to"boundary"
.
Self
- The locator with the relation added
and_
other_locator
Relatable - The locator to combine with
Self
- The locator with the relation added
below_of
- if most of A (or, more specifically, A’s bounding box) is below B (or, more specifically, the bottom border of B’s bounding box) and
- if the top border of A (or, more specifically, A’s bounding box) is below the top border of B (or, more specifically, B’s bounding box).
other_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) below the other element(s) (located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element below the other element(s). Elements’ (relative) position is determined by the top border (y-coordinate) of their bounding box. We don’t guarantee the order of elements with the same top border (y-coordinate). Defaults to0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be below the other element(s) (located by other_locator). Defaults to"boundary"
.
Self
- The locator with the relation added
containing
other_locator
Relatable - The locator to check if it’s contained
Self
- The locator with the relation added
inside_of
other_locator
Relatable - The locator to check if it contains this element
Self
- The locator with the relation added
left_of
- if most of A (or, more specifically, A’s bounding box) is left of B (or, more specifically, the left border of B’s bounding box) and
- if the right border of A (or, more specifically, A’s bounding box) is left of the right border of B (or, more specifically, B’s bounding box).
other_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) left of the other element(s) (located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element left of the other element(s). Elements’ (relative) position is determined by the right border (x-coordinate) of their bounding box. We don’t guarantee the order of elements with the same right border (x-coordinate). Defaults to0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be left of the other element(s) (located by other_locator). Defaults to"center"
.
Self
- The locator with the relation added
nearest_to
other_locator
Relatable - The locator to compare distance against
Self
- The locator with the relation added
or_
other_locator
Relatable - The locator to combine with
Self
- The locator with the relation added
raise_if_cycle
askui.locators.RelationIndex
0
), second (1
), third (2
) etc. element
above, below, right or left of the other element B. A’s position relative
to other elements above, below, right or left of B
(which determines its index) is determined by the relative position of its
lowest (above), highest (below), leftmost (right) or rightmost (left) point(s)
(edge of its bounding box).
Important: Which elements are counted (“indexed”) depends on the locator used, e.g.,
when using Text
only text matched is counted, and the reference_point
.
Examples:
reference_point
"center"
, A is the first (index=0
) element above B."boundary"
, A is the second (index=1
) element above B."any"
, A is the third (index=2
) element above B.
askui.locators.Text
text
str | None, optional - The text content of the ui element, e.g.,'Sign up'
. Defaults toNone
. IfNone
, the locator will match any text element.match_type
TextMatchType, optional - The type of match to use. Defaults to"similar"
.similarity_threshold
int, optional - A threshold for how similar the actual text content of the ui element needs to be to the specified text to be considered a match whenmatch_type
is"similar"
. Takes values between0
and100
(inclusive, higher is more similar). Defaults to70
.
askui.locators.TextMatchType
"similar"
uses a similarity threshold to determine if the text is a match."exact"
requires the text to be exactly the same (this is not the same as"similar"
with asimilarity_threshold
of100
as asimilarity_threshold
of100
can still allow for small differences in very long texts)."contains"
requires the text to contain (exactly) the specified text."regex"
uses a regular expression to match the text.
right_of
- if most of A (or, more specifically, A’s bounding box) is right of B (or, more specifically, the right border of B’s bounding box) and
- if the left border of A (or, more specifically, A’s bounding box) is right of the left border of B (or, more specifically, B’s bounding box).
other_locator
Relatable - Locator for an element / elements to relate toindex
RelationIndex, optional - Index of the element (located by self) right of the other element(s) (located by other_locator), e.g., the first (0
), second (1
), third (2
) etc. element right of the other element(s). Elements’ (relative) position is determined by the left border (x-coordinate) of their bounding box. We don’t guarantee the order of elements with the same left border (x-coordinate). Defaults to0
.reference_point
ReferencePoint, optional - Defines which element (located by self) is considered to be right of the other element(s) (located by other_locator). Defaults to"center"
.
Self
- The locator with the relation added