​
askui.locators.AiElement

class AiElement(ImageBase)

Locator for finding ui elements by data (e.g., image) collected with the AskUIRemoteDeviceSnippingTool using the name assigned to the AI element during snipping to retrieve the data used for locating the ui element(s).

Arguments:

  • name str - Name of the AI element
  • threshold float, optional - A threshold for how similar UI elements need to be to the image to be considered a match. Takes values between 0.0 (= all elements are recognized) and 1.0 (= elements need to look exactly like defined). Defaults to 0.5. Important: The threshold impacts the prediction quality.
  • stop_threshold float | None, optional - A threshold for when to stop searching for UI elements similar to the image. As soon as UI elements have been found that are at least as similar as the stop_threshold, the search stops. Should be greater than or equal to threshold. Takes values between 0.0 and 1.0. Defaults to value of threshold if not provided. Important: The stop_threshold impacts the prediction speed.
  • mask list[tuple[float, float]] | None, optional - A polygon to match only a certain area of the image. Must have at least 3 points. Defaults to None.
  • rotation_degree_per_step int, optional - A step size in rotation degree. Rotates the image by rotation_degree_per_step until 360° is exceeded. Range is between 0° - 360°. Defaults to 0°. Important: This increases the prediction time quite a bit. So only use it when absolutely necessary.
  • name str | None, optional - Name for the image. Defaults to random name.
  • image_compare_format Literal[“RGB”, “grayscale”, “edges”], optional - A color compare style. Defaults to 'grayscale'. Important: The image_compare_format impacts the prediction time as well as quality. As a rule of thumb, 'edges' is likely to be faster than 'grayscale' and 'grayscale' is likely to be faster than 'RGB'. For quality it is most often the other way around.

​
askui.locators.CircularDependencyError

class CircularDependencyError(ValueError)

Exception raised for circular dependencies in locator relations.

​
askui.locators.Element

class Element(Locator)

Locator for finding ui elements by their class.

Arguments:

  • class_name Literal[“text”, “textfield”] | None, optional - The class of the ui element, e.g., 'text' or 'textfield'. Defaults to None.

Examples:

from askui import locators as loc
# locates a text elementAdd 
text = loc.Element(class_name="text")
# locates a textfield element
textfield = loc.Element(class_name="textfield")
# locates any ui element detected
element = loc.Element()

​
askui.locators.Image

class Image(ImageBase)

Locator for finding ui elements by an image.

Arguments:

  • image Union[PIL.Image.Image, pathlib.Path, str] - The image to match against (PIL Image, path, or string)
  • threshold float, optional - A threshold for how similar UI elements need to be to the image to be considered a match. Takes values between 0.0 (= all elements are recognized) and 1.0 (= elements need to look exactly like defined). Defaults to 0.5. Important: The threshold impacts the prediction quality.
  • stop_threshold float | None, optional - A threshold for when to stop searching for UI elements similar to the image. As soon as UI elements have been found that are at least as similar as the stop_threshold, the search stops. Should be greater than or equal to threshold. Takes values between 0.0 and 1.0. Defaults to value of threshold if not provided. Important: The stop_threshold impacts the prediction speed.
  • mask list[tuple[float, float]] | None, optional - A polygon to match only a certain area of the image. Must have at least 3 points. Defaults to None.
  • rotation_degree_per_step int, optional - A step size in rotation degree. Rotates the image by rotation_degree_per_step until 360° is exceeded. Range is between 0° - 360°. Defaults to 0°. Important: This increases the prediction time quite a bit. So only use it when absolutely necessary.
  • name str | None, optional - Name for the image. Defaults to random name.
  • image_compare_format Literal[“RGB”, “grayscale”, “edges”], optional - A color compare style. Defaults to 'grayscale'. Important: The image_compare_format impacts the prediction time as well as quality. As a rule of thumb, 'edges' is likely to be faster than 'grayscale' and 'grayscale' is likely to be faster than 'RGB'. For quality it is most often the other way around.

Examples:

from askui import locators as loc
from PIL import Image as PILImage
from pathlib import Path

# locates an element using an image of the element
# passed as `str` path
image = loc.Image("path/to/image.png")
# passed as `pathlib.Path`
image = loc.Image(Path("path/to/image.png"))
# passed as `PIL.Image.Image`
image = loc.Image(PILImage.open("path/to/image.png"))
# passed as data url `str`
image = loc.Image("data:image/png;base64,...")

​
askui.locators.Locator

class Locator(Relatable, ABC)

Abstract base class for all locators. Cannot be instantiated directly. Subclassed by all locators, e.g., Prompt, Text, Image, etc.

​
askui.locators.Prompt

class Prompt(Locator)

Locator for finding ui elements by a textual prompt / description of a ui element, e.g., “green sign up button”.

Arguments:

  • prompt str - A textual prompt / description of a ui element, e.g., "green sign up button"

Examples:

from askui import locators as loc
# locates a green sign up button
button = loc.Prompt("green sign up button")
# locates an email text field, e.g., with label "Email" or a placeholder "john.doe@example.com"
textfield = loc.Prompt("email text field")
# locates the avatar in the right hand corner of the application
avatar_top_right_corner = loc.Prompt("avatar in the top right corner of the application")

​
askui.locators.ReferencePoint

ReferencePoint = Literal["center", "boundary", "any"]

Defines under which conditions an element A is considered to be above, below, right or left of another element B.

  • "center": A is considered to be above, below, right or left of B if it is above, below, right or left of A’s center (in a straight vertical or horizontal line).

Examples:

A being above B (imaginary straight vertical line also shown):

===========
|    A    |
===========
     |
===========
|    B    |
===========

     ===========
     |    A    |
     ===========
     |
===========
|    B    |
===========

A NOT being above B (imaginary straight vertical line also shown):

     |===========
     |     A    |
     |===========
     |
===========
|    B    |
===========

     |      ===========
     |      |    A    |
     |      ===========
     |
===========
|    B    |
===========

     |
     |   
===========
|    B    |
===========

===========
|    A    |
===========
  • "boundary": A is considered to be above, below, right or left of B if it is above, below, right or left of (any point of the bounding box of) A (in a straight vertical or horizontal line).

Examples:

A being above B (imaginary straight vertical line also shown):

|    ===========
|    |    A    |
|    ===========
|         |
===========
|    B    |
===========

A NOT being above B (imaginary straight vertical line also shown):

|         | ===========
|         | |    A    |
|         | ===========
|         |
===========
|    B    |
===========

|         |
|         |
===========
|    B    |
===========

===========
|    A    |
===========
  • "any": A is considered to be above, below, right or left of B if it is above, below, right or left of B no matter if it can be reached in a straight vertical or horizontal line from (a point of the bounding box of) A.

Examples:

A being above B:

            ===========
            |    A    |
            ===========

===========
|    B    |
===========

            ===========
=========== |    A    |
|    B    | ===========
===========

A NOT being above B:

===========
|    B    |
===========

===========
|    A    |
===========

=========== ===========
|    B    | |    A    |
=========== ===========

​
askui.locators.Relatable

class Relatable(ABC)

Abstract base class for locators that can be related to other locators, e.g., spatially, logically etc. Cannot be instantiated directly. Subclassed by all (relatable) locators, e.g., Prompt, Text, Image, etc.

​
above_of

def above_of(
    other_locator: "Relatable",
    index: RelationIndex = 0,
    reference_point: ReferencePoint = "boundary"
) -> Self

Defines the element (located by self) to be above another element / other elements (located by other_locator).

An element A is considered to be above another element / other elements B

  • if most of A (or, more specifically, A’s bounding box) is above B (or, more specifically, the top border of B’s bounding box) and
  • if the bottom border of A (or, more specifically, A’s bounding box) is above the bottom border of B (or, more specifically, B’s bounding box).

Arguments:

  • other_locator Relatable - Locator for an element / elements to relate to
  • index RelationIndex, optional - Index of the element (located by self) above the other element(s) (located by other_locator), e.g., the first (0), second (1), third (2) etc. element above the other element(s). Elements’ (relative) position is determined by the bottom border (y-coordinate) of their bounding box. We don’t guarantee the order of elements with the same bottom border (y-coordinate). Defaults to 0.
  • reference_point ReferencePoint, optional - Defines which element (located by self) is considered to be above the other element(s) (located by other_locator). Defaults to "boundary".

Returns:

  • Self - The locator with the relation added

Examples:


===========
|    A    |
===========
===========
|    B    |
===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element above ("center" of)
# text "B"
text = loc.Text().above_of(loc.Text("B"), reference_point="center")


       ===========
       |    A    |
       ===========
===========
|    B    |
===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element above
# ("boundary" of / any point of) text "B" 
# (reference point "center" won't work here)
text = loc.Text().above_of(loc.Text("B"), reference_point="boundary")


            ===========
            |    A    |
            ===========
===========
|    B    |
===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element above text "B" 
# (reference point "center" or "boundary" won't work here)
text = loc.Text().above_of(loc.Text("B"), reference_point="any")


            ===========
            |    A    |
            ===========
===========
|    B    |
===========
===========
|    C    |
===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element above text "C" 
# (reference point "center" or "boundary" won't work here)
text = loc.Text().above_of(loc.Text("C"), index=1, reference_point="any")


        ===========
        |    A    |
        ===========
            ===========
=========== |    B    |
|         | ===========
|    C    |
|         |
===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element above text "C"
# (reference point "center" or "boundary" won't work here)
text = loc.Text().above_of(loc.Text("C"), index=1, reference_point="any")
# locates also text "A" as it is the first (index 0) element above text "C"
# with reference point "boundary"
text = loc.Text().above_of(loc.Text("C"), index=0, reference_point="boundary")

​
and_

def and_(other_locator: "Relatable") -> Self

Logical and operator to combine multiple locators, e.g., to require an element to match multiple locators.

Arguments:

  • other_locator Relatable - The locator to combine with

Returns:

  • Self - The locator with the relation added

Examples:

from askui import locators as loc

# Searches for an element that contains the text "Google" and is a
# multi-colored Google logo (instead of, e.g., simply some text that says
# "Google")
icon_user = loc.Element().containing(
    loc.Text("Google").and_(loc.Prompt("Multi-colored Google logo"))
)

​
below_of

def below_of(
    other_locator: "Relatable",
    index: RelationIndex = 0,
    reference_point: ReferencePoint = "boundary"
) -> Self

Defines the element (located by self) to be below another element / other elements (located by other_locator).

An element A is considered to be below another element / other elements B

  • if most of A (or, more specifically, A’s bounding box) is below B (or, more specifically, the bottom border of B’s bounding box) and
  • if the top border of A (or, more specifically, A’s bounding box) is below the top border of B (or, more specifically, B’s bounding box).

Arguments:

  • other_locator Relatable - Locator for an element / elements to relate to
  • index RelationIndex, optional - Index of the element (located by self) below the other element(s) (located by other_locator), e.g., the first (0), second (1), third (2) etc. element below the other element(s). Elements’ (relative) position is determined by the top border (y-coordinate) of their bounding box. We don’t guarantee the order of elements with the same top border (y-coordinate). Defaults to 0.
  • reference_point ReferencePoint, optional - Defines which element (located by self) is considered to be below the other element(s) (located by other_locator). Defaults to "boundary".

Returns:

  • Self - The locator with the relation added

Examples:


===========
|    B    |
===========
===========
|    A    |
===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element below ("center" of)
# text "B"
text = loc.Text().below_of(loc.Text("B"), reference_point="center")


===========
|    B    |
===========
       ===========
       |    A    |
       ===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element below 
# ("boundary" of / any point of) text "B" 
# (reference point "center" won't work here)
text = loc.Text().below_of(loc.Text("B"), reference_point="boundary")


===========
|    B    |
===========
            ===========
            |    A    |
            ===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element below text "B"
# (reference point "center" or "boundary won't work here)
text = loc.Text().below_of(loc.Text("B"), reference_point="any")


===========
|    C    |
===========
===========
|    B    |
===========
            ===========
            |    A    |
            ===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element below text "C" 
# (reference point "center" or "boundary" won't work here)
text = loc.Text().below_of(loc.Text("C"), index=1, reference_point="any")


=========== 
|         | 
|    C    |
|         |===========
===========|    B    |
           ===========
        ===========
        |    A    |
        ===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element below text "C"
# (reference point "any")
text = loc.Text().below_of(loc.Text("C"), index=1, reference_point="any")
# locates also text "A" as it is the first (index 0) element below text "C"
# with reference point "boundary"
text = loc.Text().below_of(loc.Text("C"), index=0, reference_point="boundary")

​
containing

def containing(other_locator: "Relatable") -> Self

Defines the element (located by self) to contain another element (located by other_locator).

Arguments:

  • other_locator Relatable - The locator to check if it’s contained

Returns:

  • Self - The locator with the relation added

Examples:

---------------------------
|     textfield           |
|  ---------------------  |
|  |  placeholder text |  |
|  ---------------------  |
|                         |
---------------------------

from askui import locators as loc

# Returns the textfield because it contains the placeholder text
textfield = loc.Element("textfield").containing(loc.Text("placeholder"))

​
inside_of

def inside_of(other_locator: "Relatable") -> Self

Defines the element (located by self) to be inside of another element (located by other_locator).

Arguments:

  • other_locator Relatable - The locator to check if it contains this element

Returns:

  • Self - The locator with the relation added

Examples:

---------------------------
|     textfield           |
|  ---------------------  |
|  |  placeholder text |  |
|  ---------------------  |
|                         |
---------------------------

from askui import locators as loc

# Returns the placeholder text of the textfield
placeholder_text = loc.Text("placeholder").inside_of(
    loc.Element("textfield")
)

​
left_of

def left_of(
    other_locator: "Relatable",
    index: RelationIndex = 0,
    reference_point: ReferencePoint = "center"
) -> Self

Defines the element (located by self) to be left of another element / other elements (located by other_locator).

An element A is considered to be left of another element / other elements B

  • if most of A (or, more specifically, A’s bounding box) is left of B (or, more specifically, the left border of B’s bounding box) and
  • if the right border of A (or, more specifically, A’s bounding box) is left of the right border of B (or, more specifically, B’s bounding box).

Arguments:

  • other_locator Relatable - Locator for an element / elements to relate to
  • index RelationIndex, optional - Index of the element (located by self) left of the other element(s) (located by other_locator), e.g., the first (0), second (1), third (2) etc. element left of the other element(s). Elements’ (relative) position is determined by the right border (x-coordinate) of their bounding box. We don’t guarantee the order of elements with the same right border (x-coordinate). Defaults to 0.
  • reference_point ReferencePoint, optional - Defines which element (located by self) is considered to be left of the other element(s) (located by other_locator). Defaults to "center".

Returns:

  • Self - The locator with the relation added

Examples:


=========== ===========
|    A    | |    B    |
=========== ===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element left of ("center"
# of) text "B"
text = loc.Text().left_of(loc.Text("B"), reference_point="center")


            =========== 
=========== |    B    |
|    A    | =========== 
===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element left of ("boundary"
# of / any point of) text "B"
# (reference point "center" won't work here)
text = loc.Text().left_of(loc.Text("B"), reference_point="boundary")


            =========== 
            |    B    |
            =========== 
===========              
|    A    |
=========== 

from askui import locators as loc
# locates text "A" as it is the first (index 0) element left of text "B" 
# (reference point "center" or "boundary won't work here)
text = loc.Text().left_of(loc.Text("B"), reference_point="any")


===========
|    A    |
===========
            =========== ===========
            |    B    | |    C    |
            =========== ===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element left of text "C" 
# (reference point "center" or "boundary" won't work here)
text = loc.Text().left_of(loc.Text("C"), index=1, reference_point="any")


            ===========
            |    B    |
=========== =========== 
|    A    |        ===========         
===========        |    C    |         
                   ===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element left of text "C"
# (reference point "any")
text = loc.Text().left_of(loc.Text("C"), index=1, reference_point="any")
# locates also text "A" as it is the first (index 0) element right of text
# "C" with reference point "boundary"
text = loc.Text().right_of(loc.Text("C"), index=0, reference_point="boundary")

​
nearest_to

def nearest_to(other_locator: "Relatable") -> Self

Defines the element (located by self) to be the nearest to another element (located by other_locator).

Arguments:

  • other_locator Relatable - The locator to compare distance against

Returns:

  • Self - The locator with the relation added

Examples:

--------------
|    text    |
--------------
---------------
| textfield 1 |
---------------




---------------
| textfield 2 |
---------------

from askui import locators as loc

# Returns textfield 1 because it is nearer to the text than textfield 2
textfield = loc.Element("textfield").nearest_to(loc.Text())

​
or_

def or_(other_locator: "Relatable") -> Self

Logical or operator to combine multiple locators, e.g., to provide a fallback if no element is found for one of the locators.

Arguments:

  • other_locator Relatable - The locator to combine with

Returns:

  • Self - The locator with the relation added

Examples:

from askui import locators as loc

# Searches for element using a description and if the element cannot be
# found, searches for it using an image
search_icon = loc.Prompt("search icon").or_(
    loc.Image("search_icon.png")
)

​
raise_if_cycle

def raise_if_cycle() -> None

Raises CircularDependencyError if the relations form a cycle (see Cycle (graph theory)).

​
askui.locators.RelationIndex

RelationIndex = Annotated[int, Field(ge=0)]

Index of the element A above, below, right or left of the other element B, e.g., the first (0), second (1), third (2) etc. element above, below, right or left of the other element B. A’s position relative to other elements above, below, right or left of B (which determines its index) is determined by the relative position of its lowest (above), highest (below), leftmost (right) or rightmost (left) point(s) (edge of its bounding box).

Important: Which elements are counted (“indexed”) depends on the locator used, e.g., when using Text only text matched is counted, and the reference_point.

Examples:

===========
|    A    | ===========
=========== |    B    |
            ===========
                   ===========                  
                   |    C    |                  
    ===========    ===========                  
    |    D    |
    ===========

For reference_point

  • "center", A is the first (index=0) element above B.
  • "boundary", A is the second (index=1) element above B.
  • "any", A is the third (index=2) element above B.

​
askui.locators.Text

class Text(Element)

Locator for finding text elements by their textual content.

Arguments:

  • text str | None, optional - The text content of the ui element, e.g., 'Sign up'. Defaults to None. If None, the locator will match any text element.
  • match_type TextMatchType, optional - The type of match to use. Defaults to "similar".
  • similarity_threshold int, optional - A threshold for how similar the actual text content of the ui element needs to be to the specified text to be considered a match when match_type is "similar". Takes values between 0 and 100 (inclusive, higher is more similar). Defaults to 70.

Examples:

from askui import locators as loc
# locates a text element with text similar to "Sign up", e.g., "Sign up" or "Sign Up" or "Sign-Up"
text = loc.Text("Sign up")
# if it does not find an element, you can try decreasing the similarity threshold (default is `70`)
text = loc.Text("Sign up", match_type="similar", similarity_threshold=50)
# if it also locates "Sign In", you can try increasing the similarity threshold (default is `70`)
text = loc.Text("Sign up", match_type="similar", similarity_threshold=80)
# or use `match_type="exact"` to require an exact match (does not match other variations of "Sign up", e.g., "Sign Up" or "Sign-Up")
text = loc.Text("Sign up", match_type="exact")
# locates a text element starting with "Sign" or "sign" using a regular expression
text = loc.Text("^[Ss]ign.*", match_type="regex")
# locates a text element containing "Sign" (exact match)
text = loc.Text("Sign", match_type="contains")

​
askui.locators.TextMatchType

TextMatchType = Literal["similar", "exact", "contains", "regex"]

The type of match to use.

  • "similar" uses a similarity threshold to determine if the text is a match.
  • "exact" requires the text to be exactly the same (this is not the same as "similar" with a similarity_threshold of 100 as a similarity_threshold of 100 can still allow for small differences in very long texts).
  • "contains" requires the text to contain (exactly) the specified text.
  • "regex" uses a regular expression to match the text.

​
right_of

def right_of(
    other_locator: "Relatable",
    index: RelationIndex = 0,
    reference_point: ReferencePoint = "center"
) -> Self

Defines the element (located by self) to be right of another element / other elements (located by other_locator).

An element A is considered to be right of another element / other elements B

  • if most of A (or, more specifically, A’s bounding box) is right of B (or, more specifically, the right border of B’s bounding box) and
  • if the left border of A (or, more specifically, A’s bounding box) is right of the left border of B (or, more specifically, B’s bounding box).

Arguments:

  • other_locator Relatable - Locator for an element / elements to relate to
  • index RelationIndex, optional - Index of the element (located by self) right of the other element(s) (located by other_locator), e.g., the first (0), second (1), third (2) etc. element right of the other element(s). Elements’ (relative) position is determined by the left border (x-coordinate) of their bounding box. We don’t guarantee the order of elements with the same left border (x-coordinate). Defaults to 0.
  • reference_point ReferencePoint, optional - Defines which element (located by self) is considered to be right of the other element(s) (located by other_locator). Defaults to "center".

Returns:

  • Self - The locator with the relation added

Examples:


=========== ===========
|    B    | |    A    |
=========== ===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element right of ("center"
# of) text "B"
text = loc.Text().right_of(loc.Text("B"), reference_point="center")


=========== 
|    B    | 
=========== ===========
            |    A    |
            ===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element right of 
# ("boundary" of / any point of) text "B" 
# (reference point "center" won't work here)
text = loc.Text().right_of(loc.Text("B"), reference_point="boundary")


=========== 
|    B    |
===========
            ===========
            |    A    |
            ===========

from askui import locators as loc
# locates text "A" as it is the first (index 0) element right of text "B" 
# (reference point "center" or "boundary" won't work here)
text = loc.Text().right_of(loc.Text("B"), reference_point="any")


                        ===========
                        |    A    |
                        ===========
=========== ===========
|    C    | |    B    |
=========== ===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element right of text "C" 
# (reference point "center" or "boundary" won't work here)
text = loc.Text().right_of(loc.Text("C"), index=1, reference_point="any")


        ===========
        |    B    |
        =========== ===========
===========         |    A    |
|    C    |         ===========
===========

from askui import locators as loc
# locates text "A" as it is the second (index 1) element right of text "C"
# (reference point "any")
text = loc.Text().right_of(loc.Text("C"), index=1, reference_point="any")
# locates also text "A" as it is the first (index 0) element right of text
# "C" with reference point "boundary"
text = loc.Text().right_of(loc.Text("C"), index=0, reference_point="boundary")