Selecting UI Elements

UI testing is all about interacting with UI elements. UIs are built as a human interfaces and not as a machine interfaces. During the development process the underlying source code gets extended, refactored or deleted. This leads to hard to understand, unstable and unreliable element selectors. askui enables you to select UI elements based on the visual features.

Hovering over the red bounding boxes in Figure 1 shows a tooltip with the detected UI element and the detected text. As you can see, the login button was detected as a UI element button containing a UI element text with the text 'Login'. Additionally, the main three colors were detected.

If we want to click on the login button, we can write the following:

await aui.click().button().withText('Login').exec()

Figure 1: Interactive Annotation

1. Generalization vs. Specialization

A selector should be as specialized as needed and as generalized as possible. This helps a selector to be reliable against source code changes.

Let's assume we want to click on any button, then we can write the following:

await aui.click().button().exec();

In many cases, the given instruction is too general. In the example from above where multiple buttons are visible, sometimes the 'Login' button and sometimes the 'Sign in with Google' button would be clicked.

We could make the instruction more specialized by making it click the text 'Login'.

await aui.click().text().withText("Login").exec();

But if the text 'Sign in' is renamed to 'Login', the test step would fail again. Therefore, we need to further specialize it.

Now we filter for the login button:

await aui.click().button().withText("Login").exec();

This is an almost perfect mix between generalization and specialization. The login button can now be moved to the right or somewhere else on the page and the test step would click it.

The following instruction is an example of too much specialization. The dependency between the position of the two buttons can easily lead to a failing instruction when the buttons are positioned differently.

await aui
  .click()
  .button()
  .withText("Login")
  .below()
  .text()
  .withText("Sign in")
  .exec();

2. Selecting with Text

A human usually needs only the written text or visual properties of elements to use a UI. The askui library provides multiple methods to interact with text of the UI.

The OCR model that converts the image to text, like a human, sometimes makes spelling mistakes. To implement a reliable text selector, we have implemented several text filtering methods.

The withText('text') method is based on a fuzzy matching algorithm and, therefore, forgives minor spelling mistakes. This method should be used by default.

await aui.click().withText("Login").exec();

Sometimes two words are so similar, e.g., 'text' and 'test', that the previous method is selecting the wrong text. In this cases the withExactTest('text') method can be used.

await aui.click().withExactText("Login").exec();

If you have an article number, e.g., 'AN-8463', which consists of a dynamic part and a static part, then you can use the containsText('AN-') method.

await aui.click().containsText("AN-").exec();

3. Visual Relations

UIs are designed the same across websites to make the design concepts easier to understand. E.g., a relation between a textfield and a label is visually categorized in 3 classes:

Figure 2: textfield label relation

1. Label is above the textfield.
2. Label is left of the textfield.
3. Label is in the textfield.

The askui library provides the following visual relation selectors: nearestTo(), above(), below(), rightOf(), leftOf(), contains()and in().

To fill out the form, the nearestTo() relation should be used by default:

await aui
  .typeIn("text to type")
  .textfield()
  .nearestTo()
  .text()
  .withText("One")
  .exec();

This prevents that the instruction fails, when the developer decides to change the position of the label.

As an alternative of the first example, the relation below() could be used.

await aui
  .typeIn("text to type")
  .textfield()
  .below()
  .text()
  .withText("One")
  .exec();

For the second example, the rightOf() relation can be used:

await aui
  .typeIn("text to type")
  .textfield()
  .rightOf()
  .text()
  .withText("Two")
  .exec();

The third example is trickier, because the textfield contains (contains()) the label.

await aui
  .typeIn("text to type")
  .textfield()
  .contains()
  .text()
  .withText("Third")
  .exec();

The in() relation can be used, e.g., to click on the 'Home' text in the header in the example shown in Figure 1.

await aui.click().text().withText("Home").in().header().exec();

4. Selecting Undetected Elements

Sometimes, elements are not detected or classified well. If you hover over the Discord icon right of the 'Get Early Access' button, you will see that the icon is not classified as Discord.

Figure 3: Custom Elements
In these cases and cases where elements are not detected at all, custom elements can be used to interact with them.

Take a screenshot and crop out the element. Then, you can use the customElement() method:

await aui
  .click()
  .customElement({ customImage: "path/to/cropped_out_image", name: "discord" })
  .withText("discord")
  .exec();