Skip to main content
Version: 0.3.2

Speed Up Execution

Lazy inference is a transitive feature to speed up the execution of instructions. Following the principle: Only execute that what needs to be executed. To benefit from this feature, you have to write the commands in a certain way.

Our AI model consists of the following submodels:

ModelTasksSpeed
Object DetectorDetect all UI elementsfast 🚀
Custom Element DetectorSearch via template matching for a cropped image inside the screenshotslow 🐌
Icon ClassifierPredict the class of an icon, e.g., a user iconfast 🚀
Optical Character Recognition (OCR)Convert the image of a text into textfast 🚀
Color ClassifierDetect the colors of all elementsslow 🐌

The submodels have different execution times. It depends on wether they are executable and, therefore, executed on a GPU or a CPU. The fast models are executed on a GPU and the slow ones on a CPU.

During the execution of an instruction, the lazy inference can deactivate submodels to speed up execution. What can be deactivated can be derived from the instruction, e.g.:

  • await aui.click().text().withText('Login').exec() only needs the Object Detector and the OCR but not the Icon Classifier, Color Classifier or the Custom Element Detector.
  • await aui.pressKey('enter').exec() is not relying on any of the models at all as no classification is necessary. In this case. no submodel is executed.

1. Avoid CPU Inference

The submodels Custom Element Detector and Color Classifier are executed on the CPU and are slow. Theses submodels should therefore be avoided.

Don't:
await aui.click().customElement({
customImage: '.../login_button.png',
name: 'login button',
}).exec();
await aui.click().customElement({
customImage: '.../text_overview.png',
name: 'overview button',
}).exec();
await aui.click().text().withText('Best Practice').withColor('green').exec();
Do:
await aui.click().button().withText('Login').exec();
await aui.click().text().withText('Overview').exec();
await aui.click().text().withText('Best Practice').exec();

2. Combine a Text Filter with an Element Filter

The OCR model is applied to multiple UI elements to extract the text, e.g., links, text etc. The text filters withText(), withExactText() or containsText() are applied to all UI elements containing text. To avoid that OCR is applied on all elements you should use it in combination with an element filter.

Don't:
await aui.click().withText('See here').exec();
await aui.click().withText('Sign in').exec();
Do:
await aui.click().link().withText('See here').exec();
await aui.click().text().withText('Sign in').exec();