Speed Up Execution

Lazy inference is a transitive feature to speed up the execution of instructions. Following the principle: Only execute that what needs to be executed. To benefit from this feature, you have to write the commands in a certain way.

Our AI model consists of the following submodels:

Model	Tasks	Speed
Object Detector	Detect all UI elements	fast 🚀
Custom Element Detector	Search via template matching for a cropped image inside the screenshot	slow 🐌
Icon Classifier	Predict the class of an icon, e.g., a user icon	fast 🚀
Optical Character Recognition (OCR)	Convert the image of a text into text	fast 🚀
Color Classifier	Detect the colors of all elements	slow 🐌

The submodels have different execution times. It depends on wether they are executable and, therefore, executed on a GPU or a CPU. The fast models are executed on a GPU and the slow ones on a CPU.

During the execution of an instruction, the lazy inference can deactivate submodels to speed up execution. What can be deactivated can be derived from the instruction, e.g.:

• await aui.click().text().withText('Login').exec() only needs the Object Detector and the OCR but not the Icon Classifier, Color Classifier or the Custom Element Detector.
• await aui.pressKey('enter').exec() is not relying on any of the models at all as no classification is necessary. In this case. no submodel is executed.

1. Avoid CPU Inference

The submodels Custom Element Detector and Color Classifier are executed on the CPU and are slow. Theses submodels should therefore be avoided.

Don't:

await aui.click().customElement({
  customImage: '.../login_button.png', 
  name: 'login button',
}).exec();
await aui.click().customElement({
  customImage: '.../text_overview.png', 
  name: 'overview button',
}).exec();
await aui.click().text().withText('Best Practice').withColor('green').exec();

Do:

await aui.click().button().withText('Login').exec();
await aui.click().text().withText('Overview').exec();
await aui.click().text().withText('Best Practice').exec();

2. Combine a Text Filter with an Element Filter

The OCR model is applied to multiple UI elements to extract the text, e.g., links, text etc. The text filters withText(), withExactText() or containsText() are applied to all UI elements containing text. To avoid that OCR is applied on all elements you should use it in combination with an element filter.

Don't:

await aui.click().withText('See here').exec();
await aui.click().withText('Sign in').exec();

Do:

await aui.click().link().withText('See here').exec();
await aui.click().text().withText('Sign in').exec();