Speed Up Execution
Lazy inference is a transitive feature to speed up the execution of instructions. Following the principle: Only execute that what needs to be executed. To benefit from this feature, you have to write the commands in a certain way.
Our AI model consists of the following submodels:
Model | Tasks | Speed |
---|---|---|
Object Detector | Detect all UI elements | fast 🚀 |
Custom Element Detector | Search via template matching for a cropped image inside the screenshot | slow 🐌 |
Icon Classifier | Predict the class of an icon, e.g., a user icon | fast 🚀 |
Optical Character Recognition (OCR) | Convert the image of a text into text | fast 🚀 |
Color Classifier | Detect the colors of all elements | slow 🐌 |
The submodels have different execution times. It depends on wether they are executable and, therefore, executed on a GPU or a CPU. The fast models are executed on a GPU and the slow ones on a CPU.
During the execution of an instruction, the lazy inference can deactivate submodels to speed up execution. What can be deactivated can be derived from the instruction, e.g.:
await aui.click().text().withText('Login').exec()
only needs the Object Detector and the OCR but not the Icon Classifier, Color Classifier or the Custom Element Detector.await aui.pressKey('enter').exec()
is not relying on any of the models at all as no classification is necessary. In this case. no submodel is executed.
1. Avoid CPU Inference
The submodels Custom Element Detector and Color Classifier are executed on the CPU and are slow. Theses submodels should therefore be avoided.
Don't:await aui.click().customElement({
customImage: '.../login_button.png',
name: 'login button',
}).exec();
await aui.click().customElement({
customImage: '.../text_overview.png',
name: 'overview button',
}).exec();
await aui.click().text().withText('Best Practice').withColor('green').exec();
await aui.click().button().withText('Login').exec();
await aui.click().text().withText('Overview').exec();
await aui.click().text().withText('Best Practice').exec();
2. Combine a Text Filter with an Element Filter
The OCR model is applied to multiple UI elements to extract the text, e.g., links, text etc. The text filters withText()
, withExactText()
or containsText()
are applied to all UI elements containing text. To avoid that OCR is applied on all elements you should use it in combination with an element filter.
await aui.click().withText('See here').exec();
await aui.click().withText('Sign in').exec();
await aui.click().link().withText('See here').exec();
await aui.click().text().withText('Sign in').exec();