WIP: Best Practices for Reliable Automation

​

1. Misspellings of Words

The OCR model sometimes misreads characters, especially in certain fonts or noisy images. This can result in words being misclassified or misspelled, which then causes the automation to fail when it searches for exact matches.

​

Solutions

​

1. Re-Teach the OCR Model Using AskUI’s OCR Re-Teaching App

You can directly correct OCR predictions and improve model accuracy by training your workspace-specific model.

​

Steps:

1. Start the AskUI shell:

   bash
   
   
   
   askui-shell
   

2. Launch the OCR Teaching App:

   bash
   
   
   
   AskUI-StartOCRTeaching
   

3. Upload a screenshot containing the misclassified word (e.g., “Hallo”).

4. Switch to Character-Level Mode for precise corrections.

5. Select the wrongly detected word (HaII0) and replace it with the correct label: Hallo.

6. Click “Copy Model” to copy the newly trained model ID.

7. In your automation code (e.g., askui-helper.ts), update the exec() call to use the new model:

ts

CopyEdit

await aui.click().text("Hallo").exec([{

    "task": "e2e_ocr",

    "architecture": "easy_ocr",

    "version": "1",

    "interface": "online_learning",

    "useCase": "<your-workspace-id>",

    "tags": ["char_level"]

}])

​

Text Detection Issues

​

Icon Text Merging

Problem: Sometimes, Text Detector/ annotation tool, merges an icon and texts into one, even though they look merged on screen.

Example: Say you want to click just the name “Alice Johnson” field or just the position field in a interface - but OCR detects them as one long string:

✅ Expected Behavior

🖼️ Icon and Text are detected separately:

🧑 ✅ Name ✅  🤖 ✅ Role ✅

👍 Works with click().text("Name") or click().text("Name")

❌ Actual Issue

🖼️ Icon and text are detected together:

🧑 Name ❌  🤖 ✅ Role ✅

👎 Can’t find click().text("Name").

Solutions:

​

Option 1: Re-Teach the OCR Model to ignore the icon.

1. Start Re-Teachting App
2. Teach the Text Recognition to ignore the icon
3. Use the Custom Model.

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "<your-workspace-id>",

        "tags": ["word_level"]

    }])

​

Option 2: Use Default Word-Level Detection (Best Practice)

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "00000000_0000_0000_0000_000000000000",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.

​

Option 3: Use Custom Model Word-Level Detection

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "<your-workspace-id>",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.y.

​

Merged Texts

Problem: Sometimes, Text Detector/ annotation tool, merges two separate texts into one, even though they look clearly split on screen.

Example: Say you want to click just the name “Alice Johnson” field or just the position field in a interface - but OCR detects them as one long string:

✅ Expected Behavior

🖼️ Text fields detected separately:

Alice Johnson ✅  Software Engineer ✅

👍 Works with text("Alice Johnson") or text("Software Engineer")

❌ Actual Issue

🖼️ Texts merged into one block:

Alice Johnson Software Engineer❌

👎 Can’t find either one on its own.

Solutions:

​

Option 1: Use Default Word-Level Detection (Best Practice)

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "00000000_0000_0000_0000_000000000000",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.

​

Option 2: Use Custom Model Word-Level Detection

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "<your-workspace-id>",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.

​

Option 3: Use Relative Positioning (Fallback)

await aui.moveMouseRelativeTo(0, left).containsText("Name").exec()

Useful when text detection still merges blocks. Navigates close enough to act anyway.

​

Text Seperation

Problem: Sometimes, Text Detector/ annotation tool, septerates a text into two texts, even though they look clearly merged on screen.

Example: Say you want to click just the name “Alice Johnson” field or just the position field in a interface - but OCR detects them as two words:

✅ Expected Behavior

🖼️ Words are detected as one sentence:

Alice Johnson ✅ 

👍 Works with text("Alice Johnson")

❌ Actual Issue

🖼️ Words are detected as seperated texts:

Alice❌ Johnson❌

👎 Can’t find either text("Alice Johnson") on its own.

Solutions:

​

Option 1: Use Default Word-Level Detection (Best Practice)

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "00000000_0000_0000_0000_000000000000",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.

​

Option 2: Use Custom Model Word-Level Detection

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "<your-workspace-id>",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.

​

Vertical Text Merging

Problem: Sometimes, Text Detector/ annotation tool, merges two lines to one text, even though they look clearly as two lines on screen.

Example: Say you want to click just the name “Alice Johnson” field or just the position field in a interface - but OCR detects them as one:

✅ Expected Behavior

🖼️ Texts are detected as two lines:

Alice Johnson ✅ 

👍 Works with text("Alice Johnson")

❌ Actual Issue

🖼️ Texts are detected as one text:

<no words recognized>❌

👎 Can’t find either text("Alice Johnson") on its own.

Solutions:

​

Option 1: Use Default Word-Level Detection (Best Practice)

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "00000000_0000_0000_0000_000000000000",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.

​

Option 2: Use Custom Model Word-Level Detection

await aui.click().text("Alice Johnson").exec([{

        "task": "e2e_ocr",

        "architecture": "easy_ocr",

        "version": "1",

        "interface": "online_learning",

        "useCase": "<your-workspace-id>",

        "tags": ["word_level"]

    }])

Breaks up merged strings into individual words.

​

Single Character not Detected

Problem: Sometimes, Text Detector/ annotation tool, does not detect single charactors, even though they look clearly on screen.

Example: Say you want to click just the char “3” - but OCR does not detects them:

✅ Expected Behavior

🖼️ Single chars are detected*:*

1 ✅ 2 ✅ 3 ✅ 

👍 Works with text("2")

❌ Actual Issue

🖼️ Char 2 is not detected*:*

1 ✅ 2 ❌ 3 ✅ 

👎 Can’t find either text("2") on its own.

Solutions:

​

Option 1: AI Element

1. Follow tutorial

​

Text not Detected

Problem: Sometimes, for no apparent reason, Text Detector/ annotation tool does not detect a text, even though they see it clearly on screen.

Example: Say you want to click just the name “Alice Johnson” field - but OCR does not detects the text at all:

✅ Expected Behavior

🖼️ Text was detected:

Alice Johnson ✅ 

👍 Works with text("Alice Johnson")

❌ Actual Issue

🖼️ Text wasn’t detected

Alice Johnson❌

👎 Can’t find either text("Alice Johnson") on its own.

Solutions:

​

Option 1: AI Element

1. Select the Text as AI Element