Best Practices
Improve Model Performance
Learn about re-teaching vs training and how to improve model performance for your specific needs.
Understanding Re-Teaching vs Training
Before diving into model improvement techniques, it’s important to understand the distinction between re-teaching and training:Re-Teaching
Re-teaching is about helping the model improve or adapt without changing its underlying neural network weights. Instead, you guide it using examples, rules, or context. It’s often about using prompts, memory, or interaction history to adjust model behavior in a targeted and efficient way. Re-teaching in AskUI includes:- Prompt Engineering: Giving clear, contextual instructions like “Click on the login button” helps models interpret intent more accurately.
- LLMs and AMs are re-teachable models, like GPT4, Anthropic Claude etc
Training
Training involves changing the internal parameters of a model - its weights - through exposure to large datasets and feedback signals. This is computationally expensive and typically happens during development, online training or in batch processes. OCR re-teaching is one example of training and AskUI supports online training.Why Use Both?
AskUI combines training and teaching because:- Teaching is agile: It empowers users to refine behavior instantly.
- Training is foundational: It builds the core capabilities of the model.
| Aspect | Teaching | Training |
|---|---|---|
| Speed | Immediate or near real-time | Takes time (minutes to hours) |
| Who does it? | Users or automation | Developers/Engineers |
| Affects Model Weights | ❌ No | ✅ Yes |
| Flexibility | High - works with new scenarios | Medium - needs structured data |
| Example in AskUI | PTA-1 prompt learning, OCR tweaks | UIDT-1 model expansion |
Improving AI Models for Your Specific Use Cases
AskUI’s AI models can be re-trained to better recognize and interact with your specific application interfaces. This customization can significantly improve automation accuracy and reliability, especially for:- Custom UI frameworks
- Industry-specific applications
- Internal enterprise tools
- Applications with unique visual elements
Benefits of Model Re-Teaching
- Improved accuracy: Models specifically trained on your interfaces perform better
- Faster execution: Reduced false positives means quicker automation
- Better reliability: More consistent results across different environments
- Support for edge cases: Better handling of unusual UI patterns
| Model Type | Model Name | Purpose | Teachable | Online Trainable |
|---|---|---|---|---|
| Locator | UIDT-1 | Locate elements & understand screen | No | Partial |
| Locator | PTA-1 | Convert prompts into one-click actions | No | Yes |
| Query | GPT-4 | Understand & respond to user queries | Yes | No |
| Query | Computer Use | Understand & respond to user queries | Yes | No |
| Large Action (act) | Computer Use | Plan and execute full workflows | Yes | No |
| Large Action (act) | UI-Tars | Plan and execute full workflows | Yes | No |
Online Training Methods
Online-Training in AskUI includes:- OCR Re-Training (UIDT-1): Our OCR engine is a composite of teachable and trainable models. While you can “teach” it by correcting recognized text, deeper improvements (like recognizing new font types or edge cases) require training on more labeled images.
- Prompt-to-Action (e.g., PTA-1): These models incorporate feedback from user corrections. For example, if a button click fails, and the user clarifies the intended target, the model updates its interpretation next time.