Retraining AI Models for Your Specific Use Cases

AskUI’s AI models can be retrained to better recognize and interact with your specific application interfaces. This customization can significantly improve automation accuracy and reliability, especially for:

  • Custom UI frameworks
  • Industry-specific applications
  • Internal enterprise tools
  • Applications with unique visual elements

Benefits of Model Retraining

  • Improved accuracy: Models specifically trained on your interfaces perform better
  • Faster execution: Reduced false positives means quicker automation
  • Better reliability: More consistent results across different environments
  • Support for edge cases: Better handling of unusual UI patterns

Following models are supported:

Model TypeModel NamePurposeTeachableOnline Trainable
GroundingUIDT-1Locate elements & understand screenNoPartial
GroundingPTA-1Convert prompts into one-click actionsNoYes
QueryGPT-4Understand & respond to user queriesYesNo
QueryComputer UseUnderstand & respond to user queriesYesNo
Large Action (act)Computer UsePlan and execute full workflowsYesNo
Large Action (act)UI-TarsPlan and execute full workflowsYesNo

Teaching

Teaching is about helping the model improve or adapt without changing its underlying neural network weights.

Instead, you guide it using examples, rules, or context. It’s often about using prompts, memory, or interaction history to adjust model behavior in a targeted and efficient way.

Teaching in AskUI includes:

  • Prompt Engineering: Giving clear, contextual instructions like “Click on the login button” helps models interpret intent more accurately.
  • LLMs and LAMs are teacheable models, like GPT4, Anthropic Claude etc

Training

Training involves changing the internal parameters of a model - its weights - through exposure to large datasets and feedback signals.

This is computationally expensive and typically happens during development, online training or in batch processes. OCR re-teaching is on example of training and AskUI supports online training.

Online-Training in AskUI includes:

  • OCR Re-Training (UIDT-1):

    Our OCR engine is a composite of teachable and trainable models. While you can “teach” it by correcting recognized text, deeper improvements (like recognizing new font types or edge cases) require training on more labeled images.

  • Prompt-to-Action (e.g., PTA-1):

    These models incorporate feedback from user corrections. For example, if a button click fails, and the user clarifies the intended target, the model updates its interpretation next time.

Why Use Both ?

AskUI combines training and teaching because:

  • Teaching is agile: It empowers users to refine behavior instantly.
  • Training is foundational: It builds the core capabilities of the model.
AspectTeachingTraining
SpeedImmediate or near real-timeTakes time (minutes to hours)
Who does it?Users or automationDevelopers/Engineers
Affects Model Weights❌ No✅ Yes
FlexibilityHigh - works with new scenariosMedium - needs structured data
Example in AskUIPTA-1 prompt learning, OCR tweaksUIDT-1 model expansion