You can use the agent to extract structured sidebar information from your application UI. This is useful for validating navigation, generating documentation, or automating UI tests.

To do this, define a response schema that models the sidebar structure as a recursive tree. For example:

class SidebarItem(ResponseSchemaBase):
    name: str
    children: List["SidebarItem"]

class Sidebare(ResponseSchemaBase):
    sidebar: List["SidebarItem"]

This schema is recursive because each SidebarItem can contain a list of more SidebarItem objects as its children, forming a tree structure.

Then, use the agent to extract the sidebar below a specific section, such as “Introduction”:

with VisionAgent() as agent:
    result = agent.get(
        "Can you extract me the sidebar below section 'Introduction'?",
        response_schema=Sidebare
    )
    print(result.model_dump_json(indent=2))

Example

Suppose you have the following sidebar in your application:

The extracted output would look like:

{
  "sidebar": [
    {
      "name": "Overview",
      "children": []
    },
    {
      "name": "How AskUI Works",
      "children": []
    },
    {
      "name": "Example Automations",
      "children": []
    }
  ]
}

This approach allows you to retrieve the sidebar as a structured JSON object, representing a recursive tree, making it easy to inspect or use in further automation.

Tip:
Adjust the schema to match your sidebar’s recursive tree structure, and use clear instructions to get accurate results.