​
askui.tools.AgentOs

class AgentOs(ABC)

Abstract base class for Agent OS. Cannot be instantiated directly.

This class defines the interface for operating system interactions including mouse control, keyboard input, and screen capture functionality. Implementations should provide concrete functionality for these abstract methods.

​
click

def click(
    button: Literal["left", "middle", "right"] = "left",
    count: int = 1
) -> None

Simulates clicking a mouse button.

Arguments:

  • button Literal[“left”, “middle”, “right”], optional - The mouse button to click. Defaults to "left".
  • count int, optional - Number of times to click. Defaults to 1.

​
connect

def connect() -> None

Establishes a connection to the Agent OS.

This method is called before performing any OS-level operations. It handles any necessary setup or initialization required for the OS interaction.

​
disconnect

def disconnect() -> None

Terminates the connection to the Agent OS.

This method is called after all OS-level operations are complete. It handles any necessary cleanup or resource release.

​
keyboard_pressed

def keyboard_pressed(
    key: PcKey | ModifierKey,
    modifier_keys: list[ModifierKey] | None = None
) -> None

Simulates pressing and holding a keyboard key.

Arguments:

  • key PcKey | ModifierKey - The key to press.
  • modifier_keys list[ModifierKey] | None, optional - List of modifier keys to press along with the main key. Defaults to None.

​
keyboard_release

def keyboard_release(
    key: PcKey | ModifierKey,
    modifier_keys: list[ModifierKey] | None = None
) -> None

Simulates releasing a keyboard key.

Arguments:

  • key PcKey | ModifierKey - The key to release.
  • modifier_keys list[ModifierKey] | None, optional - List of modifier keys to release along with the main key. Defaults to None.

​
keyboard_tap

def keyboard_tap(
    key: PcKey | ModifierKey,
    modifier_keys: list[ModifierKey] | None = None,
    count: int = 1
) -> None

Simulates pressing and immediately releasing a keyboard key.

Arguments:

  • key PcKey | ModifierKey - The key to tap.
  • modifier_keys list[ModifierKey] | None, optional - List of modifier keys to press along with the main key. Defaults to None.
  • count int, optional - The number of times to tap the key. Defaults to 1.

​
mouse_move

def mouse_move(x: int, y: int) -> None

Moves the mouse cursor to specified screen coordinates.

Arguments:

  • x int - The horizontal coordinate (in pixels) to move to.
  • y int - The vertical coordinate (in pixels) to move to.

​
mouse_down

def mouse_down(button: Literal["left", "middle", "right"] = "left") -> None

Simulates pressing and holding a mouse button.

Arguments:

  • button Literal[“left”, “middle”, “right”], optional - The mouse button to press. Defaults to "left".

​
mouse_scroll

def mouse_scroll(x: int, y: int) -> None

Simulates scrolling the mouse wheel.

Arguments:

  • x int - The horizontal scroll amount. Positive values scroll right, negative values scroll left.
  • y int - The vertical scroll amount. Positive values scroll down, negative values scroll up.

​
mouse_up

def mouse_up(button: Literal["left", "middle", "right"] = "left") -> None

Simulates releasing a mouse button.

Arguments:

  • button Literal[“left”, “middle”, “right”], optional - The mouse button to release. Defaults to "left".

​
poll_event

def poll_event() -> InputEvent | None

Poll for a single input event.

IMPORTANT: This method is still experimental and may not work at all and may change in the future.

​
run_command

def run_command(command: str, timeout_ms: int = 30000) -> None

Executes a shell command.

Arguments:

  • command str - The command to execute.
  • timeout_ms int, optional - The timeout for command execution in milliseconds. Defaults to 30000 (30 seconds).

​
screenshot

def screenshot(report: bool = True) -> Image.Image

Captures a screenshot of the current display.

Arguments:

  • report bool, optional - Whether to include the screenshot in reporting. Defaults to True.

Returns:

  • Image.Image - A PIL Image object containing the screenshot.

​
set_display

def set_display(display: int = 1) -> None

Sets the active display for screen interactions.

Arguments:

  • display int, optional - The display ID to set as active.

​
start_listening

def start_listening() -> None

Start listening for mouse and keyboard events.

IMPORTANT: This method is still experimental and may not work at all and may change in the future.

​
stop_listening

def stop_listening() -> None

Stop listening for mouse and keyboard events.

IMPORTANT: This method is still experimental and may not work at all and may change in the future.

​
type

def type(text: str, typing_speed: int = 50) -> None

Simulates typing text as if entered on a keyboard.

Arguments:

  • text str - The text to be typed.
  • typing_speed int, optional - The speed of typing in characters per minute. Defaults to 50.

​
askui.tools.askui.AskUiControllerClient

class AskUiControllerClient(AgentOs)

Implementation of AgentOs that communicates with the AskUI Remote Device Controller via gRPC.

Arguments:

  • reporter Reporter - Reporter used for reporting with the "AgentOs".
  • display int, optional - Display number to use. Defaults to 1.
  • controller_server AskUiControllerServer | None, optional - Custom controller server. Defaults to ControllerServer.

​
__enter__

def __enter__() -> Self

Context manager entry point that establishes the connection.

Returns:

  • Self - The instance of AskUiControllerClient.

​
__exit__

def __exit__(exc_type, exc_value, traceback) -> None

Context manager exit point that disconnects the client.

Arguments:

  • exc_type - The exception type if an exception was raised.
  • exc_value - The exception value if an exception was raised.
  • traceback - The traceback if an exception was raised.

​
click

def click(
    button: Literal['left', 'middle', 'right'] = 'left',
    count: int = 1
) -> None

Simulates clicking a mouse button.

Arguments:

  • button Literal[“left”, “middle”, “right”], optional - The mouse button to click. Defaults to "left".
  • count int, optional - Number of times to click. Defaults to 1.

​
connect

def connect() -> None

Establishes a connection to the AskUI Remote Device Controller.

This method starts the controller server, establishes a gRPC channel, creates a session, and sets up the initial display.

​
disconnect

def disconnect() -> None

Terminates the connection to the AskUI Remote Device Controller.

This method stops the execution, ends the session, closes the gRPC channel, and stops the controller server.

​
keyboard_pressed

def keyboard_pressed(
    key: PcKey | ModifierKey,
    modifier_keys: list[ModifierKey] | None = None
) -> None

Simulates pressing and holding a keyboard key.

Arguments:

  • key PcKey | ModifierKey - The key to press.
  • modifier_keys list[ModifierKey] | None, optional - List of modifier keys to press along with the main key. Defaults to None.

​
keyboard_release

def keyboard_release(
    key: PcKey | ModifierKey,
    modifier_keys: list[ModifierKey] | None = None
) -> None

Simulates releasing a keyboard key.

Arguments:

  • key PcKey | ModifierKey - The key to release.
  • modifier_keys list[ModifierKey] | None, optional - List of modifier keys to release along with the main key. Defaults to None.

​
keyboard_tap

def keyboard_tap(
    key: PcKey | ModifierKey,
    modifier_keys: list[ModifierKey] | None = None,
    count: int = 1
) -> None

Press and immediately release a keyboard key.

Arguments:

  • key PcKey | ModifierKey - The key to tap.
  • modifier_keys list[ModifierKey] | None, optional - List of modifier keys to press along with the main key. Defaults to None.
  • count int, optional - The number of times to tap the key. Defaults to 1.

​
mouse_move

def mouse_move(x: int, y: int) -> None

Moves the mouse cursor to specified screen coordinates.

Arguments:

  • x int - The horizontal coordinate (in pixels) to move to.
  • y int - The vertical coordinate (in pixels) to move to.

​
mouse_down

def mouse_down(button: Literal['left', 'middle', 'right'] = 'left') -> None

Simulates pressing and holding a mouse button.

Arguments:

  • button Literal[“left”, “middle”, “right”], optional - The mouse button to press. Defaults to "left".

​
mouse_scroll

def mouse_scroll(x: int, y: int) -> None

Simulates scrolling the mouse wheel.

Arguments:

  • x int - The horizontal scroll amount. Positive values scroll right, negative values scroll left.
  • y int - The vertical scroll amount. Positive values scroll down, negative values scroll up.

​
mouse_up

def mouse_up(button: Literal['left', 'middle', 'right'] = 'left') -> None

Simulates releasing a mouse button.

Arguments:

  • button Literal[“left”, “middle”, “right”], optional - The mouse button to release. Defaults to "left".

​
screenshot

def screenshot(report: bool = True) -> Image.Image

Captures a screenshot of the current display.

Arguments:

  • report bool, optional - Whether to include the screenshot in reporting. Defaults to True.

Returns:

  • Image.Image - A PIL Image object containing the screenshot.

​
set_display

def set_display(display: int = 1) -> None

Sets the active display for screen interactions.

Arguments:

  • display int, optional - The display ID to set as active.

​
type

def type(text: str, typing_speed: int = 50) -> None

Simulates typing text as if entered on a keyboard.

Arguments:

  • text str - The text to be typed.
  • typing_speed int, optional - The speed of typing in characters per second. Defaults to 50.

​
askui.tools.askui.exceptions.AskUiControllerError

class AskUiControllerError(Exception)

Base exception for AskUI controller errors.

This exception is raised when there is an error in the AskUI controller (client), which handles the communication with the AskUI controller (server).

Arguments:

  • message str - The error message.

​
askui.tools.askui.exceptions.AskUiControllerOperationFailedError

class AskUiControllerOperationFailedError(AskUiControllerError)

Exception raised when a controller operation fails.

This exception is raised when a specific operation in the AskUI controller fails, such as starting/stopping the controller or executing an action.

Arguments:

  • operation str - The operation that failed.
  • error Exception - The original error that caused the failure.

​
askui.tools.askui.exceptions.AskUiControllerOperationTimeoutError

class AskUiControllerOperationTimeoutError(AskUiControllerError)

Exception raised when a controller action times out.

This exception is raised when an action in the AskUI controller takes longer than the expected timeout period to complete.

Arguments:

  • message str - The error message.
  • timeout_seconds float | None - Optional timeout duration in seconds.

​
askui.tools.askui.AskUiControllerServer

class AskUiControllerServer()

Concrete implementation of ControllerServer for managing the AskUI Remote Device Controller process. Handles process discovery, startup, and shutdown for the native controller binary.

​
start

def start(clean_up: bool = False) -> None

Start the controller process.

Arguments:

  • clean_up bool, optional - Whether to clean up existing processes (only on Windows) before starting. Defaults to False.

​
stop

def stop(force: bool = False) -> None

Stop the controller process.

Arguments:

  • force bool, optional - Whether to forcefully terminate the process. Defaults to False.

​
askui.tools.exceptions.ToolError

class ToolError(Exception)

Raised when a tool encounters an error.

Arguments:

  • message str - The error message.
  • result ToolResult - The ToolResult that caused the error.

​
askui.tools.exceptions.ToolResult

class ToolResult()

Represents the result of a tool execution.

Arguments:

  • output str | None, optional - The output of the tool.
  • error str | None, optional - The error message of the tool.
  • base64_image str | None, optional - The base64 image of the tool.
  • system str | None, optional - The system message of the tool.