Skip to main content

AI API

Provides an API to interact with LLMs

Every function raises when the session's Sidecar.Session.AIPolicy does not permit AI methods, or does not allow the model the call resolves to.

ai.agent​

Signatures​

agent(goal, opts \ [], state)

Description​

AI agent that iteratively navigates and interacts with devices using vision and control.

Parameters​

  • goal (string) - Natural language description of what to accomplish
  • opts (table) - Optional configuration
    • model - AI model to use (default: from Sidecar.LLM)
    • systemPrompt - Optional custom system prompt
    • maxAttempts - Maximum iterations (default: 20)
    • screenshotDelayMs - Optional. Milliseconds to wait before each screenshot between iteration loops (default: 0 when omitted)
    • screenshotRetention - How to handle older screenshots once past the retention limit: "Omit" (default), "Keep All", or "Summarize"
    • tools - List of tool names (default: ["remote_control", "screenshot"])

Returns​

Returns two values: success (boolean) and result (table)

  • success: true if goal completed, false otherwise
  • result.actions: String of actions taken (e.g., "up,right,tap,complete")
  • result.attempts: Number of iterations
  • result.finalScreen: Screenshot after completion (userdata)
  • result.reasoning: AI's explanation of what it did

Example​

local success, result = ai.agent("Navigate to Settings", {maxAttempts = 10})
if success then
  print("Completed in " .. result.attempts .. " attempts")
  print("Actions: " .. result.actions)
end

ai.ask​

Signatures​

ask(question, opts \ [], lua)

ask(image_ref, question, opts, lua)

Description​

Ask a yes/no question about the current screen or a specified region.

This function captures the current screen and optionally crops it to a specific region before asking the AI a yes/no question about it.

Parameters​

  • question (string) - A yes/no question about the screen
  • opts (table) - Optional parameters

Options​

  • region - Optional region name to focus the AI analysis on a specific area of the screen
  • systemPrompt - Optional system prompt to guide the AI analysis
  • temperature - Optional temperature for the AI model
  • model - Optional AI model

Example​

local is_playing = ai.ask("Is the video currently playing?")

if is_playing then
  print("Video is playing")
else
  print("Video is not playing")
end

-- With region
local has_error = ai.ask("Is there an error message?", {region = "error_dialog"})

ai.ask​

Signatures​

ask(question, opts \ [], lua)

ask(image_ref, question, opts, lua)

Description​

Ask a yes/no question about a provided image and returns a boolean answer.

Uses OpenAI's structured output feature to guarantee a reliable true/false response.

Parameters​

  • image (userdata) - Image object to analyze
  • question (string) - A yes/no question about the image
  • opts (table) - Optional parameters

Options​

  • systemPrompt - Optional system prompt to guide the AI analysis
  • temperature - Optional temperature for the AI model
  • model - Optional AI model

Example​

local screenshot = screen.capture()
local is_visible = ai.ask(screenshot, "Is the Netflix app visible on the screen?")

if is_visible then
  print("Netflix is visible!")
end

ai.codegen​

Signatures​

codegen(prompt)

Description​

Generate lua code from the prompt.

Parameters​

  • prompt (string) - Natural language description of actions to convert to Lua code

Example​

local codegen = ai.codegen("Navigate home, wait 5 seconds, then press up and enter")

print(codegen)
-- control.home() wait(5000) control.up() control.ok()

ai.navigate​

Signatures​

navigate(target, opts \ [], lua)

navigate(image_ref, target, opts, lua)

Description​

Navigate menus using AI with the current screen or a specified region.

This function captures the current screen and optionally crops it to a specific region before attempting AI-guided navigation to the target.

Parameters​

  • target (string) - Navigation target description
  • opts (table) - Optional parameters

Options​

  • region - Optional region name to focus the navigation on a specific area of the screen

Example​

local sequence, prompt, result, viewpoint = ai.navigate("Settings menu")

-- With region
local sequence, prompt, result, viewpoint = ai.navigate("Volume control", {region = "audio_panel"})

ai.navigate​

Signatures​

navigate(target, opts \ [], lua)

navigate(image_ref, target, opts, lua)

Description​

Navigate menus using AI with a provided image.

Parameters​

  • image (userdata) - Image object to analyze for navigation
  • target (string) - Navigation target description
  • opts (table) - Optional parameters

Options​

  • model - AI model to use for navigation
  • temperature - AI temperature setting
  • imageScale - Scale factor for image processing
  • promptType - Type of navigation prompt
  • additionalInstructions - Extra instructions for the AI

ai.prompt​

Signatures​

prompt(user_prompt, opts \ [], lua)

prompt(image_ref, user_prompt, opts, lua)

Description​

Runs a prompt with the current screen or a specified region and returns the response and viewpoint.

This function captures the current screen and optionally crops it to a specific region before sending the prompt to the AI.

Parameters​

  • user_prompt (string) - The prompt to send to the AI
  • opts (table) - Optional parameters

Options​

  • region - Optional region name to focus the AI analysis on a specific area of the screen
  • systemPrompt - Optional system prompt to guide the AI response
  • temperature - Optional temperature for the AI model
  • model - Optional AI model

Example​

local response, viewpoint = ai.prompt("What is the text on the screen?")

print(response)
-- "The screen shows a login form with username and password fields."

-- With region
local response, viewpoint = ai.prompt("What color is this button?", {region = "submit_button"})

ai.prompt​

Signatures​

prompt(user_prompt, opts \ [], lua)

prompt(image_ref, user_prompt, opts, lua)

Description​

Runs a prompt with a provided image, or the current screen, and returns the response and the viewpoint of the AI.

Parameters​

  • image (userdata) - Image object to analyze
  • prompt (string) - The prompt to send to the AI
  • opts (table) - Optional parameters

Options​

  • systemPrompt - Optional system prompt to guide the AI response
  • temperature - Optional temperature for the AI model
  • model - Optional AI model

Example​

local response, viewpoint = ai.prompt("What is the color of the screen?")

print(response)
-- "The general color of the screen is blue."

ai.run​

Signatures​

run(prompt, state)

Description​

The same as codegen, but runs the generated code immediately.

Parameters​

  • prompt (string) - Natural language description of actions to execute

Example​

ai.run("Navigate home, wait 5 seconds, then press up and enter")