Execute Methods

note

All examples in this documentation are using the webdriverio client.

Overview

TV Labs provides a superset of Appium functionality through execute methods that expose additional functionality of the platform that aren't traditionally available through Appium. This page outlines the available execute methods.

Device

tvlabs: currentApp

Queries the device for the app currently running on it. Returns an object with app_id, app_version, and app_name fields, otherwise returns an error.

app_version and app_name may be null if the underlying device protocol can't cheaply report them.

A device sitting on its home screen or showing a screensaver is not running an app, and returns an error rather than reporting one. On Roku in particular the home screen reports itself through the same query as a running channel, and carries an app id of its own — it is still treated as no app running.

note

This method is currently available on Roku, webOS, Android TV, and Android mobile devices. Support for the remaining platforms is coming soon.

Example:

const currentApp = await driver.execute('tvlabs: currentApp');

console.log(currentApp);
// { app_id: 'dev', app_name: 'TV Labs', app_version: '1.0.0' }

AI

These methods give a WebDriver session the AI primitives of NoCode, and are available on every platform.

caution

AI models can return incorrect results.

tvlabs: aiPrompt

Sends a prompt about the current screen to a vision model and returns its text response.

Arguments:

• prompt (required, string)
  • What to ask the model about the screen.
• region (optional, object)
  • Crops the screen to { x, y, width, height }, in screen pixels, before it is sent to the model. Defaults to the whole screen.
• model (optional, string)
  • The model to use, as provider:model for an OpenAI or Anthropic model that accepts images, such as openai:gpt-4o. Defaults to anthropic:claude-sonnet-4-6.
• includeView (optional, boolean)
  • Adds the image that was sent to the model to the response as view. Defaults to false.

Returns:

• prompt (string)
  • The prompt that was sent to the model.
• answer (string)
  • The model's response.
• view (string)
  • The base64 JPEG that was sent to the model, cropped to region when one is given. Only present when includeView is true.

Example:

const { answer, view } = await driver.execute('tvlabs: aiPrompt', {
  prompt: 'What does this button say?',
  region: { x: 100, y: 800, width: 400, height: 120 },
  includeView: true,
});

console.log(answer);
// 'Sign in'

tvlabs: aiAsk

Asks a vision model a yes/no question about the current screen and returns its answer as a boolean.

Arguments:

• question (required, string)
  • A yes/no question about the screen.
• region (optional, object)
  • Crops the screen to { x, y, width, height }, in screen pixels, before it is sent to the model. Defaults to the whole screen.
• model (optional, string)
  • The model to use, as provider:model for an OpenAI or Anthropic model that accepts images, such as openai:gpt-4o. Defaults to anthropic:claude-sonnet-4-6.
• includeView (optional, boolean)
  • Adds the image that was sent to the model to the response as view. Defaults to false.

Returns:

• prompt (string)
  • The question that was sent to the model.
• answer (boolean)
  • true if the model answered yes, false otherwise.
• view (string)
  • The base64 JPEG that was sent to the model, cropped to region when one is given. Only present when includeView is true.

Example:

const { answer } = await driver.execute('tvlabs: aiAsk', {
  question: 'Is the Netflix app visible on the screen?',
});

console.log(answer);
// true

tvlabs: aiAgent

Runs an AI agent that looks at the screen and presses remote keys until it reaches a goal, the same as the AI Agent block in NoCode. Returns an error if the agent does not reach the goal, with the agent's reasoning in the message.

note

An agent run can take several minutes. It must finish within the session's appium:newCommandTimeout capability, which is in seconds, defaults to 60, and can be raised to 300. A run that is still going at that point is stopped, and the command returns a timeout error. Raise the capability or lower maxAttempts to fit the run.

Arguments:

• goal (required, string)
  • What the agent should accomplish, such as Open Settings or Select the Netflix app.
• model (optional, string)
  • The model to use, as provider:model for an OpenAI or Anthropic model that accepts images, such as openai:gpt-4o. Defaults to anthropic:claude-sonnet-4-6.
• screenshot (optional, boolean)
  • Allows the agent to capture extra screenshots. Defaults to true.
• remoteControlActions (optional, array of strings)
  • The remote keys the agent can press. Defaults to ["up", "down", "left", "right", "ok", "back", "home"]. An empty array takes remote control away from the agent.
• sleep (optional, boolean)
  • Allows the agent to wait, for example after launching an app. Defaults to true.
• maxAttempts (optional, number)
  • The maximum number of steps before the agent gives up. Defaults to 20.
• screenshotDelayMs (optional, number)
  • Milliseconds to wait before each screen capture. Defaults to 500.
• screenshotRetention (optional, string)
  • How the agent handles older screenshots so a long run fits the model's context: Omit (default), Keep All, or Summarize.

Returns:

• success (boolean)
  • true when the agent reached the goal.
• actions (string)
  • The actions the agent took, comma-separated, such as down,down,ok.
• attempts (number)
  • The number of steps the agent took.
• finalScreen (string)
  • A base64 JPEG of the screen when the agent finished.
• reasoning (string)
  • The agent's explanation of what it did.

Example:

const { actions, reasoning } = await driver.execute('tvlabs: aiAgent', {
  goal: 'Open Settings',
  maxAttempts: 10,
});

console.log(actions);
// 'down,down,right,ok'

Roku

Roku sessions are driven by the Roku WebDriver, which exposes the device's External Control Protocol (ECP) through the roku: execute methods below. They are available on any Roku session.

A sideloaded build always has the app id dev.

note

roku: installApp and roku: removeApp also exist, but sideloading is managed by the platform. Select a build with the tvlabs:build capability instead.

roku: activateApp

Launches a channel.

Arguments:

• appId (required, string)
  • The channel to launch
• contentId (optional, string)
  • Deep-link content id
• mediaType (optional, string)
  • Deep-link media type

Example:

await driver.execute('roku: activateApp', { appId: 'dev' });

roku: deepLink

Deep-links into content in the channel that is already running. Use roku: activateApp to deep-link into a channel that isn't.

Arguments:

• contentId (required, string)
  • The content to open
• mediaType (optional, string)
  • One of Roku's valid media types
  • Default: "movie"

Example:

await driver.execute('roku: deepLink', { contentId: '12345', mediaType: 'series' });

roku: pressKey

Presses a key on the remote. See the key map for the accepted names. Any other value is sent to ECP as a raw key value.

Arguments:

• key (required, string)
  • The key to press

Example:

await driver.execute('roku: pressKey', { key: 'Home' });

roku: longPress

Holds a key down and then releases it.

Arguments:

• key (required, string)
  • The key to hold, from the same key map as roku: pressKey
• durationMs (optional, number)
  • How long to hold it, in milliseconds
  • Default: 1000

Example:

await driver.execute('roku: longPress', { key: 'Select', durationMs: 2000 });

roku: deviceInfo

Returns the device's /query/device-info fields as an object.

Arguments: None

Example:

const info = await driver.execute('roku: deviceInfo');

console.log(info['software-version']);
// 14.10.5

roku: getApps

Returns every channel installed on the device, as a list of objects with id, name, type, subtype, and version.

Arguments: None

Example:

const apps = await driver.execute('roku: getApps');

console.log(apps);
// [{ id: '12', name: 'Netflix', type: 'appl', subtype: 'ndka', version: '5.1.10' }, ...]

roku: activeApp

Returns the channel currently in the foreground, in the same shape as roku: getApps.

Unlike tvlabs: currentApp, this reports the raw ECP answer: the home screen comes back as an entry of its own rather than as an error.

Arguments: None

Example:

const app = await driver.execute('roku: activeApp');

console.log(app);
// { id: 'dev', name: 'TV Labs', type: 'appl', subtype: 'rsga', version: '1.0.0' }

roku: appUI

Returns the XML app hierarchy from /query/app-ui, as the <topscreen> element. Only available while the sideloaded channel is running.

Example:

const topscreen = await driver.execute('roku: appUI');

roku: playerState

Returns the state of the media player, from /query/media-player.

Arguments: None

Example:

const { player } = await driver.execute('roku: playerState');

console.log(player.state);
// play

roku: selectElement

Moves focus onto an element, pressing whichever remote keys get it there. Errors if focus can't be reached.

Arguments:

• elementId (required, string)
  • The element to focus — the id returned by a previous findElement, or an xpath locator

Example:

await driver.execute('roku: selectElement', { elementId: '//Label[@text="Play"]' });

roku: ecp

Calls an arbitrary ECP endpoint and returns the response body as text. The escape hatch for anything the methods above don't cover.

Arguments:

• url (required, string)
  • The ECP path, starting with /
• method (optional, string)
  • HTTP method
  • Default: "POST"
• body (optional, string)
  • Request body
  • Default: ""

Example:

const xml = await driver.execute('roku: ecp', {
  url: '/query/device-info',
  method: 'GET',
});

roku: ecpInput

Posts to the /input ECP endpoint. The argument object is the parameter set itself — any key/value pairs, URL-encoded for you.

Example:

await driver.execute('roku: ecpInput', {
  'acceleration.x': '0.0',
  'acceleration.y': '0.0',
  'acceleration.z': '9.8',
});

Device Performance Metrics

CPU and memory are sampled in the background for the lifetime of the session. These methods return the most recent sample, so they answer immediately without querying the device.

note

These methods are currently available on Roku, webOS, Tizen, Android TV (including Fire TV), Apple TV, iPhone, and iPad. Support for the remaining platforms is coming soon.

Apple TV, iPhone, and iPad additionally require the device to be running TV Labs' own driver stack. On an Apple device that isn't, no sampler starts and these methods report available: false with reason: "sampler_not_running" for the whole session. Reach out if you need metrics on an Apple device and see that reason persist.

None of these methods throw when a reading can't be produced. Every response carries an available flag:

• available (boolean)
  • Whether a reading was available. When false, reason and message are the only other fields present
• reason (string)
  • Present only when available is false. A stable snake_case identifier, safe to branch on. See the table below
• message (string)
  • Present only when available is false. Human-readable detail, for logs and failure output. Never branch on it — it can embed a platform name or text supplied by the device itself

reason	Meaning
never_sampled	A sampler is running but hasn't produced its first reading yet. Retry — webOS and Android TV sample about every 5 seconds
sampler_not_running	No sampler is running for this session. Persists for the session's lifetime
unsupported_platform	The device's platform has no sampler
unavailable	The device declined to report. On Roku this means no developer channel is running
no_device	No device is bound to the session

Treat any other value as an unexpected failure and log message — new reasons may be added, and unknown is used for anything unforeseen.

const metrics = await driver.execute('tvlabs: getDevicePerformanceMetrics');

if (!metrics.available) {
  if (metrics.reason === 'never_sampled') {
    // Too early — try again shortly
  }

  console.log(`no reading: ${metrics.message}`);
  // no reading: unsupported platform: smartcast
}

Two fields are common to all three methods when available is true:

• pts (number | null)
  • Presentation timestamp of the video frame current at the time of the reading, for lining a measurement up against the session recording. null when no frame is available
• raw (object)
  • The platform's own metric fields, under their native names. The keys vary by platform

The normalized numbers describe whatever the platform measures. Roku reports the running channel; webOS, Tizen, Android TV, and Apple devices report the system as a whole. On webOS and Apple devices the foreground app's own CPU and memory are additionally reported under raw as the app_* keys, and on Fire TV thermal readings appear as the temp_* keys.

warning

available: true means the device reported a reading, not that every number in it is populated. Any normalized field can be null when the platform reported a sample without that particular value, so check before doing arithmetic on it:

if (metrics.cpu_percent !== null) {
  console.log(metrics.cpu_percent.toFixed(1));
}

tvlabs: getDevicePerformanceMetrics

Returns the full reading — CPU and memory together, with every native field under raw.

Returns:

• cpu_percent (number | null)
  • CPU utilization from 0-100
• memory_used_bytes (number | null)
  • Memory in use, in bytes
• memory_total_bytes (number | null)
  • Total memory available, in bytes. Always null on Roku, which measures the running channel rather than the system
• pts, raw, available — as described above

Example:

// On a webOS device
const metrics = await driver.execute('tvlabs: getDevicePerformanceMetrics');

console.log(metrics);
// {
//   available: true,
//   cpu_percent: 23.4,
//   memory_used_bytes: 1342177280,
//   memory_total_bytes: 2147483648,
//   pts: 18432,
//   raw: {
//     cpu_utilization_pct: 23.4,
//     mem_total_bytes: 2147483648,
//     mem_free_bytes: 536870912,
//     mem_available_bytes: 805306368,
//     mem_cached_bytes: 268435456,
//     mem_used_bytes: 1342177280,
//     load_average_1m: 1.12,
//     uptime_seconds: 84021,
//     app_id: 'com.example.app',
//     app_cpu_pct: 8.1,
//     app_mem_resident_bytes: 209715200,
//     app_mem_virtual_bytes: 1073741824,
//   },
// }

To sample repeatedly over the course of a test, poll the method and record each reading against its pts:

const samples = [];

for (let i = 0; i < 10; i++) {
  const sample = await driver.execute('tvlabs: getDevicePerformanceMetrics');

  if (sample.available && sample.cpu_percent !== null) {
    samples.push(sample);
  }

  await driver.pause(1000);
}

const peakCpu = Math.max(...samples.map((s) => s.cpu_percent));

tvlabs: getDeviceCpuMetrics

Returns just the CPU view of the latest reading. raw is narrowed to the platform's CPU-specific fields.

Returns:

• cpu_percent (number | null)
  • CPU utilization from 0-100
• pts, raw, available — as described above

Example:

// On a Roku device
const cpu = await driver.execute('tvlabs: getDeviceCpuMetrics');

console.log(cpu);
// {
//   available: true,
//   cpu_percent: 12.5,
//   pts: 18432,
//   raw: {
//     cpu_user_pct: 8,
//     cpu_sys_pct: 4.5,
//     cpu_total_pct: 12.5,
//     duration_seconds: 1,
//   },
// }

tvlabs: getDeviceMemoryMetrics

Returns just the memory view of the latest reading. raw is narrowed to the platform's memory-specific fields.

Returns:

• memory_used_bytes (number | null)
  • Memory in use, in bytes
• memory_total_bytes (number | null)
  • Total memory available, in bytes. Always null on Roku, which measures the running channel rather than the system
• pts, raw, available — as described above

Example:

// On a Roku device
const memory = await driver.execute('tvlabs: getDeviceMemoryMetrics');

console.log(memory);
// {
//   available: true,
//   memory_used_bytes: 209715200,
//   memory_total_bytes: null,
//   pts: 18432,
//   raw: {
//     mem_used_bytes: 209715200,
//     mem_anon_bytes: 167772160,
//     mem_file_bytes: 33554432,
//     mem_shared_bytes: 8388608,
//   },
// }

Network Control

tvlabs: disableNetwork

Disables the network connection for the device, simulating a network outage. Returns true if the network was successfully disabled, otherwise returns an error.

Example:

await driver.execute('tvlabs: disableNetwork');

note

If a network throttle is active, it will be disabled when tvlabs: disableNetwork is executed. After tvlabs: enableNetwork is executed, the network throttle command will need to be explicitly re-enabled if desired.

await driver.execute('tvlabs: enableNetworkThrottle', { limitDown: 5 });

// Network is throttled

await driver.execute('tvlabs: disableNetwork');

// Network is offline

await driver.execute('tvlabs: enableNetwork');

// Network is online, no throttling enabled

await driver.execute('tvlabs: enableNetworkThrottle', { limitDown: 5 });

// Throttling re-enabled

tvlabs: enableNetwork

Restores the network connection after tvlabs: disableNetwork was executed. Returns true if the network was successfully enabled, otherwise returns an error.

Example:

await driver.execute('tvlabs: enableNetwork');

tvlabs: isNetworkEnabled

Returns true if the network is enabled.

Example:

await driver.execute('tvlabs: disableNetwork');

const isNetworkEnabled = await driver.execute('tvlabs: isNetworkEnabled');

console.log(isNetworkEnabled);
// false

Network Throttling

tvlabs: enableNetworkThrottle

Enables network rate limiting for the device. Returns true if the rate limit was successfully enabled, otherwise returns an error.

Arguments:

• limitDown (required, number)
  • The download limit in mbits / second
• limitUp (optional, number | string)
  • The upload limit in mbits / second. Accepts "infinity" as a value.
  • Default: "infinity"

Example:

// Throttle download to 5 mbits / second
await driver.execute('tvlabs: enableNetworkThrottle', {
  limitDown: 5,
});

tvlabs: disableNetworkThrottle

Disables network rate limiting for the device. Returns true if the rate limit was successfully disabled, otherwise returns an error.

Example:

await driver.execute('tvlabs: disableNetworkThrottle');

tvlabs: isNetworkThrottleEnabled

Returns true if the rate limit is enabled.

Example:

await driver.execute('tvlabs: enableNetworkThrottle', {
  limitDown: 5,
});

const isNetworkThrottleEnabled = await driver.execute('tvlabs: isNetworkThrottleEnabled');

console.log(isNetworkThrottleEnabled);
// true

Network Usage

tvlabs: getNetworkUsage

Returns a snapshot of throughput and interface statistics for the device's network connection, measured over the most recent sampling interval (reported as interval_ms). Packet, error, drop, multicast, and collision values are the change observed during that interval, not cumulative totals.

Returns:

• rx_kbps (number)
  • Download throughput in kbits / second
• tx_kbps (number)
  • Upload throughput in kbits / second
• rx_packets (number)
  • Packets received during the interval
• tx_packets (number)
  • Packets transmitted during the interval
• rx_errors (number)
  • Receive errors during the interval
• tx_errors (number)
  • Transmit errors during the interval
• rx_dropped (number)
  • Inbound packets dropped during the interval
• tx_dropped (number)
  • Outbound packets dropped during the interval
• multicast (number)
  • Multicast frames received during the interval
• collisions (number)
  • Packet collisions detected during the interval
• interval_ms (number)
  • Length of the sampling window in milliseconds

Example:

const usage = await driver.execute('tvlabs: getNetworkUsage');

console.log(usage);
// {
//   rx_kbps: 1.03,
//   tx_kbps: 0.67,
//   rx_packets: 2,
//   tx_packets: 1,
//   rx_errors: 0,
//   tx_errors: 0,
//   rx_dropped: 0,
//   tx_dropped: 0,
//   multicast: 0,
//   collisions: 0,
//   interval_ms: 1000,
// }

Network Capture

More details on network capture functionality can be found on the main Network Capture page.

tvlabs: startNetworkCapture

Starts network capture for the current session.

Arguments:

• name (optional, string)
  • A name for the capture, used to identify it in the sessions dashboard
• mode (optional, string)
  • Which capture method to use: "devtools" or "proxy" ("cdp" is accepted as another name for "devtools"). See Choosing the Capture Mode
  • Default: the platform's default, from the Supported Platforms table
  • Any other value falls back to the platform's default
• direct_proxy (optional, boolean)
  • Enables direct proxy mode, where your application sends traffic to the proxy directly rather than having it intercepted transparently
  • Default: false
  • proxy mode only
  • Ignored on VegaOS, which always uses direct proxy mode
• filters (optional, array)
  • A list of filter objects to reduce HAR file size by dropping or omitting matched entries, or to exclude a host from interception
• reload (optional, boolean)
  • Whether to reload the page after starting network capture
  • devtools mode only
  • Default: true
• cache (optional, boolean)
  • Whether to enable or disable the page cache
  • devtools mode only
  • Default: false

Example:

// Start a network capture with defaults
await driver.execute('tvlabs: startNetworkCapture');

// Start a named capture with direct proxy mode and a filter
await driver.execute('tvlabs: startNetworkCapture', {
  name: 'my-capture',
  direct_proxy: true,
  filters: [{ action: 'drop', mime_type: 'image/.+' }],
});

// Capture from the application's debugger instead of the platform's default
await driver.execute('tvlabs: startNetworkCapture', { mode: 'devtools' });

A capture must be stopped before starting another one in a different mode.

warning

On non-Chromium based platforms, after starting a network capture in an application that uses HTTP keep-alive, the application may need to be re-launched to ensure any open network connections are re-established and intercepted. This can be done with the terminateApp and activateApp methods.

On Tizen, webOS, and SmartCast, re-launching the app this way will also terminate the active ChromeDriver session — see ChromeDriver session lifecycle before doing this mid-test.

tvlabs: stopNetworkCapture

Stops network capture and finalizes the HAR file for download. If the session network capture was started using capabilities, this method may be used to stop the capture before the session ends.

Arguments: None

Example:

// Stop an already running capture
await driver.execute('tvlabs: stopNetworkCapture');