Skip to main content

Execute Methods

note

All examples in this documentation are using the webdriverio client.

Overview​

The TV Labs WebDriver server is Appium compatible, and extends it with execute methods that expose functionality of the platform that isn't available through standard WebDriver or Appium commands. This page outlines the available execute methods.

The methods on this page work across platforms. Methods that only apply to one platform are documented on that platform's page:

Device​

tvlabs: currentApp​

Queries the device for the app currently running on it. Returns an object with app_id, app_version, and app_name fields, otherwise returns an error.

app_version and app_name may be null if the underlying device protocol can't cheaply report them.

A device sitting on its home screen or showing a screensaver is not running an app, and returns an error rather than reporting one. On Roku in particular the home screen reports itself through the same query as a running channel, and carries an app id of its own — it is still treated as no app running.

note

This method is currently available on Roku, webOS, Android TV, and Android mobile devices. Support for the remaining platforms is coming soon.

Example:

const currentApp = await driver.execute('tvlabs: currentApp');

console.log(currentApp);
// { app_id: 'dev', app_name: 'TV Labs', app_version: '1.0.0' }

mobile: deviceInfo​

Returns information about the device. Every platform returns these fields:

  • name — the name the device advertises on the network. AirPlay and Google Cast pickers show this name, so a test that casts to the device can use it to choose the correct target.
  • model — the device model.
  • manufacturer — the device manufacturer.
  • platformVersion — the OS or firmware version.

A field the device does not report is null. Some platforms return more fields, listed below.

Arguments: None

PlatformWhere name comes fromAdditional fields
iOS, iPadOS, tvOSThe device name set on the device (Settings → General → About → Name). If it cannot be read, the generic name WebDriverAgent reports, such as iPhone.The fields WebDriverAgent reports
Android TV, Android mobileThe device name set on the device. If none is set, its Bluetooth name. If neither can be read, name is omitted.The fields the UiAutomator2 server reports
RokuThe name the user gave the device. If none, its friendly name, then its default name (model and serial number).The ECP device-info fields, in snake_case (software_version)
Samsung TizenThe TV's device name, as the TV advertises it on the network—
LG webOSThe TV's device name, as the TV advertises it on the networkfirmwareVersion
Vizio SmartCastThe name the TV advertises to Google Cast—
Amazon Vega OS (Fire TV)The friendly name the device advertises over DIAL. null if the device did not answer when it was discovered.The VegaOS identity and build fields
Entertainment OSThe room set in the device's settings. If no room is set, the device's serial number.—
Scaled (Sky Q)The device's serial number—

Example:

const info = await driver.execute('mobile: deviceInfo');

console.log(info.name);
// Living Room TV

AI​

These methods give a WebDriver session the AI primitives of NoCode, and are available on every platform.

caution

AI models can return incorrect results.

tvlabs: aiPrompt​

Sends a prompt about the current screen to a vision model and returns its text response.

Arguments:

  • prompt (required, string)
    • What to ask the model about the screen.
  • region (optional, object)
    • Crops the screen to { x, y, width, height }, in screen pixels, before it is sent to the model. Defaults to the whole screen.
  • model (optional, string)
    • The model to use, as provider:model for an OpenAI or Anthropic model that accepts images, such as openai:gpt-4o. Defaults to anthropic:claude-sonnet-4-6.
  • includeView (optional, boolean)
    • Adds the image that was sent to the model to the response as view. Defaults to false.

Returns:

  • prompt (string)
    • The prompt that was sent to the model.
  • answer (string)
    • The model's response.
  • view (string)
    • The base64 JPEG that was sent to the model, cropped to region when one is given. Only present when includeView is true.

Example:

const { answer, view } = await driver.execute('tvlabs: aiPrompt', {
  prompt: 'What does this button say?',
  region: { x: 100, y: 800, width: 400, height: 120 },
  includeView: true,
});

console.log(answer);
// 'Sign in'

tvlabs: aiAsk​

Asks a vision model a yes/no question about the current screen and returns its answer as a boolean.

Arguments:

  • question (required, string)
    • A yes/no question about the screen.
  • region (optional, object)
    • Crops the screen to { x, y, width, height }, in screen pixels, before it is sent to the model. Defaults to the whole screen.
  • model (optional, string)
    • The model to use, as provider:model for an OpenAI or Anthropic model that accepts images, such as openai:gpt-4o. Defaults to anthropic:claude-sonnet-4-6.
  • includeView (optional, boolean)
    • Adds the image that was sent to the model to the response as view. Defaults to false.

Returns:

  • prompt (string)
    • The question that was sent to the model.
  • answer (boolean)
    • true if the model answered yes, false otherwise.
  • view (string)
    • The base64 JPEG that was sent to the model, cropped to region when one is given. Only present when includeView is true.

Example:

const { answer } = await driver.execute('tvlabs: aiAsk', {
  question: 'Is the Netflix app visible on the screen?',
});

console.log(answer);
// true

tvlabs: aiAgent​

Runs an AI agent that looks at the screen and presses remote keys until it reaches a goal, the same as the AI Agent block in NoCode. Returns an error if the agent does not reach the goal, with the agent's reasoning in the message.

note

An agent run can take several minutes. It must finish within the session's appium:newCommandTimeout capability, which is in seconds, defaults to 60, and can be raised to 300. A run that is still going at that point is stopped, and the command returns a timeout error. Raise the capability or lower maxAttempts to fit the run.

Arguments:

  • goal (required, string)
    • What the agent should accomplish, such as Open Settings or Select the Netflix app.
  • model (optional, string)
    • The model to use, as provider:model for an OpenAI or Anthropic model that accepts images, such as openai:gpt-4o. Defaults to anthropic:claude-sonnet-4-6.
  • screenshot (optional, boolean)
    • Allows the agent to capture extra screenshots. Defaults to true.
  • remoteControlActions (optional, array of strings)
    • The remote keys the agent can press. Defaults to ["up", "down", "left", "right", "ok", "back", "home"]. An empty array takes remote control away from the agent.
  • sleep (optional, boolean)
    • Allows the agent to wait, for example after launching an app. Defaults to true.
  • maxAttempts (optional, number)
    • The maximum number of steps before the agent gives up. Defaults to 20.
  • screenshotDelayMs (optional, number)
    • Milliseconds to wait before each screen capture. Defaults to 500.
  • screenshotRetention (optional, string)
    • How the agent handles older screenshots so a long run fits the model's context: Omit (default), Keep All, or Summarize.

Returns:

  • success (boolean)
    • true when the agent reached the goal.
  • actions (string)
    • The actions the agent took, comma-separated, such as down,down,ok.
  • attempts (number)
    • The number of steps the agent took.
  • finalScreen (string)
    • A base64 JPEG of the screen when the agent finished.
  • reasoning (string)
    • The agent's explanation of what it did.

Example:

const { actions, reasoning } = await driver.execute('tvlabs: aiAgent', {
  goal: 'Open Settings',
  maxAttempts: 10,
});

console.log(actions);
// 'down,down,right,ok'

Workflows​

These methods run a workflow on the device from inside a WebDriver session, so a test can hand part of its work to a workflow built in Workflows and get its result back.

To find a workflow's workflowId, open the project in Workflows, open the dropdown next to Trace, and select CLI & WebDriver. The WebDriver tab shows a ready-to-use example for the selected workflow. You can also copy a workflow's ID from the project's file tree.

A run uses the workflow as it is saved when the run starts, draft or published. Changes saved later apply to the next run.

Background and foreground runs​

A run happens in the background by default: tvlabs: executeWorkflow answers at once with a runId, and the test polls tvlabs: getWorkflowRun until the run finishes. Long workflows should run in the background.

const { runId } = await driver.execute('tvlabs: executeWorkflow', {
  workflowId: '<your workflow id>',
});

const run = await driver.waitUntil(
  async () => {
    const current = await driver.execute('tvlabs: getWorkflowRun', { runId });
    return current.status !== 'running' && current;
  },
  { interval: 10000, timeout: 30 * 60 * 1000 },
);

console.log(run.status);
// 'completed'

Short workflows can run in the foreground, in a single request: pass waitForComplete: true, and the call answers when the run finishes. The time limit on this type of request is 10 minutes. Raise your WebDriver client's request timeout to match, because many clients give up on a request after about 2 minutes and then retry it.

const run = await driver.execute('tvlabs: executeWorkflow', {
  workflowId: '<your workflow id>',
  waitForComplete: true,
});

Run results​

tvlabs: executeWorkflow, tvlabs: getWorkflowRun and tvlabs: stopWorkflowRun return a run result:

  • runId (string)
    • The ID of the run, for tvlabs: getWorkflowRun and tvlabs: stopWorkflowRun.
  • workflowId (string)
    • The workflow that was run.
  • status (string)
    • One of the statuses below.
  • durationMs (number)
    • How long the run has taken so far, or took, in milliseconds.
  • failure (object)
    • Present only when the run failed or timed out.
    • message (string): why the run failed.
    • stepId (string or null): the ID of the step that failed, when the failure came from a step. Each step shows its ID in Workflows, so this finds the step even when several share a name.
    • stepName (string or null): the name of that step.
StatusMeaning
runningThe run is still in progress.
completedThe workflow finished, including one that ended itself with an End Workflow step.
failedA step failed, or the workflow raised an error. failure says why.
interruptedThe run was stopped with tvlabs: stopWorkflowRun, or timed out. A run that timed out also includes failure.

A workflow result that is a failure will still respond 200 at the HTTP level, but will include a failure key with the failure reason. Only a run that cannot start answers with a WebDriver error.

tvlabs: executeWorkflow​

Starts a run of a workflow.

Arguments:

  • workflowId (required, string)
    • The ID of the workflow to run. It must belong to your organization, and be a workflow rather than a library.
  • keyValues (optional, object)
    • Overrides the project's key-values for this run, by name, such as { region: 'eu', retries: 3 }. Only key-values marked overwritable can be overridden, and each value must match the key-value's type: a string, a whole number for an integer, a number, a boolean, or one of an enum's options. Secret key-values can be overridden too; their values are never recorded in the session's request log.
  • waitForComplete (optional, boolean)
    • true runs the workflow in the foreground and answers when it finishes. Defaults to false, which answers at once.
  • timeout (optional, number)
    • How long the run may take before it is stopped, in milliseconds. Defaults to 5 minutes. Can be up to 30 minutes, or 10 minutes when waitForComplete is true.

Returns: A run result. Its status is running unless waitForComplete is true.

Errors:

  • invalid argument (400) when an argument is missing or not valid, such as an unknown key-value, a key-value that is not overwritable, a value of the wrong type, or a timeout over the limit.
  • not found (404) when no workflow with that ID exists in your organization.
  • bad request (400) when another workflow run is still in progress in the session. A session runs one workflow at a time.

Example:

const { runId } = await driver.execute('tvlabs: executeWorkflow', {
  workflowId: '<your workflow id>',
  keyValues: { region: 'eu' },
  timeout: 10 * 60 * 1000,
});

tvlabs: getWorkflowRun​

Returns the current result for a run.

Arguments:

  • runId (required, string)
    • The runId that tvlabs: executeWorkflow returned.

Returns: A run result.

Example:

const run = await driver.execute('tvlabs: getWorkflowRun', { runId });

console.log(run);
// { runId: '…', workflowId: '…', status: 'failed', durationMs: 4210,
//   failure: { message: 'Home screen never appeared', stepId: 'w.yyxu6z', stepName: 'Open the home screen' } }

tvlabs: stopWorkflowRun​

Stops a run, and answers once it has stopped. Stopping a run that already finished returns its result unchanged.

Arguments:

  • runId (required, string)
    • The runId that tvlabs: executeWorkflow returned.

Returns: A run result, with status interrupted when the run was still going.

Example:

await driver.execute('tvlabs: stopWorkflowRun', { runId });

tvlabs: describeWorkflow​

Returns a workflow's name and project, and the key-values a run can override, so a test can check them before it starts a run.

Arguments:

  • workflowId (required, string)
    • The ID of the workflow.

Returns:

  • workflowId (string)
  • name (string)
    • The workflow's name.
  • projectId (string)
  • projectName (string)
  • keyValues (array)
    • One entry for each key-value a run can override:
    • name (string)
    • type (string): string, integer, number, boolean or enum.
    • default (any): the project's value. null for a secret key-value.
    • secret (boolean)
    • options (array of strings or null): the allowed values of an enum.

Example:

const workflow = await driver.execute('tvlabs: describeWorkflow', {
  workflowId: '<your workflow id>',
});

console.log(workflow.keyValues);
// [{ name: 'region', type: 'enum', default: 'us', secret: false, options: ['us', 'eu'] }]

Things to know​

  • The workflow drives the same device as your test. Other WebDriver commands sent while it runs can interfere with it, so wait for the run to finish before driving the device again.
  • A run starts on whatever is on screen. Unlike a run started from Workflows, there is no warmup and no app launch first, so a workflow that expects to start from a freshly launched app should open the app itself.
  • Avoid workflows with steps that end the device session or restart the device's controller. In a WebDriver session they end your test as well.
  • Runs belong to the WebDriver session that started them. Their results stay available until the session ends, and deleting the session stops a run still in progress.
  • When the workflow's Store Screenshots setting is All, the screenshots it uploads are saved to the session's Screenshots tab, as they are for a run started from Workflows.

Device Performance Metrics​

CPU and memory are sampled in the background for the lifetime of the session. These methods return the most recent sample, so they answer immediately without querying the device.

note

These methods are currently available on Roku, webOS, Tizen, Android TV (including Fire TV), Apple TV, iPhone, and iPad. Support for the remaining platforms is coming soon.

Apple TV, iPhone, and iPad additionally require the device to be running TV Labs' own driver stack. On an Apple device that isn't, no sampler starts and these methods report available: false with reason: "sampler_not_running" for the whole session. Reach out if you need metrics on an Apple device and see that reason persist.

None of these methods throw when a reading can't be produced. Every response carries an available flag:

  • available (boolean)
    • Whether a reading was available. When false, reason and message are the only other fields present
  • reason (string)
    • Present only when available is false. A stable snake_case identifier, safe to branch on. See the table below
  • message (string)
    • Present only when available is false. Human-readable detail, for logs and failure output. Never branch on it — it can embed a platform name or text supplied by the device itself
reasonMeaning
never_sampledA sampler is running but hasn't produced its first reading yet. Retry — webOS and Android TV sample about every 5 seconds
sampler_not_runningNo sampler is running for this session. Persists for the session's lifetime
unsupported_platformThe device's platform has no sampler
unavailableThe device declined to report. On Roku this means no developer channel is running
no_deviceNo device is bound to the session

Treat any other value as an unexpected failure and log message — new reasons may be added, and unknown is used for anything unforeseen.

const metrics = await driver.execute('tvlabs: getDevicePerformanceMetrics');

if (!metrics.available) {
  if (metrics.reason === 'never_sampled') {
    // Too early — try again shortly
  }

  console.log(`no reading: ${metrics.message}`);
  // no reading: unsupported platform: smartcast
}

Two fields are common to all three methods when available is true:

  • pts (number | null)
    • Presentation timestamp of the video frame current at the time of the reading, for lining a measurement up against the session recording. null when no frame is available
  • raw (object)
    • The platform's own metric fields, under their native names. The keys vary by platform

The normalized numbers describe whatever the platform measures. Roku reports the running channel; webOS, Tizen, Android TV, and Apple devices report the system as a whole. On webOS and Apple devices the foreground app's own CPU and memory are additionally reported under raw as the app_* keys, and on Fire TV thermal readings appear as the temp_* keys.

warning

available: true means the device reported a reading, not that every number in it is populated. Any normalized field can be null when the platform reported a sample without that particular value, so check before doing arithmetic on it:

if (metrics.cpu_percent !== null) {
  console.log(metrics.cpu_percent.toFixed(1));
}

tvlabs: getDevicePerformanceMetrics​

Returns the full reading — CPU and memory together, with every native field under raw.

Returns:

  • cpu_percent (number | null)
    • CPU utilization from 0-100
  • memory_used_bytes (number | null)
    • Memory in use, in bytes
  • memory_total_bytes (number | null)
    • Total memory available, in bytes. Always null on Roku, which measures the running channel rather than the system
  • pts, raw, available — as described above

Example:

// On a webOS device
const metrics = await driver.execute('tvlabs: getDevicePerformanceMetrics');

console.log(metrics);
// {
//   available: true,
//   cpu_percent: 23.4,
//   memory_used_bytes: 1342177280,
//   memory_total_bytes: 2147483648,
//   pts: 18432,
//   raw: {
//     cpu_utilization_pct: 23.4,
//     mem_total_bytes: 2147483648,
//     mem_free_bytes: 536870912,
//     mem_available_bytes: 805306368,
//     mem_cached_bytes: 268435456,
//     mem_used_bytes: 1342177280,
//     load_average_1m: 1.12,
//     uptime_seconds: 84021,
//     app_id: 'com.example.app',
//     app_cpu_pct: 8.1,
//     app_mem_resident_bytes: 209715200,
//     app_mem_virtual_bytes: 1073741824,
//   },
// }

To sample repeatedly over the course of a test, poll the method and record each reading against its pts:

const samples = [];

for (let i = 0; i < 10; i++) {
  const sample = await driver.execute('tvlabs: getDevicePerformanceMetrics');

  if (sample.available && sample.cpu_percent !== null) {
    samples.push(sample);
  }

  await driver.pause(1000);
}

const peakCpu = Math.max(...samples.map((s) => s.cpu_percent));

tvlabs: getDeviceCpuMetrics​

Returns just the CPU view of the latest reading. raw is narrowed to the platform's CPU-specific fields.

Returns:

  • cpu_percent (number | null)
    • CPU utilization from 0-100
  • pts, raw, available — as described above

Example:

// On a Roku device
const cpu = await driver.execute('tvlabs: getDeviceCpuMetrics');

console.log(cpu);
// {
//   available: true,
//   cpu_percent: 12.5,
//   pts: 18432,
//   raw: {
//     cpu_user_pct: 8,
//     cpu_sys_pct: 4.5,
//     cpu_total_pct: 12.5,
//     duration_seconds: 1,
//   },
// }

tvlabs: getDeviceMemoryMetrics​

Returns just the memory view of the latest reading. raw is narrowed to the platform's memory-specific fields.

Returns:

  • memory_used_bytes (number | null)
    • Memory in use, in bytes
  • memory_total_bytes (number | null)
    • Total memory available, in bytes. Always null on Roku, which measures the running channel rather than the system
  • pts, raw, available — as described above

Example:

// On a Roku device
const memory = await driver.execute('tvlabs: getDeviceMemoryMetrics');

console.log(memory);
// {
//   available: true,
//   memory_used_bytes: 209715200,
//   memory_total_bytes: null,
//   pts: 18432,
//   raw: {
//     mem_used_bytes: 209715200,
//     mem_anon_bytes: 167772160,
//     mem_file_bytes: 33554432,
//     mem_shared_bytes: 8388608,
//   },
// }

Network Control​

tvlabs: disableNetwork​

Disables the network connection for the device, simulating a network outage. Returns true if the network was successfully disabled, otherwise returns an error.

Example:

await driver.execute('tvlabs: disableNetwork');
note

If a network throttle is active, it will be disabled when tvlabs: disableNetwork is executed. After tvlabs: enableNetwork is executed, the network throttle command will need to be explicitly re-enabled if desired.

await driver.execute('tvlabs: enableNetworkThrottle', { limitDown: 5 });

// Network is throttled

await driver.execute('tvlabs: disableNetwork');

// Network is offline

await driver.execute('tvlabs: enableNetwork');

// Network is online, no throttling enabled

await driver.execute('tvlabs: enableNetworkThrottle', { limitDown: 5 });

// Throttling re-enabled

tvlabs: enableNetwork​

Restores the network connection after tvlabs: disableNetwork was executed. Returns true if the network was successfully enabled, otherwise returns an error.

Example:

await driver.execute('tvlabs: enableNetwork');

tvlabs: isNetworkEnabled​

Returns true if the network is enabled.

Example:

await driver.execute('tvlabs: disableNetwork');

const isNetworkEnabled = await driver.execute('tvlabs: isNetworkEnabled');

console.log(isNetworkEnabled);
// false

Network Throttling​

tvlabs: enableNetworkThrottle​

Enables network rate limiting for the device. Returns true if the rate limit was successfully enabled, otherwise returns an error.

Arguments:

  • limitDown (required, number)
    • The download limit in mbits / second
  • limitUp (optional, number | string)
    • The upload limit in mbits / second. Accepts "infinity" as a value.
    • Default: "infinity"

Example:

// Throttle download to 5 mbits / second
await driver.execute('tvlabs: enableNetworkThrottle', {
  limitDown: 5,
});

tvlabs: disableNetworkThrottle​

Disables network rate limiting for the device. Returns true if the rate limit was successfully disabled, otherwise returns an error.

Example:

await driver.execute('tvlabs: disableNetworkThrottle');

tvlabs: isNetworkThrottleEnabled​

Returns true if the rate limit is enabled.

Example:

await driver.execute('tvlabs: enableNetworkThrottle', {
  limitDown: 5,
});

const isNetworkThrottleEnabled = await driver.execute('tvlabs: isNetworkThrottleEnabled');

console.log(isNetworkThrottleEnabled);
// true

Network Usage​

tvlabs: getNetworkUsage​

Returns a snapshot of throughput and interface statistics for the device's network connection, measured over the most recent sampling interval (reported as interval_ms). Packet, error, drop, multicast, and collision values are the change observed during that interval, not cumulative totals.

Returns:

  • rx_kbps (number)
    • Download throughput in kbits / second
  • tx_kbps (number)
    • Upload throughput in kbits / second
  • rx_packets (number)
    • Packets received during the interval
  • tx_packets (number)
    • Packets transmitted during the interval
  • rx_errors (number)
    • Receive errors during the interval
  • tx_errors (number)
    • Transmit errors during the interval
  • rx_dropped (number)
    • Inbound packets dropped during the interval
  • tx_dropped (number)
    • Outbound packets dropped during the interval
  • multicast (number)
    • Multicast frames received during the interval
  • collisions (number)
    • Packet collisions detected during the interval
  • interval_ms (number)
    • Length of the sampling window in milliseconds

Example:

const usage = await driver.execute('tvlabs: getNetworkUsage');

console.log(usage);
// {
//   rx_kbps: 1.03,
//   tx_kbps: 0.67,
//   rx_packets: 2,
//   tx_packets: 1,
//   rx_errors: 0,
//   tx_errors: 0,
//   rx_dropped: 0,
//   tx_dropped: 0,
//   multicast: 0,
//   collisions: 0,
//   interval_ms: 1000,
// }

Network Capture​

More details on network capture functionality can be found on the main Network Capture page.

tvlabs: startNetworkCapture​

Starts network capture for the current session.

Arguments:

  • name (optional, string)
    • A name for the capture, used to identify it in the sessions dashboard
  • mode (optional, string)
    • Which capture method to use: "devtools" or "proxy" ("cdp" is accepted as another name for "devtools"). See Choosing the Capture Mode
    • Default: the platform's default, from the Supported Platforms table
    • Any other value falls back to the platform's default
  • direct_proxy (optional, boolean)
    • Enables direct proxy mode, where your application sends traffic to the proxy directly rather than having it intercepted transparently
    • Default: false
    • proxy mode only
    • Ignored on VegaOS, which always uses direct proxy mode
  • filters (optional, array)
  • reload (optional, boolean)
    • Whether to reload the page after starting network capture
    • devtools mode only
    • Default: true
  • cache (optional, boolean)
    • Whether to enable or disable the page cache
    • devtools mode only
    • Default: false

Example:

// Start a network capture with defaults
await driver.execute('tvlabs: startNetworkCapture');

// Start a named capture with direct proxy mode and a filter
await driver.execute('tvlabs: startNetworkCapture', {
  name: 'my-capture',
  direct_proxy: true,
  filters: [{ action: 'drop', mime_type: 'image/.+' }],
});

// Capture from the application's debugger instead of the platform's default
await driver.execute('tvlabs: startNetworkCapture', { mode: 'devtools' });

A capture must be stopped before starting another one in a different mode.

warning

On non-Chromium based platforms, after starting a network capture in an application that uses HTTP keep-alive, the application may need to be re-launched to ensure any open network connections are re-established and intercepted. This can be done with the terminateApp and activateApp methods.

On Tizen, webOS, and SmartCast, re-launching the app this way will also terminate the active ChromeDriver session — see ChromeDriver session lifecycle before doing this mid-test.

tvlabs: stopNetworkCapture​

Stops network capture and finalizes the HAR file for download. If the session network capture was started using capabilities, this method may be used to stop the capture before the session ends.

Arguments: None

Example:

// Stop an already running capture
await driver.execute('tvlabs: stopNetworkCapture');