# Addon SDK reference

> The Modern addon execution context — adapters, runtime variables, AI, OCR, the underlying Playwright and WebdriverIO objects, and what is refused.

The second argument to `execute()` is the execution context. Which fields it carries depends on the platforms declared in `applicationType` and the permissions requested. Requesting nothing still gives you logging, a scratch directory, runtime variables, and an abort signal.

This reference covers the Modern (TypeScript) SDK. Classic addons use Java and Selenium directly.

## Context fields

| Field | Available | Purpose |
|---|---|---|
| `ctx.ui.browser` | Platforms include **WEB** or **MOBILE_WEB** | Drive the page under test |
| `ctx.ui.mobile` | Platforms include **ANDROID** or **IOS** | Drive the app under test |
| `ctx.runtime` | Always | Read and write runtime variables by name |
| `ctx.logger` | Always | Structured logging: `debug`, `info`, `warn`, `error` |
| `ctx.tmpDir` | Always | Per-step scratch directory, auto-cleaned, always readable and writable |
| `ctx.signal` | Always | `AbortSignal` that fires if the run is cancelled |
| `ctx.execution` | Always | Read-only metadata: `sessionId`, `applicationType`, result identifiers |
| `ctx.ai` | With `permissions.ai` | Platform-brokered LLM generation |
| `ctx.ocr` | With `permissions.ocr` | Platform-brokered OCR and image finding |
| `ctx.spawn` | With `permissions.spawn` | Host-brokered subprocess, local agent only |

Honor `ctx.signal` in loops and long waits, so a cancelled run stops promptly.

## What differs from a normal test

Your addon runs in a worker while the browser lives in the host process. That has 3 consequences, and each one produces a bug that looks like something else.

- **Await everything**, including calls that are synchronous in Playwright, such as `page.url()`. Composing without awaiting is still free, so `page.getByRole("form").locator("input")` reaches the host only once, when you finally await it
- **Await every assertion**: generic matchers return a promise here. An un-awaited failing assertion becomes an unhandled rejection, and the step passes when it should not
- **No closures**: function arguments are rejected. Anything evaluating in the browser or on the device takes a string instead, including `evaluate("el => el.textContent")`, `waitForFunction`, `$$eval`, and WebdriverIO's `execute`

## Browser adapter

| Method | Notes |
|---|---|
| `locate(elementRef \| selector)` | Returns a locator handle for a declared element or an ad-hoc selector |
| `goto(url, opts?)` | `waitUntil: "load" \| "domcontentloaded" \| "networkidle"` |
| `screenshot(opts?)` | `{ fullPage?: boolean }`, returns a Buffer |
| `title()` and `url()` | Current page title and URL |
| `evaluate(scriptString, args?)` | Has to be a string. Function values cannot be transferred |
| `waitForSelector(sel, opts?)` | `{ state?: "attached" \| "visible" \| "hidden", timeout? }` |

A locator handle supports `click()`, `fill(text)`, `textContent()`, `getAttribute(name)`, `isVisible()`, `count()`, `boundingBox()`, and string-based `evaluate()`.

## Mobile adapter

`locate()` accepts a declared `ElementRef` or an ad-hoc `{ strategy, value }` literal. The strategies are `accessibilityId`, `id`, `xpath`, `className`, `androidUiAutomator` for Android, and `iosPredicate` or `iosClassChain` for iOS.

| Surface | Methods |
|---|---|
| Element handle | `tap()`, `longPress({ms?})`, `fill(text)`, `clear()`, `text()`, `getAttribute(name)`, `isVisible()`, `boundingBox()`, `swipe(direction)`, `count()` |
| Device | `screenshot()`, `getContexts()`, `getCurrentContext()`, `switchContext(name)`, `getOrientation()`, `setOrientation(...)`, `hideKeyboard()` |

A platform-specific strategy used on the other platform fails at run time. Coordinates from `boundingBox()` are device pixels, so divide by the device pixel ratio for density-independent units.

Prefer `ctx.ui.mobile.locate(...)` over `driver.$(...)` where it suffices. `locate()` applies UiAutomator `resourceId` wrapping, the iOS class-name predicate workaround, and xpath normalization that the raw driver does not.

### Waiting on mobile

WebdriverIO's `waitUntil` is unavailable, because its condition callback runs in Node and no string form can replace it. Use `ctx.ui.mobile.waitUntil`, which re-evaluates a driver expression on the device each interval.

```ts
await ctx.ui.mobile.waitUntil(driver.$("~ok").isDisplayed(), { timeout: 10_000 });
```

Pass the expression without awaiting it. Awaiting first would check it once, and is refused with an explanation. The options are WebdriverIO's `timeout`, `interval`, and `timeoutMsg`.

## The underlying libraries

`ctx.ui.browser` and `ctx.ui.browser.page` are the same object at 2 levels of abstraction. Where the adapter does not reach far enough, every session also exposes the real automation object and the real assertion library.

| Platform | Automation | Assertions |
|---|---|---|
| Web | `ctx.ui.browser.page`, a Playwright `Page` | `ctx.ui.browser.expect` |
| Mobile | `ctx.ui.mobile.driver`, a WebdriverIO driver | `ctx.ui.mobile.expect` |

No permission declaration is needed for any of them. These are the genuine libraries rather than a curated subset, so every matcher, `.not`, `.soft`, and the generic value matchers work as documented.

Reach for them for frames, keyboard and mouse, dialogs, downloads, and accessibility snapshots on web, or raw Appium commands on mobile.

### Structural validation with aria snapshots

Asserting the shape of a form or menu in one call beats a pile of per-element checks. Matching is a subset check, so unrelated markup can change freely, and it auto-retries like any Playwright assertion.

```ts
const page = ctx.ui.browser.page;
const expect = ctx.ui.browser.expect;

await expect(page.getByRole("form", { name: "Filters" })).toMatchAriaSnapshot(`
  - heading "Filters"
  - textbox "Search"
  - button "Apply"
`);

const snapshot = await page.getByRole("form", { name: "Filters" }).ariaSnapshot();
```

## What is refused

Your addon is one step inside someone else's session, so anything that outlives the step or hands over credentials is refused with a clear error.

| Group | Members |
|---|---|
| Persistent callbacks | `on`, `once`, `route`, `exposeFunction`, `addInitScript` |
| Session lifecycle | `close`, `pause`, `setDefaultTimeout`, `deleteSession`, `reloadSession`, `addCommand` |
| Escapes that outlive the step | `newPage`, `newCDPSession`, `tracing`, `browser` |
| Credentials, mobile | `options`, `requestedCapabilities` |

What remains available is worth knowing. One-shot `waitForEvent(...)` covers dialogs, downloads, and popups. `capabilities` stays available, so you can branch on `platformVersion`. And `page.context()` and `page.request` both work, so cookies, `storageState()`, and API calls are all fine. Only the dangerous members inside them are refused.

File paths passed to `screenshot({ path })`, `Download.saveAs()`, `setInputFiles()`, or WebdriverIO's savers have to sit under `ctx.tmpDir`. Anywhere else is refused.

## Runtime variables

```ts
const v = await ctx.runtime.get("orderId");   // undefined when unset
await ctx.runtime.set("orderId", "ORD-1042");
await ctx.runtime.set("token", jwt, { isEncrypted: true });
```

## AI

```ts
const answer = await ctx.ai.invoke("Summarize this receipt", {
  files: ["receipt.png"],
});
```

File paths are relative to `ctx.tmpDir`, or within your `fs` allowlist.

## OCR

| Method | Returns |
|---|---|
| `extractTextFromPage()` | Recognized text spans with bounding boxes for the current page |
| `extractTextFromImage(path)` | The same, for an image file you supply |
| `extractTextFromElement(elementRef)` | The same, for one element. Web only |
| `findImage(refImage, opts?)` | `{ isFound, x1, y1, x2, y2 }` |

`findImage` accepts `threshold`, `scale`, and `occurrence` options.