control_url is a WebSocket that carries it;
inside a Halyard sandbox, a phone also answers it on a local socket. The Python
driver speaks it for you (browser, phone,
computer),
but the connection is yours: open it directly to drive a session from any
language.
How the wire looks
DCP is framed like the Chrome DevTools Protocol: each frame is one JSON object, with no envelope around it. A command:id:
Domain.method. Commands on one connection run one at a time,
in the order sent. Timestamps are epoch milliseconds.
| Field | On | Meaning |
|---|---|---|
id | command, response | An integer you choose, unique on the connection; the response echoes it. |
method | command | Domain.method, such as Locator.click. |
params | command | The method’s arguments. Omit or send {} when it takes none. |
targetId | command | The tab a target-scoped method acts on. Omit for the session’s current tab. |
frameId | command | The frame a frame-scoped method (Locator, DOM, CSS, Page.evaluate, Page.waitForFunction) acts in, from Frame.getFrames. Omit for the tab’s main frame. Cross-origin frames work too. |
result | response | The result on success, at least {}. |
error | response | { code, message, data: { kind, retryable, reason? } } on failure. See Error kinds. |
seq | response | The session event log’s position before the command acted. See Event log. |
Two tiers of methods
- The action tier does the work a person would:
Locator.clickfinds an element, waits until it is visible, stable and enabled, scrolls it into view and clicks it with real input, all in one round trip. Your client never polls. - The raw tier is input at a point and reads of the screen:
Pointer.clickon a browser,Touch.tapandTouch.swipeon a phone,Keyboard.typeText,Screen.screenshot, and on a phoneScreen.observe, which returns every text region OCR found and every icon on screen.
Locator.tap where a browser answers Locator.click: a phone
taps.
What each session serves
| Domain | What it does | Browser | Phone | Computer |
|---|---|---|---|---|
Protocol | The handshake: protocol version and the exact methods served. | Yes | Yes | Yes |
Device | What the session runs on. | Yes | Yes | Yes |
Screen | Screenshots; on a phone, also what is on screen. | Yes | Yes | Screenshots |
Keyboard | Typing and key presses at the focus. | Yes | Yes | Yes |
Pointer | Mouse input at a point. | Yes | Yes | |
Locator | Find a target, wait for it, then read it or act on it. | Yes | Yes, a subset | Yes: native apps, and its browser’s pages |
Browser | Browser version and window size. | Yes | On its browser | |
Target | Tabs: list, open, close, switch, wait for one. | Yes | On its browser | |
Page | Navigation, page info, scripts, content and the aria snapshot. | Yes | On its browser | |
Storage | Cookies. | Yes | On its browser | |
Touch | Tap, long press and swipe at a point. | Yes | ||
Accessibility | The accessibility tree of a phone, or of a computer’s native apps. | While the phone has a tree | getFullAXTree and queryAXTree | |
Display | The desktop’s resolution. | Yes | ||
Window | Top-level windows. | Yes | ||
Process | Commands and programs. | Yes | ||
File | Files. | Yes | ||
Clipboard | Clipboard text. | Yes |
StrategyUnavailable. The Frame,
DOM, CSS and
Network domains are a browser’s, served on a browser
session and on a computer’s browser. Read the
handshake’s capabilities for the exact set: a phone
lists the Accessibility methods only while it has an accessibility tree to
serve.
Coordinates
On a browser, points and boxes default to window pixels: Chrome’s whole window as the live view shows it, from its top-left. Pass"space": "viewport"
on Pointer.*, Locator.boundingBox and Screen.screenshot to use the page’s
CSS pixels from the viewport’s top-left instead, as Playwright does. A point
outside the window answers InvalidArgs.
On a computer, everything is screen pixels: the desktop’s own, from its
top-left ("space": "screen", the only space and the default there). It covers
the whole desktop, the taskbar included, and window bounds use it too.
On a phone, everything is screen pixels: the phone’s frame from its
top-left, at the size Device.info reports (screenWidth, screenHeight).
Touch.* points, Screen.observe boxes, Locator result bounds and
accessibility node bounds all use it.
How a computer differs
A computer session drives a whole Windows desktop rather than one browser:- The whole desktop. The taskbar, every window and every program are there;
Screen.screenshotand the live view show all of it, the mouse cursor included. - Every chord reaches Windows. The input safety a browser applies (refusing
the Windows key,
Alt+Tab,Alt+F4and the rest) doesn’t apply: a computer is yours to drive. - Typing. Characters a US keyboard has are pressed as keys; the rest are pasted through the clipboard, which is put back afterwards.
- The operating system.
Display,Window,Process,FileandClipboardreach past the screen. Commands run as the session’s user, who has administrator rights: the machine is the customer’s alone and is reset after the session.C:\axholds the session’s own software and is read-only exceptC:\ax\downloads; Windows’ own processes and the session’s agent and streamer can’t be ended.
StrategyUnavailable.
The browser on a computer
A computer also serves every browser method, on a Chrome on its own desktop: the computer is the base and the browser sits on top of it, so one session can read a website and work with the desktop’s apps and files. Chrome opens on the first browser call (one already open is used) and again after it closes. The SDK reaches it ascomputer.browser,
a Browser like a browser session’s.
Screen, Pointer, Keyboard and Locator are in both. A call that names a
page’s targetId acts in that page, in its coordinates. One without acts on the
desktop, in screen pixels. So a Locator call with no targetId resolves on the
desktop’s UI Automation tree: the native apps’ windows, buttons and fields. The
Locator methods only a page has (Locator.evaluate, Locator.blur,
Locator.dragTo, Locator.dispatchEvent, Locator.setInputFiles) always go to
the browser, to its current tab when no targetId is named. Locator.innerHTML
needs a page’s targetId; without one the desktop answers StrategyUnavailable.
Accessibility.getFullAXTree and Accessibility.queryAXTree read the same UI
Automation tree. Roles there are UI Automation control types (windows.button,
windows.edit), and each node carries its bounds in screen pixels. Before the browser’s input acts, Chrome’s window
goes back where the browser methods put it and comes to the front, so another
window never catches a click meant for the page.
Timeouts
| Kind of call | Field | Default | Most |
|---|---|---|---|
| Locator actions and reads (both) | timeoutMs | 5,000 | 60,000 |
| Navigation and page waits (browser) | timeoutMs | 30,000 | 120,000 |
Window.waitFor (computer) | timeoutMs | 5,000 | 60,000 |
Process.run (computer) | timeoutMs | 30,000 | 600,000 |
ActionTimeout naming the check that failed. On a phone, a vision model that is
already running when the budget ends is allowed to finish.
Event log
On a browser, each response carriesseq, the session event log’s position just before the
command acted. Waits that take afterSeq only consider what happened after it,
so “click, then wait for the tab it opens” never matches a tab that was already
open:
afterSeq
answers EventsExpired.
Limits
| Limit | Value |
|---|---|
| Script result (browser) | 1 MiB |
| File read or write (computer) | 32 MiB |
Process.run output (computer) | 1 MiB per stream |
File.list entries (computer) | 10,000 |
| Aria snapshot | 20,000 characters, then cut with a marker line |
| Selector nesting | 4 levels |
| Navigation schemes | http, https, data, blob and about:blank |
Next steps
Drive a session with raw DCP
Open the socket, handshake, and send your first command.
Selectors and strategies
How Locator methods find their target on each kind of session.
Reconnect and resume
Dropped sockets, cursors and idempotency keys.
Locator methods
Every Locator method, its params and its result.