There are two fundamentally different ways to let an AI agent control a browser. One is screenshot-based: take a picture of the page, have a vision-capable model interpret it, click at the coordinates it names. The other is structural: give the agent a machine-readable description of what's actually on the page — roles, names, hierarchy — and let it act on that directly. Playwright MCP is the second approach, and it's worth understanding even if you're not building an AI agent yourself, because it's a genuinely different way of thinking about what a page "is" to something automating it.
What it actually returns
Instead of a screenshot, Playwright MCP gives an agent an accessibility snapshot — the same kind of structural tree a screen reader would consume, and conceptually the same data a getByRole() locator resolves against:
- heading "Sauce Labs Backpack" [level=1]
- button "Add to cart"
- text "$29.99"
An agent working from this doesn't need to guess where a button is by looking at pixels — it acts on "the button named Add to cart" directly, the same way a well-written Playwright test does. This is more token-efficient than sending screenshots back and forth on every step, and it's deterministic in a way vision-based coordinate clicking generally isn't — the same accessible name resolves to the same element regardless of viewport size or how the page happened to render that particular run.
Setup, if you're driving this from an MCP-compatible client
{
"mcpServers": {
"playwright": {
"command": "npx",
"args": ["@playwright/mcp@latest"]
}
}
}
That's the whole install for most MCP clients — the server launches on demand, no separate browser or driver setup beyond what Playwright itself needs.
The vision-mode escape hatch
Structural snapshots break down exactly where regular accessibility-based Playwright locators break down: canvas-rendered UI, custom drawing surfaces, anything without a meaningful accessibility tree. For that case, Playwright MCP has an opt-in vision mode (--caps=vision) that switches to coordinate-based tools — browser_mouse_click_xy and similar — for pixel-level interaction when semantic targeting genuinely has nothing to grab onto.
Why this is worth knowing even if you're not automating with an agent
The interesting part, for anyone who already writes Playwright tests by hand, isn't the AI angle — it's that Playwright MCP is proof that the accessibility-tree-first approach to identifying elements (the same philosophy behind getByRole() as the top of Playwright's own locator hierarchy) generalizes past human-written test code. If an accessibility snapshot is precise and stable enough for an autonomous agent to reliably act on without a human reviewing each step, that's a fairly strong argument that it's precise and stable enough to be your first choice when you're writing the selector by hand, too — which loops back to a lesson from earlier debugging in this exact tutorial series: role-based selectors are only reliable when the markup actually carries the role, and checking that assumption takes ten seconds in dev tools either way, whether a human or an agent is the one that's going to depend on it.
