Web interaction modes | Technical Whitepaper

# Web interaction modes

Playground supports four Docker / no-use.computer ways to run persona web scenarios against live public URLs, plus CUA in Docker via persona-computer-1. Pick the mode that fits the study — none is the global default.

Quick pick

| Mode | Task example | Agent | Environment | When to use | |------|----------------|-------|-------------|-------------| | Playwright | application/tasks/example-web-playwright_quote-choice/ | persona-openhands-sdk | docker + network_mode = "public" | Terminal + Python Playwright; CI-friendly, lower cost | | browser-use | application/tasks/example-web-browser-use_laptop-choice/ | persona-browser-use | docker + network_mode = "public" | Dedicated browser agent loop; persona via extend_system_message | | Cocoa | application/tasks/example-web-cocoa_plan-choice/ | persona-cocoa | docker + AIO Sandbox image + network_mode = "public" | Unified browser + shell + files in one container | | CUA | application/tasks/example-web-cua_bookshop-choice/ | persona-computer-1 | docker (Linux Xvfb) | Screenshot loop in Linux desktop; finish with done action | | CLI (experimental) | any live-web task above | persona-claude-code / persona-codex / persona-gemini-cli | application/shared-web-cli (Playground override) | Terminal CLI + Playwright; same task instruction/verifier |

Checked-in smoke recipes: [configs/jobs/example-job-recipe/](../../configs/jobs/example-job-recipe/) (appSim-example-web-playwright-local.yaml, …-browser-use-local.yaml, etc.).

Shared submission contract

Live-web browse-and-choose tasks in this repo share a persona decision JSON shape. Each task names its output file under /app/output/:

| Task | Output file | Live URL | |------|-------------|----------| | Playwright quote | quote_choice.json | https://quotes.toscrape.com/ | | browser-use laptop | laptop_choice.json | https://webscraper.io/test-sites/e-commerce/static/computers/laptops | | Cocoa plan | plan_choice.json | https://www.pythonanywhere.com/pricing/ | | CUA bookshop | book_interest.json | https://books.toscrape.com/ |

Example shape (fields vary slightly per task — follow each task's instruction.md):

`json { "decision_subject_id": "", "decision_subject_label": "", "decision_outcome": "selected", "basis_primary": "<price|quality|features|convenience|taste|trust|familiarity|novelty|fit|other>", "exploration_style": "<quick_pick|compared_multiple|deep_research|hesitant>", "reason": "<why this matched you as this persona>" } <code></code>` Verifiers check schema and task-specific required fields, not semantic match to the live catalog (pages change). On use.computer macOS (computer-use tasks only), Matraix Playground remaps <code>/app</code> → <code>/Users/lume</code> in shell commands. Verifiers resolve this automatically; instructions always say <code>/app/output/</code>. Full metric contract: [task-spec/web/README.md](../../application/task-spec/web/README.md). <h2 id="playwright-mode">Playwright mode</h2> How it works: Chromium is controlled through the Playwright API (DOM selectors). The agent (via OpenHands terminal) runs Python that calls <code>page.goto()</code>, <code>locator()</code>, etc. Pros: Faster, cheaper, more repeatable than CUA; JavaScript-rendered pages work. Cons: Not a literal “human looking at the screen”; anti-bot / complex UX may need extra handling. API key: <code>persona-openhands-sdk</code> requires <code>LLM_API_KEY</code> on the host (not <code>ANTHROPIC_API_KEY</code>). If you only have Anthropic set up: <code></code>`bash export LLM_API_KEY="${ANTHROPIC_API_KEY}" <code></code>` <code></code>`bash uv run harbor run \ -a persona-openhands-sdk \ -m anthropic/claude-sonnet-4-6 \ --ak persona_path=persona/datasets/matraix-persona-dev-sample/persona_0042.yaml \ -p application/tasks/example-web-playwright_quote-choice <code></code>` Or the checked-in smoke recipe: <code></code>`bash uv run harbor run -c configs/jobs/example-job-recipe/appSim-example-web-playwright-local.yaml <code></code>` Oracle (no LLM): <code></code>`bash uv run harbor run -p application/tasks/example-web-playwright_quote-choice -a oracle <code></code>` task.toml: set <code>[environment].network_mode = "public"</code> and <code>[agent].network_mode = "public"</code>. Runtime: <code>environment/task-environments/application/shared-web-playwright/</code>. <h2 id="browser-use-mode">browser-use mode</h2> How it works: The [browser-use](https://github.com/browser-use/browser-use) library runs an agent loop over Chromium (DOM tools + optional vision). Persona maps to <code>extend_system_message</code>; the task instruction stays in the <code>task</code> field. Pros: Purpose-built web agent; MIT license; no <code>use-computer</code> cost. Cons: Slower than hand-written Playwright; less flexible than full terminal access. API key: <code>ANTHROPIC_API_KEY</code>, <code>OPENAI_API_KEY</code>, or <code>LLM_API_KEY</code> (mapped by model provider). <code></code>`bash uv run harbor run -c configs/jobs/example-job-recipe/appSim-example-web-browser-use-local.yaml <code></code>` Oracle: <code></code>`bash uv run harbor run -p application/tasks/example-web-browser-use_laptop-choice -a oracle <code></code>` Runtime: <code>environment/task-environments/application/shared-web-browser-use/</code>. <h2 id="cocoa-mode-aio-sandbox">Cocoa mode (AIO Sandbox)</h2> How it works: The task Docker image is [<code>agent-infra/sandbox</code>](https://github.com/agent-infra/sandbox) (browser + shell + files on <code>localhost:8080</code>). [CocoaAgent](https://github.com/cocoabench/cocoa-agent) connects in-process with <code>skip_docker</code>. Persona is merged into the instruction (same slot as OpenHands / computer-1). Pros: All-in-one digital agent without <code>use-computer</code>; no Docker socket mount. Cons: Heavier base image than Playwright-only tasks. <code></code>`bash uv run harbor run -c configs/jobs/example-job-recipe/appSim-example-web-cocoa-local.yaml <code></code>` Runtime: <code>environment/task-environments/application/shared-web-cocoa/</code>. <h2 id="cua-mode-chromium-computer-use">CUA mode (Chromium + computer-use)</h2> How it works: A real desktop browser window in Docker (Xvfb + XFCE); each turn the model receives a screenshot and returns actions (<code>navigate</code>, <code>click</code>, <code>scroll</code>, …). The agent finishes with a done JSON action; the task verifier recovers <code>/app/output/book_interest.json</code> from the mirrored final answer / trajectory when needed. Pros: Closest to end-user behavior among Docker web modes; no <code>USE_COMPUTER_API_KEY</code> for Docker Linux web CUA. Cons: Slower and costlier than Playwright/browser-use; first run builds a desktop image. <code></code>`bash uv sync --extra computer-1 export ANTHROPIC_API_KEY=... uv run harbor run -c configs/jobs/example-job-recipe/appSim-example-web-linux-cua-local.yaml <code></code>` Oracle: <code></code>`bash uv run harbor run -p application/tasks/example-web-cua_bookshop-choice -a oracle <code></code>` Runtime: <code>environment/task-environments/application/shared-web-cua-linux/</code>. For macOS / iOS screenshot CUA (system settings, not live web), use <code>application/tasks/example-computer-use-*</code> with <code>-e use-computer</code> — see [agents.md](agents.md). <h2 id="what-we-do-not-treat-as-a-web-mode">What we do not treat as a web mode</h2> | Approach | Status | |----------|--------| | [Skyvern](https://github.com/Skyvern-AI/skyvern) | No <code>persona-skyvern</code> in this repo; use browser-use or CUA for vision-first browsing. | | <code>curl</code> / <code>wget</code> only | Not a web interaction mode — no JS, no layout. OK for smoke, not persona web browsing. | | Mock HTML sidecar | Deprecated for web examples here; use live URL tasks under <code>application/tasks/example-web-*</code>. | <h2 id="cli-harness-on-web-tasks-experimental">CLI harness on web tasks (experimental)</h2> Browser harness tasks keep their own <code>shared-web-*</code> Docker images. When Playground (or a Matraix Playground job) selects a CLI harness on a web task (<code>persona-claude-code</code>, <code>persona-codex</code>, <code>persona-gemini-cli</code>), the launcher stages a copy of the task with `[environment].definition = application/shared-web-cli`. That image includes: <ul class="article-list"> <li>Playwright + Chromium — same baseline as <code>shared-web-playwright</code> so the</li> </ul> CLI agent can write terminal Python to browse live URLs. <ul class="article-list"> <li>Terminal CLI deps — <code>curl</code>, <code>python3</code>, <code>uv</code>, and pre-installed Claude Code</li> </ul> (Codex / Gemini install via Matraix Playground <code>install()</code> at trial start, same as survey/chat). It does not include browser-use, Cocoa, or CUA stacks — those are separate browser products and are not used by CLI harnesses. Task <code>instruction.md</code>, verifier, and output contract stay unchanged; only the runtime image switches. Credentials: API keys or CLI subscription auth are configured on the Matraix Playground runner — see [agents.md](agents.md) § CLI subscription auth and <code>application/playground/.env.local.example</code>. <h2 id="authoring-a-new-live-web-application">Authoring a new live-web application</h2> 1. Choose a mode from the table above. 2. Copy the closest <code>example-web-*</code> task; reuse the decision JSON contract or document a new schema inline in <code>instruction.md</code> and the task README. 3. Set <code>network_mode = "public"</code> where the agent must reach the internet. 4. Point <code>[environment].definition</code> at the matching <code>shared-web-*</code> runtime (or create a task-specific environment only when the stack is genuinely new). 5. Register the task for Playground — [task-guide.md § Playground registration](../application/task-guide.md#playground-registration). 6. Add Suggested setup (non-binding) in <code>tasks/.../README.md</code> — do not put agent names in <code>instruction.md</code>. 7. Document URL stability and login requirements in README Known limitations. <h2 id="reference-tasks">Reference tasks</h2> | Task | Mode | |------|------| | <code>application/tasks/example-web-playwright_quote-choice/</code> | Playwright + live URL | | <code>application/tasks/example-web-browser-use_laptop-choice/</code> | browser-use + live URL | | <code>application/tasks/example-web-cocoa_plan-choice/</code> | Cocoa + live URL | | <code>application/tasks/example-web-cua_bookshop-choice/</code> | CUA + live URL (Docker Linux) | See also [task-guide.md](../application/task-guide.md) and [agents.md](agents.md). Play tasks in the Playground: [quickstart.md §10](../quickstart.md#10-playground-play-tasks-visually). </section> </article>  <aside class="toc-sidebar"> <div class="toc-title">Contents</div> <ul class="toc-list"> <li><a href="#quick-pick">Quick pick</a></li> <li><a href="#shared-submission-contract">Shared submission contract</a></li> <li><a href="#playwright-mode">Playwright mode</a></li> <li><a href="#browser-use-mode">browser-use mode</a></li> <li><a href="#cocoa-mode-aio-sandbox">Cocoa mode (AIO Sandbox)</a></li> <li><a href="#cua-mode-chromium-computer-use">CUA mode (Chromium + computer-use)</a></li> <li><a href="#what-we-do-not-treat-as-a-web-mode">What we do *not* treat as a web mode</a></li> <li><a href="#cli-harness-on-web-tasks-experimental">CLI harness on web tasks (experimental)</a></li> <li><a href="#authoring-a-new-live-web-application">Authoring a new live-web application</a></li> <li><a href="#reference-tasks">Reference tasks</a></li> </ul> </aside> </main>  <footer class="site-footer"> <div style="max-width: 1200px; margin: 0 auto;"> SWARPH AI — AGNOSTIC MULTI-AGENT COORDINATION SUBSTRATE © 2026 Swarph AI · BrainSurfing Research · MetaEdge Security </div> </footer> </body> </html>