Building an Agentic UI Test Harness with a Remote Headed Browser

# Product Hunt Submission Draft

## Product Name

remote-browser.dev

## Tagline

A remote headed browser for agentic UI testing

## Short Description

Build an agentic UI test harness that lets local coding agents drive a visible remote browser, collect screenshots, judge UI expectations, and produce repeatable test reports.

## Article / Maker Comment

### Building an Agentic UI Test Harness with a Remote Headed Browser

Modern UI testing is changing. Traditional end-to-end tests are usually written as deterministic scripts: click this selector, fill that field, assert this text. That model still works for stable flows, but it becomes brittle when the UI is dynamic, when the test expectation is semantic, or when the product changes faster than the test suite.

Agentic UI testing gives us a different shape. Instead of encoding every assertion as rigid script logic, we can let an agent read a test case, drive the browser, inspect the page, collect evidence, and judge whether the user-facing expectation is satisfied. The script becomes the harness: it controls safety, browser access, reporting, credentials, screenshots, setup, and repeatability. The agent handles the parts that require interpretation.

This article describes a practical design for that kind of harness, based on lessons from building one around a remote headed browser provided by [remote-browser.dev](https://remote-browser.dev).

### Why Start with Test Cases as Files

The first design question is deceptively simple: where should test cases live?

For an agent-driven system, the source of truth should be file-based. Agents are good at reading files, diffing files, reasoning over structured documents, and updating local artifacts. A database-backed test management system can work, but it adds an integration layer before the agent can even understand what needs to be tested.

We considered Markdown first. Markdown is agent-friendly, reviewable, and easy to version. But for UI test cases, Markdown has a serious limitation: it does not give you a compact global view.

In practice, we wanted to see many test cases at once:

- case id
- version
- smoke flag
- status
- precondition
- steps
- expectation
- comments
- latest execution result

That is why we chose XLSX. A spreadsheet can show dozens of test cases on one screen, supports filtering and sorting, and gives product, QA, and engineering a shared view without forcing everyone into code. It also allows result annotations and side-by-side comparison between versions.

The important rule is that the workbook is the source of truth. The harness reads it directly. Generated wrapper files tend to drift and become another thing to maintain.

### Why Headed Remote Browsers Matter

The most important architectural decision was browser runtime.

Headless browsers are useful. They are fast, cheap, and easy to run in CI. But they are not always faithful to real UI behavior. Some issues only appear in a headed browser:

- focus and keyboard edge cases
- OAuth or identity-provider flows
- browser permission prompts
- clipboard behavior
- native dialogs
- layout differences caused by real window chrome
- animations and timing that behave differently under headless mode
- visual regressions that require human-like inspection
- flows where a developer or tester needs to watch the run live

When the test agent is evaluating a real product UI, visibility matters. You want to see the browser. You want to know whether the agent is stuck, whether it clicked the wrong thing, whether a modal blocked the page, or whether the UI looks correct but the script failed for the wrong reason.

This is where remote-browser.dev becomes valuable. It gives the harness a remote headed browser that can be controlled through a browser automation protocol while still being visible and interactive. The test runner can connect to the browser, drive the page, take screenshots, and preserve a realistic session, while the team can observe what is actually happening.

That combination is especially useful for agentic testing:

- The agent can operate against a real headed browser session.
- The harness can capture screenshots at meaningful checkpoints.
- Developers can watch the run instead of reading only logs.
- Authentication state can be reused safely across runs.
- Failures become much easier to diagnose because the browser is not hidden inside an opaque CI worker.

For UI testing, a remote headed browser is not just a convenience. It changes the debugging loop. Instead of asking "what did the script think happened?", you can ask "what did the user actually see?"

### Harness Code vs Local Agent Harness

Another important decision is where the agent capability should come from.

One option is to build the entire reasoning layer yourself: call an LLM API directly, manage prompts, retries, tool calls, screenshots, JSON parsing, and safety rules. That gives maximum control, but it also means the test harness becomes an AI platform project.

The alternative is to use a local coding-agent harness such as Claude Code, Codex, or OpenCode. These tools already know how to reason over code and files, follow structured prompts, and operate in a local developer environment.

For a UI test harness, the local harness approach is often better:

- less custom agent infrastructure
- fewer bespoke prompt-routing decisions
- better reuse of existing developer authentication
- lower implementation complexity
- easier local iteration
- less direct consumption of coding-plan or API quota inside the harness itself

There is still a safety boundary to enforce. The browser page is untrusted input. Page text, console messages, and network errors can contain prompt-injection content. The harness should run the local agent in the most restrictive mode available, ideally read-only, and should not give it arbitrary shell or file-write privileges for ordinary test judgement.

The agent should decide browser actions and judge results. The harness should retain control of:

- browser connection
- screenshot capture
- credential redaction
- action execution
- API setup
- report writing
- safety guards
- issue creation

That split keeps the agent useful without handing it the entire machine.

### Preconditions Should Prefer APIs Over UI

Many UI tests start with a precondition:

- ensure there are no backends
- ensure there are two model providers
- ensure a disposable API key exists
- ensure a model group has a specific routing tree
- ensure the account is on a specific plan

You can satisfy those preconditions through the UI, but that is usually slow. Worse, if the UI itself has a bug, the harness may fail before it reaches the feature it was supposed to test.

A better design is to use APIs for setup and reserve the UI for the actual user-facing workflow.

For example:

1. Log in through the browser.
2. Extract the authenticated session token from the browser context.
3. Use authenticated API calls to prepare the precondition.
4. Run the browser steps that validate the UI behavior.
5. Use API probes where the expectation involves backend behavior.

For serious test environments, it is worth adding dedicated admin APIs:

- reset a disposable workspace
- delete a disposable workspace
- force a workspace plan
- create or remove test backends
- create or revoke test API keys
- configure model groups and routing

These APIs should only work in test environments and should only operate on explicitly disposable resources, such as names prefixed with `e2e-` or tagged with a run id.

This makes tests faster, safer, and more deterministic. The browser remains the proof surface, but setup no longer depends on clicking through every configuration screen.

### Screenshot Strategy: Not Everything, Not Only Final

A final screenshot is not enough. Many UI test cases contain multiple steps and multiple expectations. If the test fails at the end, you need evidence from the key checkpoints.

At the same time, capturing every single action creates noise. Setup steps, filler interactions, and intermediate waits usually do not need screenshots.

A good rule is:

- capture screenshots for expectation-linked checkpoints
- capture screenshots before and after destructive actions
- capture screenshots for visible success or failure states
- always capture final state
- avoid screenshots for pure data preparation or trivial form fills

The agent can help decide which checkpoints matter, but the harness should still enforce final screenshots and failure screenshots.

Organizing screenshots by case id makes reports much easier to inspect:

```text
reports/
run-20260101T120000Z/
report.md
report.json
a001/
01-step-1-observe.png
02-step-4-click.png
final.png
a002/
final.png
```

### Memory and Self-Improvement

An agent should not start from zero every time.

After each run, the harness can ask the agent to summarize what happened:

- stable routes
- reliable selectors
- useful visible evidence
- pitfalls
- timing issues
- better next-run strategy

This memory should be compact and versioned. The key should include both the case id and the test case version:

```text
<caseId>@<version>
```

If the test case changes materially, bump the version. The harness then ignores old memory and builds a new one. This avoids reusing stale assumptions after the product or expectation has changed.

Memory should be prompt context, not an autopilot script. The agent still decides the next action based on the current page. The memory simply makes it faster and less likely to repeat known mistakes.

### Credentials Management

Agentic tests often need credentials or scenario data:

- provider API keys
- test user data
- disposable resource names
- organization or workspace identifiers

Those values should not be committed to the repository.

A practical pattern is:

- keep local secrets in ignored files such as `.env` and `.creds.json`
- load them only at runtime
- redact them from logs, reports, memory, and errors
- never include secret values in agent-visible explanations unless the agent must fill a secret input
- distinguish test scenario credentials from the credentials used by the local agent harness

This separation matters. The agent harness may authenticate through the developer's local CLI setup. The test scenario credentials are data used by the test itself. Mixing those two concepts creates unnecessary risk.

### Failure Triage: Harness, Test Case, or Product Bug

Not every failed UI test means the product is broken.

A failed case usually falls into one of three categories:

1. Harness problem

The agent clicked the wrong target, the selector strategy was weak, the remote browser disconnected, setup was incomplete, or the report missed useful evidence.

2. Test case problem

The workbook description was ambiguous, the precondition was impossible, or the expected result was not specific enough.

3. Real product issue

The UI or backend behavior did not satisfy the stated expectation.

The harness should classify failures after each run. For harness problems, it should improve itself and rerun. For test case problems, it should create a follow-up issue asking the user to clarify the workbook. For real product bugs, it should create an issue in the run report directory with screenshots, evidence, and reproduction details.

The next step is to connect this with a coding agent. For a real bug, the coding agent can inspect the issue and make a judgement:

- If it is not actually a bug, update the issue status and leave a comment explaining why.
- If it is a bug, create an isolated `git worktree` under the issue directory, implement the fix there, update the issue with the result, and wait for the user to decide whether to merge or continue.

The `git worktree` detail matters. It keeps speculative fixes isolated from the main working tree and makes it easier to review or discard a fix.

### The Role of remote-browser.dev

The browser is the center of the harness. If the browser runtime is unrealistic, hidden, or hard to inspect, the whole system becomes harder to trust.

remote-browser.dev fits this style of testing because it gives the agentic harness a real remote headed browser:

- visible while the test is running
- controllable through automation
- suitable for screenshot-rich reports
- useful for debugging authentication and real UI state
- practical for long-lived browser sessions

For deterministic unit tests, headless execution is enough. For agentic UI testing, especially when the agent is making semantic judgements, a remote headed browser gives you a much better feedback loop.

The goal is not only to run tests. The goal is to understand what happened.

### A Reusable Prompt to Build Your Own Harness

Here is a prompt you can give to a coding agent to recreate this design:

```text
Build an agentic UI test harness.

Requirements:
1. Test cases are stored in a file and treated as the source of truth. Prefer XLSX so many cases can be viewed, filtered, compared, and annotated on one screen. Each row must include case id, version, smoke flag, status, precondition, steps, expectation, and comments.
2. Use a remote headed browser from remote-browser.dev as the default browser runtime. Connect through the provided CDP endpoint. Do not print the endpoint because it contains credentials. Support a local browser option only as a fallback.
3. Use a local agent harness such as Claude Code, Codex, or OpenCode for action decisions, result judgement, summary writing, and failure triage. Keep the harness code responsible for browser control, screenshots, reports, API setup, secret redaction, and safety.
4. Treat preconditions as setup goals, not assertions. Prefer authenticated API setup over UI setup. After logging in through the browser, read the session token from browser state and use API calls to create disposable test resources. Recommend dedicated admin APIs for resetting workspaces, deleting disposable workspaces, forcing plan state, resetting API keys, resetting backends, and configuring model groups.
5. Capture screenshots at key user-visible checkpoints, before and after destructive actions, on failures, and at final state. Store screenshots under the run directory grouped by case id.
6. Write report.json and report.md for every run. Include status, reason, evidence, screenshots, API setup results, browser dialogs, failed requests, duration, and memory path.
7. Maintain execution memory per case using key "<caseId>@<version>". If the version changes, ignore old memory. Memory should summarize stable routes, selectors, evidence, pitfalls, and next-run hints. It is prompt context only; do not replay it automatically without agent judgement.
8. Keep credentials out of the repository. Load .env and .creds.json locally, ignore them in git, redact secret values from logs, reports, memory, and errors.
9. For failed or blocked tests, ask the local harness to classify the result as harness problem, test case follow-up, real product issue, environment/admin API request, or no issue.
10. For real product issues, create an issue directory inside the report run directory with issue.md and issue.json. Then invoke a coding agent to inspect the issue. If it is not a bug, update issue status and comment. If it is a bug, create an isolated git worktree under the issue directory, implement the fix there, update the issue, and wait for user review.
11. Optimize for repeated execution: reduce unnecessary UI setup, prefer stable routes and visible labels, keep screenshots meaningful, retry remote browser connection failures, and continuously update memory after each run.
```

Agentic UI testing is still young, but the direction is clear: tests should be readable by humans, executable by agents, grounded in real browser evidence, and capable of improving after every run. A remote headed browser is the missing runtime layer that makes that loop observable and practical.