An in-app assistant reads the page it's sitting on, returns a list of actions the client executes, then reads the page again.

The second week of a two-week build went to bonus features. One of them came from my boss: “see if we can add an AI assistant inside the platform that would be generic enough to where we didn’t have to rebuild the AI assistant, but also smart enough to where it could navigate the website itself as it changes.” The platform was an Architecture Decision Review Board — ADRB — for Cigna, and what made an assistant worth building there was the branching: “if you click on 100 users versus 10,000 users, it will branch off into a different part of the workflow. So it could get tedious for a person that doesn’t know the process to go through all of those screens.”

The assistant ran inside the app, not beside it. “Because the AI assistant was running inside of the UI — inside of React — I was able to, from React, interrogate what was currently on the page: all the text that was on the page, buttons that were on the page, capabilities, errors, etc.” The chat was a pop-up on that page. Behind it, the request carried the page context — “this is the current page, this is where you’re at within the current flow” — and the answer came back as something the client could run: “the AI response would come back in a formatted way that I’d be able to parse and do actionable things literally on the website, on behalf of the user.”

It was never one shot. “The response was a formatted JSON that would just give you a list of actions that you needed to do, and then once those actions were completed it would return back to the agent and say if there was any errors on the page or any new buttons or those kinds of things. So it would do the actions and then return back the state page.”

Why drive the screens instead of calling the backend API and skipping them: no deep rationale, and I’m not going to invent one. It was the ask, and the ask was worth taking seriously — “there’s a lot of utility in that… I think it’s showing the art of the possible.” The demo did what a demo does: “I could pull up the AI agent and just say, hey, I need X, Y, and Z, and it would see buttons being clicked and fields filled in, real time.”

A second week-two feature sets the constraint the rest of this note lives inside. Admins could edit the on-screen help text without a deploy — “effectively slugs for each place where you’d be able to put in text, store it in a database, and as the page loaded it would pull up the text and render it.” So the labels and the guidance the assistant reads are not fixed at build time. Put that next to an assistant that has to keep working “as it changes,” and one design option disappears: you cannot hand the model a hand-written map of the app. Whatever it gets has to be read off the live page, every turn.

None of the code below is from that build, which ran on a different SDK against a different model. It’s reference — a current-SDK sketch of the same loop, written from the docs.

What the page hands over

The serializer’s whole job is to turn the live page into something small enough to send and specific enough to act on. Three things travel: the readable text, the controls, and where the user is in the flow. Errors travel too, because the next turn is mostly about what just went wrong.

The load-bearing decision is how a control is addressed. The model never gets a CSS selector and never emits one. It gets an opaque id minted on this read, and the map from id back to element stays on the client:

// Runs inside the app; mints a fresh id per control on every read and
// keeps the id-to-element map on the client.

const INTERACTIVE =
  "button, a[href], input, select, textarea, [role='button']";

export type Ref = `e${number}`;

export interface Control {
  ref: Ref;
  role: string;
  name: string;
  value?: string;
  disabled?: boolean;
  gated?: boolean;
}

export interface PageState {
  route: string;
  step: { id: string; index: number; of: number };
  text: string[];
  controls: Control[];
  errors: string[];
}

const textOf = (root: HTMLElement, selector: string) =>
  [...root.querySelectorAll<HTMLElement>(selector)]
    .map((el) => el.innerText.trim())
    .filter(Boolean);

export function readPage(root: HTMLElement, step: PageState["step"]) {
  const targets = new Map<Ref, HTMLElement>();
  const controls: Control[] = [];
  let n = 0;

  for (const el of root.querySelectorAll<HTMLElement>(INTERACTIVE)) {
    if (el.closest("[hidden]")) continue;
    if (el.getAttribute("aria-hidden") === "true") continue;

    const ref = `e${n++}` as Ref;
    targets.set(ref, el);
    controls.push({
      ref,
      role: el.getAttribute("role") ?? el.tagName.toLowerCase(),
      name: (el.getAttribute("aria-label") ?? el.innerText).trim().slice(0, 120),
      value: el instanceof HTMLInputElement ? el.value : undefined,
      disabled: el.matches(":disabled") || undefined,
      gated: el.dataset.agentGate === "human" || undefined,
    });
  }

  const state: PageState = {
    route: location.pathname,
    step,
    text: textOf(root, "h1, h2, p, label, [data-help]"),
    errors: textOf(root, "[role='alert']"),
    controls,
  };

  return { state, targets };
}

Admin-edited help text needs no special handling here, which is the point: it arrives as text on the page like everything else. The [data-help] selector is there so the model reads the same guidance the person is reading, whoever last edited it.

Anthropic’s own browser tooling landed on the same addressing idea. Its browser use tool presents the page as an accessibility tree with element references — link "Getting started" [ref_2], textbox "Search docs" [ref_3] — so Claude can “act on an element by reference in addition to by coordinate.” A reference is a handle the executor controls. A coordinate is a guess about pixels.

The actions it may take

The action list is a schema, not a convention. Four verbs, each with exactly the fields it needs, nothing else accepted:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["reasoning", "actions"],
  "properties": {
    "reasoning": { "type": "string" },
    "actions": {
      "type": "array",
      "items": {
        "anyOf": [
          {
            "type": "object",
            "additionalProperties": false,
            "required": ["type", "ref"],
            "properties": {
              "type": { "const": "click" },
              "ref": { "type": "string" }
            }
          },
          {
            "type": "object",
            "additionalProperties": false,
            "required": ["type", "ref", "value"],
            "properties": {
              "type": { "const": "fill" },
              "ref": { "type": "string" },
              "value": { "type": "string" }
            }
          },
          {
            "type": "object",
            "additionalProperties": false,
            "required": ["type", "ref", "option"],
            "properties": {
              "type": { "const": "select" },
              "ref": { "type": "string" },
              "option": { "type": "string" }
            }
          },
          {
            "type": "object",
            "additionalProperties": false,
            "required": ["type", "question"],
            "properties": {
              "type": { "const": "ask" },
              "question": { "type": "string" }
            }
          }
        ]
      }
    }
  }
}

Three details in that schema are doing real work. anyOf over four closed objects is a discriminated union — and it is anyOf rather than oneOf because Anthropic’s structured outputs support anyOf and allOf but not oneOf. additionalProperties: false on every variant is not optional either; the same page says it “must be set to false for objects.” And ask is a first-class action, so the model has somewhere to go that isn’t guessing at a field it can’t read.

Then attach the schema to a tool with strict: true. Anthropic’s strict tool use documentation states it plainly: setting strict: true “guarantees Claude’s tool inputs match your JSON Schema by constraining the model’s token sampling to schema-valid outputs (a technique called grammar-constrained sampling).” Without it, the failure the docs name is exactly the one that breaks an executor — “Claude might return incompatible types ("2" instead of 2) or omit required fields.”

That gives you the first allow-list: the verbs. The second one is the executor’s, and it’s the one that matters more.

// The executor is the allow-list: an action can only name an id this
// read minted, and a gated control stops the run.

export async function execute(
  actions: Action[],
  targets: Map<Ref, HTMLElement>,
): Promise<Outcome[]> {
  const outcomes: Outcome[] = [];

  for (const action of actions) {
    if (action.type === "ask") {
      outcomes.push({ action, status: "asked_user" });
      break;
    }

    const el = targets.get(action.ref);
    if (!el) {
      outcomes.push({ action, status: "unknown_ref" });
      break;
    }
    if (el.dataset.agentGate === "human") {
      outcomes.push({ action, status: "needs_human" });
      break;
    }

    if (action.type === "click") {
      el.click();
    } else {
      const field = el as HTMLInputElement | HTMLSelectElement;
      field.value = action.type === "fill" ? action.value : action.option;
      field.dispatchEvent(new Event("input", { bubbles: true }));
      field.dispatchEvent(new Event("change", { bubbles: true }));
    }

    outcomes.push({ action, status: "done" });
    // Let the app re-render before the next action in the batch.
    await new Promise((resolve) => setTimeout(resolve, 0));
  }

  return outcomes;
}

A stale id is a feature. Ids are minted per read, so if the page moved underneath the model — a branch it didn’t expect, a field that vanished — the id resolves to nothing, the batch stops, and unknown_ref goes back as the result. The model finds out on the next turn, from the page, which is where it should be finding out.

The loop in code

The loop is an ordinary tool-use loop: the page state goes up as the user turn, the action list comes back as a tool_use block, and the executor’s outcomes plus the freshly re-read page go back up as the tool_result. Anthropic’s tool use documentation describes that round trip: “Claude responds with stop_reason: "tool_use" and one or more tool_use blocks. Your code executes the operation and sends back a tool_result.”

// Written against the current Anthropic TypeScript SDK.
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const ACT_ON_PAGE: Anthropic.Tool = {
  name: "act_on_page",
  description:
    "Propose the next actions on the page the user is looking at. " +
    "Address controls only by the ref ids given in the page state.",
  strict: true,
  input_schema: ACTION_SCHEMA,
};

const SYSTEM =
  "You are driving a review-board workflow on the user's behalf. Each turn " +
  "you get the page as it is right now: its visible text, its controls with " +
  "a ref id each, any errors, and the user's position in the flow. Propose " +
  "the smallest batch of actions that makes progress, then stop and wait for " +
  "the new page state. If the page does not tell you what to enter, use the " +
  "ask action instead of guessing.";

export async function drive(
  request: string,
  readState: () => PageState,
  runActions: (actions: Action[]) => Promise<Outcome[]>,
  maxTurns = 8,
) {
  const messages: Anthropic.MessageParam[] = [
    { role: "user", content: `${request}\n\n${JSON.stringify(readState())}` },
  ];

  for (let turn = 0; turn < maxTurns; turn++) {
    const response = await client.messages.create({
      model: "claude-opus-5",
      max_tokens: 16000,
      system: SYSTEM,
      tools: [ACT_ON_PAGE],
      messages,
    });

    // Echo the assistant turn back unchanged — on Opus 5 thinking runs by
    // default, and those blocks have to survive the round trip.
    messages.push({ role: "assistant", content: response.content });
    if (response.stop_reason !== "tool_use") return response;

    const results: Anthropic.ToolResultBlockParam[] = [];
    for (const block of response.content) {
      if (block.type !== "tool_use") continue;
      const { actions } = block.input as { actions: Action[] };
      const outcomes = await runActions(actions);
      results.push({
        type: "tool_result",
        tool_use_id: block.id,
        content: JSON.stringify({ outcomes, page: readState() }),
      });
    }

    messages.push({ role: "user", content: results });
  }

  throw new Error(`act_on_page did not settle in ${maxTurns} turns`);
}

The turn cap is not decoration. An agent that can click has a way to make the page change forever, and the only thing standing between a confused model and an infinite loop is a number you picked on purpose.

What it’s called

This pattern doesn’t have one name. No primary source names the whole of it — page state serialized, structured actions back, executed, re-read — as a single canonical term. The pieces each carry their own vocabulary instead: WebVoyager’s “Observation Space” and “Action Space,” Anthropic’s “agent loop” and “tool use,” Vercel’s “Tool Calling,” Fireworks’ “Observation-Decision-Action Loop.” What follows is a description, not a term of art.

The lineage is long, and it doesn’t start here. WebGPT (arXiv, 17 December 2021) put a language model in a text browsing environment with a fixed command vocabulary. ReAct (arXiv, 6 October 2022) named the interleaving of reasoning and acting — “reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources.” WebVoyager (arXiv, 25 January 2024) drove real websites end to end from screenshots plus the text of interactive elements. Anthropic shipped computer use on 22 October 2024, letting developers direct Claude “to use computers the way people do — by looking at a screen, moving a cursor, clicking buttons, and typing text.”

So the honest placement of my build is not first. It’s inside. Every precedent above works from outside the application — pixels, a scraped DOM, a driven browser — while this one is a component of the app, reading the app’s own rendered state, with the executor sitting in the same process as the thing being executed on. That’s a different set of trade-offs, not a better one: no vision model, no coordinates, no brittleness about layout, and no ability to leave the app or handle anything React didn’t render.

Two more sources are worth naming for calibration. Fireworks’ open-source browser agent write-up (21 May 2025) is the closest published implementation account I’ve read — capture DOM, screenshot, URL, title, and scroll position, emit structured JSON carrying evaluation, memory, next goal, and actions, then loop — and it went up roughly alongside my build, which says the shape was being reached for independently, not copied. Anthropic’s computer use tool went the other way: it is schema-less by design, and “you don’t need to provide an input schema as with other tools; the schema is built into Claude’s model and can’t be modified.” That’s the right call for a general desktop and the wrong one for a line-of-business app, where the whole safety story is that you wrote the action vocabulary yourself.

Where the gate goes

The prototype could hit Submit. “As far as authority goes, yes, it could go and hit Submit on the ADRB. Now, again, this is all just a proof of concept, just to be able, as a bonus, to show what the capabilities were.” I’d draw it differently today, and drawing it is cheap: “for the autonomy, I probably wouldn’t want it to hit Submit… to have that feature functionality put in there so that it can’t hit Submit is quite trivial, one line of code.” In the exhibit above, that line is data-agent-gate="human" on the button.

The line I actually hold isn’t about how big the action is — it’s about whether the outcome can be checked by something other than a person: “in the ADRB one it’s a business process that’s being followed, and there’s a lot more subjective nature to that versus a bug fix where you have lines of errors and you know the inputs and you know the expected output, and after you run the entire test suite, if it works you should be able to ship it.” Submitting an architecture decision has no test suite — it has a reviewer’s judgment, later. Gate it. That’s the same rule as autonomy expanding with instrumentation, not capability — you get to automate what you can verify.

The field’s guidance points the same way from a different direction. OWASP’s LLM06:2025, Excessive Agency defines the vulnerability as “damaging actions… performed in response to unexpected, ambiguous or manipulated outputs from an LLM,” and its mitigations are the two allow-lists in this note plus the gate: “limit the functions that are implemented in LLM extensions to the minimum necessary,” and “utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken.”

The assistant survived into production. What it’s allowed to do there, I don’t know: “I’m not sure what the team ended up doing as they moved this thing into production, if they kept that capability or if they wanted to limit it out.” No support requests reached me in the year after, and I have no usage figure to publish. The prototype showed the ceiling, and the team that owns it draws the line. That’s the right order.

Underneath

Docs checked 2026-09-02. The build ran on Vercel’s AI SDK — my recollection, not a record I can check — against a model reached through an internal gateway run by a different team inside the enterprise: “it’s essentially using LiteLLM, and they provisioned a key that we can access [the models] with.” I’m not printing the model’s minor version. My own recollection of it doesn’t survive checking, so it stays out.

Vercel’s own documentation calls its interface Tool Calling: a tool carries a description, an inputSchema — “a Zod schema or a JSON schema that defines the input parameters” — and an optional execute function, with multi-step runs continuing until a stopping condition is met. Same three parts as the Anthropic shape above, one abstraction layer up.

Related system

More field notes

Start