import HtmlEmbed from '../../components/HtmlEmbed.astro';
import Wide from '../../components/Wide.astro';
import Note from '../../components/Note.astro';
import Sidenote from '../../components/Sidenote.astro';

# The Opacity Problem

Watch one large language model (LLM) [agent](https://texonom.com/5efeaf8cd77f495988083eb2084d7f01) work and you get a wall of text. Watch a thousand of them and you get a machine: a handful of states the agent cycles through on almost every task, a `search`, `edit`, `execute` loop it never declares and the system prompt never specifies. When a run fails, that is where it broke. The catch is that nobody can see it.

This article recovers that machine. From nothing but the agent's own execution traces, with no labels and no task descriptions, we extract a compact [finite-state machine](https://texonom.com/6ccd60d415e74d07a615b9714ce83510), or FSM, of 7 to 43 states. That machine does the two things an operator actually needs: predict what the agent will do next, and catch a failing run before it wastes the compute.

The whole article follows one thread. Heterogeneous agent traces pass through one deterministic abstraction $\phi$, pile up into a prefix tree, and collapse in a single classical merge into one automaton. That one object, not four bespoke pipelines, is then read four different ways: as workflow memory, a next-step predictor, a failure detector, and a runtime monitor.

<Wide>
<HtmlEmbed
  src="system-overview.html"
  title="How the machine is built, and how it is used"
  caption="Data and execution flow. On the left, construction (offline, once per agent): each trace message is mapped to a symbol by φ, the symbol sequences accumulate into a prefix tree, and one state-merge collapses it into a single automaton. On the right, the four runtime consumers and what each reads from that same machine. Hover a consumer to trace it back to the automaton."
/>
</Wide>

LLM-based agents are now deployed across demanding domains: resolving GitHub issues [@yang2024sweagent; @yang2025swesmith], navigating websites [@zhou2024webarena], operating desktops [@xie2024osworld], managing customer service interactions [@yao2024taubench], and orchestrating multi-agent pipelines. Following the ReAct pattern, they interleave chain-of-thought reasoning with tool calls, and every run produces an execution trace of tool calls, natural language, and environment feedback. The **behavioral structure** governing that trace stays implicit.

A coding agent cycles through `search`, `edit`, `execute`. A customer service agent alternates between database queries and user communication. This structure emerges from the interaction between the system prompt, the available tools, and the task distribution, yet it is **nowhere written down**.

Understanding this latent structure matters the moment you deploy. Safety auditing needs to verify that an agent visits the right states and avoids attack chains. Debugging needs to locate the bottleneck states where agents get stuck. Production monitoring needs to flag behavioral drift before it costs anything.

Yet current agent analysis works at the level of a single trace. It offers no structural model of the behavior that links one run to the next.

<Wide>
<HtmlEmbed src="tape-to-fsm.html" title="A trace, read symbol by symbol, drives a finite-state machine" desc="Pick a dataset, then scrub the tape (drag or scroll) or let it play. The read-head advances through the agent's activity tape (left) and lights up the current FSM state and the transition it takes (right). A successful run threads to submit; a stuck run loops and never reaches it. These are representative traces; the full set of twelve datasets with their real extracted FSMs is in the live dashboard." />
</Wide>


## The Inverse Problem

We frame behavioral recovery as an **inverse problem**: given a corpus of execution traces, reconstruct a finite-state machine that explains the observed behavior. This is grammatical inference [@gold1967language; @oncina1992rpni], but the classical setting assumes both positive and negative examples. Agent traces give us positive examples only, the runs that happened, with no labeled counter-examples. Identifying the target language from positive examples alone is **impossible in the limit** [@gold1967language].

What rescues the problem is a property specific to agents. Unlike arbitrary regular languages, agent behavior is generated by a **bounded set of tools and actions**, so traces draw from a small activity alphabet, 6 to 42 symbols across our twelve datasets.

<Note>
A small alphabet is the whole reason this works. It makes the compact automaton both small and, as later chapters show, statistically dense enough to predict from. The only modeling choice in the entire pipeline is how a message becomes a symbol.
</Note>

## Twelve Datasets, Eight Domains

We evaluate on twelve public datasets spanning **coding**, **web navigation**, **desktop GUI**, **mobile GUI**, **customer service**, and **multi-agent coordination**. The trace counts below are the trajectories we use in our experiments, from 184 to 8,337 each, not the size of each source corpus: for the largest sources we draw a fixed slice rather than the whole set.

| Dataset | Domain | Traces used | Actions | States | Fitness |
|:--------|:-------|-----------:|--------:|-------:|--------:|
| SWE-smith | Coding | 500 | 9 | 10 | 1.000 |
| SWE-agent | Coding | 2,000 | 24 | 25 | 0.999 |
| Mind2Web | Web | 500 | 7 | 8 | 1.000 |
| WebArena | Web | 8,337 | 24 | 25 | 1.000 |
| AgentNet | Desktop GUI | 5,000 | 24 | 25 | 1.000 |
| GUI-Odyssey | Mobile GUI | 7,735 | 6 | 7 | 1.000 |
| Who & When | Multi-agent | 184 | 8 | 9 | 1.000 |
| tau2-bench airline | Customer service | 800 | 17 | 18 | 1.000 |
| tau2-bench retail | Customer service | 1,824 | 18 | 19 | 1.000 |
| tau2-bench telecom | Customer service | 1,824 | 42 | 43 | 1.000 |
| ATBench | Safety | 1,000 | 14 | 15 | 1.000 |
| OSWorld | Desktop OS | 2,166 | 26 | 27 | 0.997 |

Every dataset replays held-out traces at fitness of at least 0.997.

The three largest source datasets are subsampled to a fixed slice: SWE-agent uses 2,000 of the 80,036 available trajectories, Mind2Web 500 of 2,350, and AgentNet 5,000 from the OpenCUA Ubuntu subset. The other nine datasets are used in full.

<Note variant="info" title="Every case is in the live dashboard">
Every figure in this article is a fixed snapshot; **every dataset and every case is live** in the [interactive dashboard](https://seongland.com/article/asg/browser), which renders all twelve datasets straight from the experiment outputs — the FSMs, the failure predictor, the runtime monitor, and more.
</Note>