README.md

metadata

title: traceforge
emoji: 🧭
colorFrom: indigo
colorTo: green
sdk: gradio
sdk_version: 6.30.0
app_file: app.py
pinned: false
license: other
short_description: Turn real agent traces into ideal-world references

traceforge

Turn a real agent trace into its ideal-world reference: the same work, with the dead ends removed, gated for structure and grounding.

This Space is the portable face of the traceforge pipeline. The pipeline itself is unchanged — triage labels every message, a windowed rewriter produces the ideal path, and a judge admits only traces that pass the six pillars. The UI hides that machinery behind three actions.

Submit a trace

Upload a JSON/JSONL trace (traceforge native, OpenAI chat, or a bare message list) or paste it. The app computes a content hash of the seed and queues it.

One seed, one output. The hash covers the conversation content — message text, tool calls and the set of tools the agent had — and excludes session ids, timestamps, cwd and tool-call ids. Submitting the same trace twice returns the existing output instead of creating a second one. That is what makes the output dataset a set, not a log.

Sponsor a dataset

Upload a JSONL of traces. Every row is hashed; new seeds are queued and attributed to the sponsor name you give, known seeds are skipped. Processed rows join the public output dataset. The dashboard shows the sponsor leaderboard.

Dashboard

Public statistics over this Space's corpus and, when bundled, the reference corpus: seeds, outputs, queue, failure rate, task mix, languages, runtime distribution, and time sensitivity.

Compare seed → output

A diff of the seed against its ideal rewrite, one row per seed message, with the reason it changed: removed (a dead end the ideal path routes around), rewritten (same work, better shape — inline word diff), added (a turn the writer introduced), unchanged (collapsed). The reason is the triage verdict for that message, so the rewrite is auditable instead of opaque.

Model endpoint

The Space has no on-device model. It talks to an OpenAI-compatible endpoint configured with the TRACEFORGE_BASE_URL, TRACEFORGE_LLM_KEY and TRACEFORGE_MODEL secrets. Without them the app still starts and the dashboard still works; the Submit tab says the pipeline is not configured.

The rewrite is gated structurally, for grounding, and for identity, then a rewrite identical to the seed is rejected as a no-op.

A seed that already reads as ideal comes back as no change rather than an output or a failure — the pipeline had nothing to edit.

Time sensitivity

A trace whose correct answer is a fact that changes over time is a bad training target when the agent had no way to look that fact up. Train on it and the model learns to assert the fact from its weights — "the current president is X" — which is wrong the moment X changes.

The same question is a good target when the agent had a retrieval tool. The ideal path calls the tool and answers from its result, so what the model imitates is the procedure (search, read, answer), not the fact. The procedure does not decay.

class	meaning
time_sensitive	temporal fact, no retrieval tool — the answer decays
time_anchored	temporal fact, but a retrieval tool can fetch it
not_time_sensitive	no time-dependent fact in the task

time_anchored carries a sub-flag: whether the tool was actually called. A temporal task with a search tool available but never used is anchored in principle — the ideal rewrite should make the agent call it — but the raw trace answered from weights, so it needs a rewrite rather than a pass.

The detector is deliberately conservative: a false time_sensitive throws away a good trace, so a pattern must name the fact (an officeholder, a price, a released version, the weather), not just a temporal word. "The current branch" and "the latest fix round" are local artifacts of the session, not decaying facts.

Configuration

Set these as Space secrets / variables:

variable	meaning
TRACEFORGE_BASE_URL	OpenAI-compatible endpoint (…/v1)
TRACEFORGE_LLM_KEY	bearer token
TRACEFORGE_MODEL	writer/judge model id
TRACEFORGE_TRIAGE_MODEL	cheap labeler (default: TRACEFORGE_MODEL)
TRACEFORGE_JUDGE_MODEL	judge override (default: TRACEFORGE_MODEL)
TRACEFORGE_DATA_DIR	store root (default ./data/space)
TRACEFORGE_SPACE_WORKERS	parallel pipeline workers (default 1)

Without an endpoint the app still starts and the dashboard still works; the Submit tab reports that the pipeline is not configured instead of hanging.

Storage

The store is content-addressed on disk:

data/space/index.jsonl          append-only journal (last row per hash wins)
data/space/seeds/<hash>.json    the seed exactly as submitted
data/space/outputs/<hash>.json  the improved trace

A Hugging Face Space's disk is ephemeral unless persistent storage is attached. For a durable public corpus, mount persistent storage at /data and set TRACEFORGE_DATA_DIR=/data/space, or sync the store to a dataset repo.

Local use

pip install -e ".[webui]"
python app.py                       # http://127.0.0.1:7860
traceforge space serve --port 7860
traceforge space hash trace.json    # print the seed hash
traceforge space stats              # store census

License

Private research corpus. All rights reserved.