README.md

metadata

title: LLM Text Watermark Microscope
emoji: 🔬
colorFrom: blue
colorTo: indigo
sdk: static
app_file: index.html
pinned: false
license: apache-2.0
short_description: Inspect three generations of LLM text watermarks in-browser

LLM Text Watermark Microscope

An interactive research/education tool that shows how LLM text watermarking evolved — not just that watermarks work, but what happens inside the decoding loop when they do. You supply the secrets and walk the decoding forward one stage at a time.

The screen is split: an unwatermarked baseline on the left, held fixed for the prompt, and the watermark you are studying on the right. The baseline mirrors whatever stage you are on, so every claim has a control beside it. Each method is a four-stage wizard that unlocks as you go:

stage	Kirchenbauer	k-SemStamp	TextSeal
1	Seed & Split — type a seed, watch the vocabulary partition, click a token for the arithmetic	Seed & Clusters — which semantic clusters your secret allows next	Seed & R-values — the PRF numbers behind each candidate
2	Bias & Sampling — move δ and see any real decoding step re-weighted	Rejection Sampling — candidate sentences accepted/rejected live	Gumbel Sampling — dual-key routing vs. the baseline draw
3	Result — every generated token coloured by the list it came from	Result — every sentence by its cluster verdict	Evidence & Regions — per-token evidence heatmap, localized spans
4	Edit & Detect — all three detectors re-score as you type

The four runs share sampling parameters:

run	year	carrier	detection
Baseline	—	none	(reference)
Kirchenbauer et al.	2023	token identities (green/red vocabulary partition, +δ logit bias)	one-proportion z-test on green counts
k-SemStamp (Hou et al.)	2024	sentence semantics (k-means clusters over embeddings, rejection sampling)	z-test on valid-region sentence counts
TextSeal (Sander et al., Meta FAIR)	2026	distortion-free Gumbel-max sampling (dual-key PRF)	entropy-weighted Gamma p-value + localized multi-region detection

Every stage exposes its internals rather than summarizing them: PRF seeds and the U-vs-γ comparison for a token you click, per-step logits before and after biasing, candidate sentences with their cluster assignments and accept/reject reasons, per-token Gumbel evidence, and the localized region scan. Stage 4 runs all three detectors on whatever you type, which is where the generations visibly separate — token-level edits destroy Kirchenbauer evidence, meaning-preserving rewrites leave k-SemStamp intact, and excerpting shrinks a global score while the localized scan still finds the span. Each method exports its seed, parameters, trace and scores as JSON.

Measured on a run of 96 tokens with Qwen3-0.6B (your numbers will differ): Kirchenbauer reached z = 5.48 against a baseline of −0.61; k-SemStamp reached z = 3.00 against 2.31; TextSeal's unweighted score sat just under threshold at −log₁₀p = 3.39 and crossed it at 4.51 once entropy weighting was enabled — a direct demonstration of what that part of the paper buys you.

Everything runs in your browser

• No server GPU, no API keys. Model inference runs on your GPU via WebGPU (Transformers.js v3 / ONNX Runtime Web). Your prompt and generated text never leave the browser.
• Default model: Qwen3-0.6B q4f16 (570 MB download, cached by the browser; thinking mode disabled). Swappable to Qwen2.5-0.5B-Instruct (483 MB) or SmolLM2-360M-Instruct (~273 MB).
• Sentence embeddings for k-SemStamp: EmbeddingGemma-300m (~167 MB, downloaded only when you open that method). It is multilingual, so the semantic clusters mean something in every language the demo model can write.
• Fallbacks: GPUs without shader-f16 get q4; browsers without WebGPU fall back to WASM (CPU — slow; keep max tokens low). Chrome/Edge ≥ 113 recommended.

All four runs share one custom autoregressive loop (manual forward() + KV cache — the framework's generate() is bypassed so each algorithm can hook logits or replace sampling). Baseline and Kirchenbauer consume the same base RNG stream with the same seed, isolating the watermark's effect from sampling randomness. TextSeal replaces sampling with a deterministic PRF rule and k-SemStamp's rejection loop changes randomness consumption, so stream-sharing is impossible there — the UI says so rather than hiding it.

What this tool is not

This is a mechanism verifier, not an AI-text detector. It checks for a specific watermark inserted with a known secret key. No detection ≠ human-written; detection ≠ proof of authorship. The editor also demonstrates honestly that iterative editing against a visible score degrades detection — detectability and unremovability are different properties.

Faithfulness notes (educational deviations from the papers)

component	reference	this implementation	consequence
Kirchenbauer PRNG	torch.Generator (MT19937) + randperm, seed = 15485863 × prev_token mod 2⁶⁴−1	same seed formula, but splitmix64-based per-token judgment U(seed, key, id) < γ, with the user's seed mixed in at the judgment rather than the seed so the displayed derivation stays literal	green-list size is Binomial(|V|, γ) instead of exactly γ|V|; z-test unaffected. Not interoperable with the reference detector
Kirchenbauer scheme	lefthash and selfhash	lefthash only	selfhash (context 4 + self-salt) not implemented
k-SemStamp encoder	SBERT fine-tuned with paraphrase-contrastive loss per domain	general-purpose EmbeddingGemma-300m, called through AutoModel for its sentence_embedding output (the feature-extraction pipeline would mean-pool hidden states and skip the model's dense projection)	paraphrase robustness weaker than published results, but multilingual
k-SemStamp centroids	k-means on 8k in-domain paragraphs	k-means (K=8) on ~440 wikitext-2, narrative and Korean sentences, shipped as JSON (scripts/compute-centroids.mjs). The script uses the same encoder and the same task: clustering | query: prefix as the browser — a mismatch there would put centroids and runtime assignments in different spaces	clusters are generic; margin default relaxed (0.01 vs paper 0.035)
k-SemStamp trials	MAX_TRIALS = 100	default 16 (browser latency), adjustable; each retry raises the temperature slightly, capped at 1.5x, because an unbounded climb buys candidate diversity by turning the text into gibberish	occasional forced accepts, shown in the trace
k-SemStamp counting	every sentence scored	a sentence in the same cluster as the one before it is not counted again, and fewer than three independent sentences reports inconclusive	consecutive same-cluster sentences re-ask one question ("is q in valid(q)?"), so counting them separately turns one coin flip into a confident-looking z. This is the same reasoning as ignore_repeated_ngrams on the Kirchenbauer side
TextSeal PRF	cryptographic hash (paper appendix A.1 / official repo)	splitmix64 mixing over (key, k=3 context tokens, candidate)	statistically equivalent construction, not interoperable with official TextSeal
TextSeal localization penalties	exact combinatorial multi-region penalty	per-region log₁₀M + log₁₀Y_max simplification, ensemble +log₁₀3	slightly conservative p-values
TextSeal entropy weights	teacher-forced model entropies	optional (one extra forward pass ≤512 tokens); default unweighted	unweighted variant is also calibrated, per the paper

KV-cache rollback for k-SemStamp re-computes the prefix on rejection instead of snapshotting GPU tensors — simpler and leak-free at demo scale.

Reproducibility

seed, secret key, sampling parameters, model repo/dtype and per-algorithm parameters are all visible in each result's metadata and adjustable under Advanced settings. Identical (model, prompt, parameters, key, seed) reproduce identical Baseline/Kirchenbauer outputs; TextSeal is deterministic given the key and routing seed. WebGPU floating-point differences across GPU vendors may change low-probability tie-breaks; bit-level determinism across devices is not guaranteed.

Development

npm install
npm run dev        # local dev server
npm test           # 74 unit + integration tests (node; mock model, no GPU needed)
npm run build      # production build (dist/)
node scripts/compute-centroids.mjs   # regenerate k-SemStamp centroids (downloads the encoder)

Architecture: src/lib/model (adapter + custom generation loop) · src/lib/watermark (algorithms) · src/lib/detectors · src/lib/workers/inference-worker.ts (dedicated worker owning the model) · src/lib/stores/app-state.ts (per-method wizard state) · src/components/stages/<method> (the stage screens) · src/components/BaselinePanel.svelte (the mirror view).

Citations

@inproceedings{kirchenbauer2023watermark,
  title={A Watermark for Large Language Models},
  author={Kirchenbauer, John and Geiping, Jonas and Wen, Yuxin and Katz, Jonathan and Miers, Ian and Goldstein, Tom},
  booktitle={ICML}, year={2023}
}
@inproceedings{hou2024ksemstamp,
  title={k-SemStamp: A Clustering-Based Semantic Watermark for Detection of Machine-Generated Text},
  author={Hou, Abe Bohan and Zhang, Jingyu and Wang, Yichen and Khashabi, Daniel and He, Tianxing},
  booktitle={Findings of ACL}, year={2024}
}
@article{sander2026textseal,
  title={TextSeal: A Localized LLM Watermark for Provenance and Distillation Protection},
  author={Sander, Tom and Chang, Hongyan and Sou{\v{c}}ek, Tom{\'a}{\v{s}} and others},
  journal={arXiv preprint arXiv:2605.12456}, year={2026}
}