Hugging Face's logo

Spaces:
hinabarun
/
repro-incremental-bpe-tokenization
Running

App Files Community
repro-incremental-bpe-tokenization
File size: 4,087 Bytes
7ff5663
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Reproduction poster — Incremental BPE Tokenization</title>
<style>
  :root { --ink:#1a1f2e; --sub:#5b6478; --ok:#116b3c; --part:#8a6100; --na:#54428e; --line:#e3e6ee; --bg:#f7f8fb; }
  * { box-sizing: border-box; margin: 0; }
  body { font-family: system-ui, -apple-system, "Segoe UI", sans-serif; background: var(--bg); color: var(--ink); padding: 20px; }
  .poster { max-width: 980px; margin: 0 auto; }
  h1 { font-size: 1.45rem; letter-spacing: -.01em; }
  .sub { color: var(--sub); margin-top: 4px; font-size: .92rem; }
  .grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: 12px; margin: 18px 0; }
  .card { background: #fff; border: 1px solid var(--line); border-radius: 10px; padding: 14px; }
  .card h2 { font-size: .8rem; text-transform: uppercase; letter-spacing: .06em; color: var(--sub); }
  .verdict { font-weight: 700; margin: 6px 0 8px; font-size: .98rem; }
  .ok { color: var(--ok); } .part { color: var(--part); } .na { color: var(--na); }
  .card p { font-size: .85rem; color: var(--ink); line-height: 1.45; }
  .nums { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 12px; }
  .stat { background: #fff; border: 1px solid var(--line); border-radius: 10px; padding: 12px 14px; text-align: center; }
  .stat b { display: block; font-size: 1.35rem; letter-spacing: -.02em; }
  .stat span { font-size: .74rem; color: var(--sub); }
  .foot { margin-top: 16px; font-size: .8rem; color: var(--sub); }
  .foot a { color: inherit; }
</style>
</head>
<body>
<div class="poster">
  <h1>Reproduction: Incremental BPE Tokenization</h1>
  <div class="sub">Jiang &amp; Gong, ICML 2026 (OpenReview ZbWgrDzCQo · arXiv 2605.30813) — independent Python reimplementation, evidence-first, every number from a committed protocol-compliant run</div>

  <div class="grid">
    <div class="card"><h2>Claim 1 · Monotonic Path Property</h2>
      <div class="verdict ok">Reproduced (empirically)</div>
      <p>Zero violations across exhaustive candidate×history cross-products, per-byte upward-closure checks, 200 random dictionaries, and real 50k-vocab corpus runs.</p></div>
    <div class="card"><h2>Claim 2 · Algorithm design</h2>
      <div class="verdict part">Partially reproduced</div>
      <p>Composition verified end-to-end at real scale. Missing §5.3 / §6.2 mechanisms: their absence <em>measured</em> (tail latency linear in depth; eager 34× vs paper's ~10%).</p></div>
    <div class="card"><h2>Claim 3 · 3.13× drop-in speedup</h2>
      <div class="verdict na">Equivalence reproduced · speedup untestable</div>
      <p>Token-for-token identical to HF <code>tokenizers</code> over the paper's English corpus recipe. The speedup sits behind a measured 52× Python-vs-Rust constant — pre-declared scope limit.</p></div>
    <div class="card"><h2>Claim 4 · Pathological robustness</h2>
      <div class="verdict ok">Reproduced (era-appropriate baseline)</div>
      <p>Ours flat (slope 0.99); tiktoken 0.8.0 quadratic (2.04–2.08, incl. CL100K). Finding: current tiktoken fixed the quadratic upstream (1.14–1.16).</p></div>
  </div>

  <div class="nums">
    <div class="stat"><b>0.987</b><span>ours, log-log slope, real vocab ('a'*n)</span></div>
    <div class="stat"><b>2.041</b><span>tiktoken 0.8.0 on CL100K (quadratic)</span></div>
    <div class="stat"><b>114,399</b><span>tokens, zero divergences vs HF tokenizers</span></div>
    <div class="stat"><b>5.7×</b><span>our Python beats old Rust tiktoken @200 kB pathological</span></div>
    <div class="stat"><b>44 / 2,126</b><span>tests / subtests, green in CI</span></div>
  </div>

  <div class="foot">
    Evidence &amp; code: <a href="https://github.com/NabarunCode/icml-2026-reproductions">github.com/NabarunCode/icml-2026-reproductions</a> ·
    per-file independence record (PROVENANCE.md) · hash-pinned data · clean-tree environment records on every run
  </div>
</div>
</body>
</html>