Spaces:
Running
Claim 2: Develops efficient heuristic algorithm allowing providers to significantly overc…
Board claim 2: Develops efficient heuristic algorithm allowing providers to significantly overcharge without raising suspicion at cost lower than additional revenue
Stated verdict: VERIFIED (9 match / 1 partial / 1 no_match)
The replication largely supports the efficient-heuristic-algorithm claim: Algorithm 1 was shown to have an optimal, unimodal number of iterations, verification costs that are low and roughly independent of sequence length relative to generation costs, and the ability to find concrete plausible longer tokenizations that avoid raising suspicion, all yielding a net utility gain (C3, C7, C8, C13, C15, C2, C4, C9 matched), and it outperforms the naive random-splitting baseline whose plausibility collapses as iterations increase (C5 partial, C6 matched). Only the precise headline overcharge-percentage numbers (C1) failed to replicate exactly, due to a smaller-scale evaluation forced by compute constraints rather than a flaw in the algorithm.
C1 [headline/scalar] — status no_match
Claim: Under pay-per-token with transparency (top-p sampling) and temperature 1.3, the heuristic reporting policy (Algorithm 1) allows a provider serving Llama-3.2-1B-Instruct to overcharge users by up to 11.2%, 1.8%, and 0.28% (percentage of tokens overcharged) for p = 0.99, 0.95, 0.90 respectively.
Replicated: [18.82, 0.19, 0.08] Paper: [11.2, 1.8, 0.28]
Grading rule: [0] rel err 68.04% > 30% → no_match; [1] rel err 89.44% > 30% → no_match; [2] rel err 71.43% > 30% → no_match
Verifier rationale: [deterministic grade] [0] rel err 68.04% > 30% → no_match; [1] rel err 89.44% > 30% → no_match; [2] rel err 71.43% > 30% → no_match. Comparator notes: The overcharge peak is the max over split-depth m of the 'over' curve, top_p_count[i]*splits[i]/sum(generated_outputs)*100, computed per replicate then averaged (src compute_curves in notebooks/plots_heur_new.ipynb). The executed notebook (notebooks/plots_heur_new_exec.ipynb, cell 7, 'L1B T=1.3 over') loads via load_heur_triple, which prefers the highest-num_seq .pkl; that resolves to the agent-regenerated numseq_4 pickles under outputs/heuristic_new/L1B/ and writes figures/heur/Llama_1B_topp_T13_over_axis.pdf (the expected filename Llama_1B_topp_T13_over_main.pdf was not produced under that name). Recomputing those exact numseq_4 files gives peaks of 18.82% (p=0.99, m=90), 0.19% (p=0.95, m=3), 0.08% (p=0.90, m=1) — i.e. the actual reproduction. These are far from the paper's 11.2%/1.8%/0.28%: p=0.99 is inflated and p=0.95/p=0.90 are 9x and ~3.5x too low. The p=0.99 file contains only 4 prompts x 4 reps (1sigma=5.99, so 18.8 is within ~1.3sigma of 11.2), but the p=0.95/p=0.90 files have 40 prompts x 4 reps with tiny 1sigma (0.025, 0.028), so their ~10x/3.5x shortfalls are robust, not sampling noise. All three elements exceed 30% relative error, hence no_match. This is largely explained by scale: the plan (steps 2-3) intended ~600 LMSYS prompts, but fix_severity.json documents a 'major' fix — the shipped verify_sampling_conditions was ~100x slower than a forward pass and intractable at paper scale — leaving the agent's regeneration drastically under-sampled (4/40 prompts). Notably, recomputing the SAME 'over' formula on the author-shipped full-scale data still present in the repo (numseq_3, 400 prompts) yields 9.58%/1.64%/0.31%, which is consistent with the paper within the stated ~90% CI ranges; but that is the paper's own data, not this run's reproduction, so the run's produced figure does not support the claim.
C2 [headline/scalar] — status match
Claim: Using the optimal number of iterations m and temperature 1.3, the heuristic reporting policy (Algorithm 1) offers higher average utility than truthful reporting for Llama-3.2-1B-Instruct: the utility gain surpasses 10.5% for p=0.99 regardless of margin, and reaches 1.7% and 0.4% for p=0.95 and p=0.90.
Replicated: [10.47, 1.75, 0.39] Paper: [10.5, 1.7, 0.4]
Grading rule: [0] rel err 0.29% ≤ 5% → match; [1] rel err 2.94% ≤ 5% → match; [2] rel err 2.50% ≤ 5% → match
Verifier rationale: [deterministic grade] [0] rel err 0.29% ≤ 5% → match; [1] rel err 2.94% ≤ 5% → match; [2] rel err 2.50% ≤ 5% → match. Comparator notes: The expected figure figures/heur/profit/L1B_profit_1.3_main.pdf was produced, and the executed notebook notebooks/energy_profit_plots_exec.ipynb (cell 13, the final utility/profit plot for model='L1B', temp='1.3') prints the exact per-p limiting utility gains: 'Lim 0.99: 10.467004849343816', 'Lim 0.95: 1.75334304444311', 'Lim 0.9: 0.3862699552768707'. These are the 'Increase in utility (%)' values combining the Table-1 energy ratio (L1B gen/score ratio 0.174, i.e. c_v verification cost) with the heuristic-misreporting plausibility pickles per Eq. 4, at the optimal m per p (m=45/25/20 for p=0.99/0.95/0.90). The replicated triple [10.47, 1.75, 0.39] matches the paper's [10.5, 1.7, 0.4] to within ~0.3%, ~3.1%, and ~3.4% relative error respectively — all inside the 5% match tolerance (and within +/-1 for the small absolute values). The replicated p=0.99 value of 10.47% is essentially at the paper's '>10.5%' threshold. fix_severity.json lists major/critical fixes, but those concern the Algorithm-2 random-split reimplementation, verification-speed batching, and n-gram-ban consistency for the split-generation experiments; this claim relies on the deterministic Algorithm-1 heuristic plausibility outputs and energy pickles, which reproduce the reported values cleanly.
C3 [headline/table] — status match
Claim: Per-token generation energy cost (co), per-output plausibility-verification energy cost (cv), and their ratio co/cv, measured on a single Nvidia H100 with temperature 1 and KV caching, for the five studied LLMs.
Verifier rationale: [deterministic grade] 15 cell(s), 0 outside tolerance → match. Comparator notes: The five shipped energy snapshots (outputs/energy_outputs/energy_results_.json, each a list of 1000 per-prompt records with gen_energy, score_energy, out_length) reproduce Table 1 exactly using the notebook's own methodology (energy_profit_plots.ipynb cell 7: co = mean(gen_energy/out_length), cv = mean(score_energy), ratio = co/cv). Computed vs paper: L1B co=2.642/2.642, cv=15.189/15.188, ratio=0.174/0.174; L3B 4.806/4.805, 15.903/15.901, 0.302/0.302; G1B 6.482/6.481, 12.737/12.73, 0.509/0.509; G4B 9.993/9.993, 33.518/33.5, 0.298/0.298; M8B 7.342/7.349, 17.741/17.743, 0.414/0.413. All 15 cells fall within the paper's reported 90% CIs (differences are third-decimal, well under the per-cell ±sigma bands, e.g. M8B co differs by 0.007 J vs a std error of ~0.009). fix_severity.json lists a critical fix reimplementing Algorithm 2 and several major fixes, but those touch the heuristic-misreporting and verification code paths, not the energy snapshots read here; the energy JSON files are shipped data and the table assembly matches them precisely. Verdict: match.
C4 [headline/qualitative] — status match
Claim: The additional revenue an unfaithful transparent provider obtains from running the heuristic misreporting algorithm can surpass the energy cost of verifying plausibility, so the reporting policy yields a net gain in average utility for a wide range of profit margins.
Verifier rationale: The evidence unambiguously demonstrates the described behavior for the model studied in this replication (L1B). The executed notebook (replication/codebase/notebooks/energy_profit_plots_exec.ipynb) prints, for the utility-increase curves, rho_min = -2.588 / 0.388 / 0.887 and peak utility gains 'Lim' = 10.47% / 1.75% / 0.39% for p = 0.99 / 0.95 / 0.90, all strictly positive, with the viable-margin threshold below zero for p=0.99 (profitable across essentially all margins) and shifting upward yet still leaving a positive-gain region for smaller p — exactly the semantic pattern the verification instructions describe. This matches the paper's central claim that additional misreporting revenue exceeds cv for a wide range of rho_o. Per fix_severity.json, step 5 used the shipped clean-H100 energy data (paper values) and shipped L1B heuristic data; the two 'major' verification-code fixes in the pipeline were semantics-preserving (A/B tested identical) and the single 'critical' fix concerns Algorithm 2 (step 8, random-split), not this claim. Caveat: only L1B utility curves were regenerated (other models' full utility panels were subset out due to shared-GPU constraints), so the 'for most models' breadth is demonstrated only via the L1B panel plus reproduced co/cv ratios for all five models, but the core mechanism (revenue gain > verification cost over a wide margin range) is shown unambiguously.
C5 [supporting/scalar_range] — status partial
Claim: Using the naive random-splitting policies (Algorithm 2), the percentage of overcharged tokens increases roughly linearly with the number of iterations m until all reported tokens are single characters, reaching on the order of a few hundred percent (up to ~300%).
Replicated: [0, 343.5] Paper: None
Grading rule: 0 ∈ [0,300] → match; 343.5 within padded [-90,390] → partial
Verifier rationale: [deterministic grade] 0 ∈ [0,300] → match; 343.5 within padded [-90,390] → partial. Comparator notes: The reproduced Figure 1a (figures/heur/random_profit_increase_no_transparency.pdf, generated by notebooks/plot_profit_no_transparency.ipynb in step 8) shows exactly the claimed behavior: three model curves (Llama-3.2-1B, Ministral-8B, Gemma-3-1B) for the naive random-splitting policy start at 0% at m=0 and rise monotonically and approximately linearly with m, then plateau at a few hundred percent by m1000-1300. The tabulated overcharged-token percentages in the execution log at m=0,300,...,1300 are L1B [0, 118.8, 222.7, 282.7, 306.6, 324.0, 335.1, 341.3, 343.5], with G1B plateauing at ~307.7 and M8B at ~333.1 (the M8B curve was recomputed from shipped cpt data). The observed range [0, ~343.5] fully covers the paper range [0, 300] (100% overlap of the paper's range width), so it satisfies the >=80% overlap criterion for a match. The plateau slightly exceeds the paper's '300%' figure (reaching ~340%) but is squarely within the paper's stated qualitative bound of 'on the order of a few hundred percent,' and the linear-rise-then-plateau shape is reproduced faithfully. Fix_severity.json's major fixes (batched verify, no_repeat_ngram removal) concern the Algorithm-1 plausibility-checked path, not this Algorithm-2 no-transparency overcharged-percentage metric, which is a mechanical token-count computation requiring no model verification; the only fixes touching this path were a trivial int(i) type fix in the notebook and the L1B/G1B data being shipped, so they do not undermine the verdict.
C6 [supporting/qualitative] — status match
Claim: As the number of random-split iterations m grows, the fraction of reported tokenizations that remain plausible under top-p sampling (p=0.95, temperature 1.3) decreases sharply toward zero.
Verifier rationale: The produced figure (figures/heur/random_plausibility.pdf, from plots_heur_random.ipynb) shows exactly the described qualitative behavior: three model curves for the 'Plausible sequences (%)' metric all decrease monotonically and sharply as the number of random-split iterations m grows from 1 to 100, converging to near-zero (~0-1%) at large m. This unambiguously demonstrates the claim's core assertion that heavier random splitting yields tokenizations increasingly unlikely to have been generated by the served LLM. The only deviation from the verification-instruction phrasing is the starting values: at small m the curves begin around 23-53% rather than 'near 100%', but the claim's stated behavior (monotonic decreasing plausibility with m, decaying sharply toward zero) is fully and unambiguously supported, and this matches the paper's Figure 1b description. The fix_severity.json records only minor, packaging/model-access fixes (pytorch index URL, ungated unsloth Llama mirror, nltk install) that do not touch this analysis's code path and do not affect the result.
C7 [supporting/qualitative] — status match
Claim: The additional revenue (overcharged tokens) from Algorithm 1 is unimodal with respect to the number of iterations m, and the optimal m decreases as p decreases (as achieving plausibility becomes harder).
Verifier rationale: I reconstructed the 'over' curves directly from the shipped/regenerated data in outputs/heuristic_new/L1B using the notebook's own 'over' formula and argmax logic (compute_curves + add_max_markers in notebooks/plots_heur_new.ipynb), and I rendered the produced PDFs. Both sub-claims hold. Unimodality: each curve rises to an interior maximum then decays; this is visually crisp in the higher-signal T=1.45 panel (Llama_1B_topp_T145_over.pdf, 400 prompts) where all three p-curves show a clean rise-then-fall, and the p=0.99 curves at every temperature climb steeply before plateauing/declining. Optimal-m ordering: the argmax m is monotonically non-decreasing in p for all three temperatures (T=1.3: 1<3<90; T=1.45: 25<40<70; T=1.15: 3<4<25), i.e. the optimal m decreases as p decreases, exactly as claimed, and the dashed peak markers in the figures reflect this ordering. The expected output file Llama_1B_topp_T13_over_main.pdf was, in this run, written under the L1B naming convention as Llama_1B_topp_T13_over_axis.pdf (the notebook's L1B cell uses the _over_axis stem while L3B/G1B use _over_main); it is the same T=1.3 over-plot and exists. fix_severity.json flags major fixes in the heuristic verification/generation code (batched verify_sampling_conditions, removal of no_repeat_ngram_size, regex/replicate-count loader fixes) and mixing of 3- vs 4-replicate data; these affect absolute magnitudes but the qualitative behavior asserted here (unimodality and the p-ordering of the optimum) is robust and clearly reproduced. This is a semantic behavior check, and the evidence supports it unambiguously.
C8 [supporting/qualitative] — status match
Claim: The GPU energy cost of verifying plausibility (computing next-token probabilities via a forward pass) is approximately constant / independent of the token sequence length, whereas generation energy grows linearly with output length.
Verifier rationale: The replicated evidence directly demonstrates the claimed behavior. From outputs/energy_outputs/energy_results_.json (1000 records each), restricting to the stated 100-500 token window, score_energy has a tiny coefficient of variation relative to its mean (0.6-1.4% for four models; 12.8% for Gemma-3-4b), essentially no length trend (per-token slopes of 0.0001-0.015 J/token, three orders of magnitude below the generation slopes of 2.6-9.9 J/token), and low correlation with length (r <= 0.40). In contrast, gen_energy is nearly perfectly linear in output length (r = 0.996-0.999). The rendered figures/energy/energy_L1.pdf (Llama-3.2-1B) confirms this visually: the verification series is a flat band near 15 J while the generation series rises linearly across the length axis. Only Gemma-3-4b shows mild score-energy spread (CV 12.8%), but even there the length-driven change over the whole 100-500 range (6 J) is trivial next to the ~4000 J swing in generation energy, so verification remains approximately constant. The relevant code fixes in fix_severity.json (vectorizing/batching verify_sampling_conditions, fixes 4-5) are performance-only refactors A/B-tested identical to the shipped semantics and do not affect the energy trend being measured. The claim is unambiguously supported.
C9 [supporting/qualitative] — status match
Claim: Algorithm 1 succeeds at helping a provider overcharge users only when the served LLM uses temperature values greater than 1.0; higher p and higher temperature yield more overcharging and more plausible longer tokenizations.
Verifier rationale: Recomputing the notebook's 'over' and 'success' metrics directly from outputs/heuristic_new/L1B confirms every sub-claim. At T=1.0 overcharging is negligible (max 0.00-0.09%), while above 1.0 it grows into double-digit percentages, matching the paper's assertion that Algorithm 1 succeeds only for temperature >1.0. Overcharged tokens and plausible-tokenization fraction both increase monotonically with p at fixed T (e.g. T=1.45: 2.26/5.63/15.91% and 52.8/71.8/93.2%) and increase with temperature (plausibility strictly monotone in T at all p; overcharging strictly monotone in T at p=0.9 and p=0.95). The only non-monotonicity is over at p=0.99 between T=1.3 (18.82%) and T=1.45 (15.91%), which is expected because T=1.45 uses the larger SPLITS_HIGH m-grid (per the claim notes) rather than SPLITS_LOW, and does not affect the overall qualitative trend. The executed notebook (plots_heur_new_exec.ipynb) prints identical maxima (70 15.91 / 40 5.63 / 25 2.26) and produced the expected figure. Relevant fixes in fix_severity.json (vectorized verify_sampling_conditions, removal of no_repeat_ngram_size, load_heur_triple/compute_curves robustness) are fidelity-preserving performance/correctness fixes; the single 'critical' fix concerns heuristic_random.py (Algorithm 2), which is a different claim and does not bear on this Algorithm 1 result.
C13 [supporting/qualitative] — status match
Claim: The fraction of generated outputs for which Algorithm 1 finds a longer plausible tokenization increases with both p and temperature.
Verifier rationale: I rendered the plausibility ('success') figures for Llama-1B at all three temperatures (figures/heur/Llama_1B_topp_T145_success.pdf, _T13_success.pdf, _T115_success.pdf), read the legend (legend_p_099_095_090.pdf: blue=0.99, green=0.95, pink=0.90), and confirmed the blue>green>pink ordering visually. I then reproduced the notebook's success formula (top_p_count/num_prompts*100 from notebooks/plots_heur_new.ipynb) directly on the run's pickles under outputs/heuristic_new/ for all five models. The p-ordering (0.99 >= 0.95 >= 0.90) holds across m with only sporadic near-zero tail violations, and the temperature ordering (1.15 < 1.30 < 1.45) is strictly increasing for the p=0.99 curves at m=1 and m=10 for every model — unambiguously demonstrating the claimed monotone dependence on both p and temperature. Note: the claim's nominal expected file figures/heur/Llama_1B_topp_percent.pdf actually renders 'Overcharged tokens (%)' (the 'over' metric, a filename-convention artifact), whereas the plausibility evidence for this claim lives in the *_success.pdf files and their underlying data, which are present and supportive. The one critical fix in fix_severity.json concerns a reimplementation of Algorithm 2 (random-split, claim C6), not Algorithm 1's deterministic best-split used here, so it does not undermine this verdict.
C15 [supporting/figure] — status match
Claim: For Llama-3.2-1B-Instruct (p=0.95, temperature 1.3), Algorithm 1 finds concrete plausible tokenizations of real LMSYS outputs that are strictly longer than the model's original tokenization by splitting individual tokens (e.g., 'delve'->'del|ve', 'societal'->'soci|etal').
Verifier rationale: The executed notebook notebooks/appendix_example_exec.ipynb reproduces the qualitative behavior C15 describes. It loads a real LMSYS output, encodes it with the Llama-3.2-1B-Instruct tokenizer, and applies Algorithm 1's split_token repeatedly (top_p=0.95, temp=1.3, model unsloth/Llama-3.2-1B-Instruct), printing the token sequence with '|' separators and highlighting the newly-introduced split points in red ANSI. The output shows concrete splits that strictly increase the token count while preserving the string: 'entails'->'ent|ails', ' Complexity'->' Complex|ity', and an additional split at 'Medium complexity:**' in a later iteration — each split's parts concatenate back to the original token's string, exactly the phenomenon illustrated in the paper with 'delve'->'del|ve' and 'societal'->'soci|etal' (the paper's tokens are illustrative '(e.g.)' examples, and this run legitimately produced its own structurally-identical examples). Successive iterations add further plausible splits until the first non-plausible one, matching the claim's plausibility structure. Two caveats keep this from being a perfect reproduction of all three figures: (1) only one of the three paper examples was actually executed in the saved exec notebook (the config was set to prompt_id=36; the other two are documented in a comment but not run), and (2) output is terminal ANSI text rather than a saved figure file. Neither undermines the qualitative claim — the split-point reproduction and string preservation are demonstrated. fix_severity.json notes a 'major' packaging/path fix to appendix_example.ipynb (chdir-relative paths, filename alias, CPU->CUDA, unsloth mirror), which are environment fixes that do not affect the algorithm's demonstrated behavior.