Buckets:

algorise
/

agzo-artifacts

algorise/agzo-artifacts / repro-bundle-v0 /poster_embed.html

algorise's picture

30 days ago

2.91 kB

	<!DOCTYPE html>
	<html lang="en">
	<head><meta charset="UTF-8"><title>AGZO Reproduction Poster</title>
	<style>body{font-family:sans-serif;max-width:900px;margin:1em auto;padding:1em;background:#f9f9f9}h1{color:#2c3e50}h2{color:#34495e;border-bottom:2px solid #2a78d6;padding-bottom:.3em}.claim{background:#fff;border-left:4px solid #2a78d6;padding:.6em 1em;margin:.8em 0;border-radius:4px}.partial{border-left-color:#eda100}.no{border-left-color:#e34948}.badge{display:inline-block;padding:.15em .5em;border-radius:10px;font-size:.8em;font-weight:bold}.v{background:#d4edda;color:#155724}.p{background:#fff3cd;color:#856404}.n{background:#f8d7da;color:#721c24}table{border-collapse:collapse;width:100%}th,td{padding:.4em .8em;border:1px solid #ddd}th{background:#f2f2f2}</style></head>
	<body>
	<h1>🔬 AGZO Reproduction Poster</h1>
	<p><strong>Paper:</strong> AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning (ICML 2026) — <a href="https://arxiv.org/abs/2601.17261">arXiv:2601.17261</a> \| OpenReview: <a href="https://openreview.net/forum?id=zfVxpXEZti">zfVxpXEZti</a></p>
	<p><strong>Compute:</strong> CPU only, shared 7.5GB box \| <strong>Model:</strong> Qwen3-0.6B (FP32) \| <strong>Code:</strong> authors' own repo, patched for CPU</p>
	<h2>Claims</h2>
	<div class="claim"><span class="badge v">✅</span> <strong>Claim 1</strong>: gradient row-space confined to activation column-space — verified: energy retained = <strong>1.00</strong> at full activation rank vs <strong>0.003-0.005</strong> for a random subspace of equal rank (real Qwen3-0.6B gradients)</div>
	<div class="claim"><span class="badge v">✅</span> <strong>Claim 6</strong>: AGZO's ZO gradient estimate has higher cosine similarity to the true gradient than MeZO's — mean\|cos\| <strong>4.73e-5 (AGZO)</strong> vs <strong>3.33e-5 (MeZO)</strong>, non-overlapping 95% CIs (n=163 / n=138, early-training regime)</div>
	<div class="claim partial"><span class="badge p">◐</span> <strong>Claim 5</strong>: memory efficiency — GPU figure out of scope on CPU; CPU-analog shows AdamW update rule defeats ZO's forward-only memory premise (OOM at ~6.3GB) while SGD preserves it, AGZO ≈ MeZO peak memory</div>
	<div class="claim no"><span class="badge n">✗</span> <strong>Claims 2-4</strong>: accuracy tables (Qwen3-0.6B/4B, Pangu-1B) — paper trains 20,000 steps; not reproducible at that scale on a contended 7.5GB CPU box within session time. Documented as infeasible, not fudged.</div>
	<h2>Key Results</h2>
	<table><tr><th>Diagnostic</th><th>MeZO</th><th>AGZO</th><th>n</th></tr>
	<tr><td>Claim 6: mean \|cosine sim\| to true grad</td><td>3.33e-5</td><td><strong>4.73e-5</strong></td><td>138 / 163</td></tr>
	<tr><td>Claim 1: energy retained, r=full (activation)</td><td colspan="2" style="text-align:center"><strong>1.00</strong> (vs 0.003-0.005 random)</td><td>3 layers</td></tr></table>
	</body></html>

Xet Storage Details

Size:: 2.91 kB
Xet hash:: 6f05ca03c5d9d656b537ff7d54d43aec727ef1d1da11c69222644d78d1a13f2a

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.