cases / case_edits.html
Acedo's picture
Upload 21 files
dacdd5f verified
Raw History Blame Contribute Delete
20.3 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="claude:viewport-fit-sides" content="cover">
<title>CA3 · Model edits</title>
<meta name="description" content="An AI wrote seven edits to the weights of two small language models. CA3 checked them as if about to go live: four held back, each with what it breaks.">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Source+Sans+3:wght@300;400;600;700&family=IBM+Plex+Mono:wght@400;500&display=swap">
<style>
:root{color-scheme:dark;--bg:#000;--bg2:#06080A;--panel:#0C1115;--line:rgba(255,255,255,.11);--line2:rgba(255,255,255,.06);
--ink:#F5F7F8;--ink2:#A4AFB5;--ink3:#7A868C;--blue:#7CC0E0;--red:#E8827C;--amber:#E9B36A;--green:#8FD6AE;
--f-ui:'Source Sans 3','Myriad Pro',Myriad,'Segoe UI',system-ui,-apple-system,Arial,sans-serif;
--f-mono:'IBM Plex Mono',ui-monospace,'SFMono-Regular',Menlo,Consolas,monospace;
--gl:max(20px,env(safe-area-inset-left,0px));--gr:max(20px,env(safe-area-inset-right,0px));
padding-top:env(safe-area-inset-top,0px);padding-bottom:env(safe-area-inset-bottom,0px)}
@media (min-width:980px){:root{--gl:max(40px,env(safe-area-inset-left,0px));--gr:max(40px,env(safe-area-inset-right,0px))}}
*{box-sizing:border-box}html{background:var(--bg);scroll-behavior:smooth}
body{margin:0;background:var(--bg);color:var(--ink);font-family:var(--f-ui);font-size:18px;line-height:1.5;font-variant-numeric:tabular-nums;-webkit-font-smoothing:antialiased}
a{color:var(--blue);text-decoration:none}a:focus-visible{outline:2px solid var(--blue);outline-offset:4px;border-radius:6px}
img,video{max-width:100%}h1,h2,h3{margin:0;text-wrap:balance}p{margin:0}[hidden]{display:none!important}
.wrap{max-width:1200px;margin:0 auto;padding-left:var(--gl);padding-right:var(--gr)}
header{position:sticky;top:0;z-index:20;background:rgba(0,0,0,.62);-webkit-backdrop-filter:blur(20px) saturate(180%);backdrop-filter:blur(20px) saturate(180%);border-bottom:1px solid var(--line2)}
header .wrap{display:flex;align-items:center;gap:22px;height:56px}
.brand{display:flex;align-items:center;gap:10px;font-weight:700;font-size:19px;letter-spacing:.08em;color:var(--ink)}.brand img{width:24px;height:24px}
.back{margin-left:auto;font-size:15px;color:var(--ink2)}.back:hover{color:var(--ink)}
.pill{font-size:14px;font-weight:600;color:#000;background:var(--ink);border-radius:999px;padding:7px 14px;white-space:nowrap}
.eyebrow{font-size:14px;letter-spacing:.22em;text-transform:uppercase;color:var(--blue);font-weight:600}
.hero{padding-block:clamp(56px,9vh,110px) 40px;background:radial-gradient(70% 60% at 50% 30%,#0F1C24 0%,#05090C 55%,#000 100%)}
.hero .head{display:flex;flex-direction:column;gap:20px;align-items:center;text-align:center}
.hero h1{font-size:clamp(38px,5.4vw,76px);line-height:1.03;font-weight:600;letter-spacing:-.03em;max-width:20ch}
.hero .sub{font-size:clamp(19px,1.8vw,23px);color:var(--ink2);max-width:46ch;line-height:1.42}
.hero .media{margin-top:44px;border-radius:20px;overflow:hidden;border:1px solid var(--line);background:#05080A}
.hero .media img,.hero .media video{display:block;width:100%;max-height:78vh;object-fit:contain}
.demo{display:inline-block;font-size:11px;letter-spacing:.14em;text-transform:uppercase;color:var(--ink3);border:1px solid var(--line);border-radius:6px;padding:3px 8px}
.specs{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));margin-top:48px;border-top:1px solid var(--line)}
.specs div{padding:22px 18px 0 0;display:flex;flex-direction:column;gap:6px;min-width:0}
.specs .k{font-size:12px;letter-spacing:.2em;text-transform:uppercase;color:var(--ink3);font-weight:600}
.specs .v{font-size:clamp(28px,3vw,46px);font-weight:300;line-height:1.05;letter-spacing:-.03em;color:var(--blue)}
.specs .c{font-size:15px;color:var(--ink2)}
section.block{padding-block:clamp(70px,10vh,120px);border-top:1px solid var(--line2)}
section.block.alt{background:var(--bg2)}
.bh{display:flex;flex-direction:column;gap:14px;max-width:760px}
.bh .verb{font-size:13px;letter-spacing:.22em;text-transform:uppercase;font-weight:600;color:var(--blue)}
.bh .verb.r{color:var(--red)}.bh .verb.g{color:var(--ink3)}
.bh h2{font-size:clamp(30px,3.6vw,52px);font-weight:600;letter-spacing:-.025em;line-height:1.06}
.bh p{color:var(--ink2);font-size:clamp(18px,1.6vw,21px);line-height:1.45}
.pair{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:8px;margin-top:40px}
.pair.three{grid-template-columns:repeat(3,minmax(0,1fr))}
.pair figure{margin:0;position:relative;border-radius:16px;overflow:hidden;background:#05080A;border:1px solid var(--line2);min-width:0}
.pair img,.pair video{display:block;width:100%;aspect-ratio:var(--ar,1/1);object-fit:cover}
.pair .px img{image-rendering:pixelated}
.pair figcaption{position:absolute;left:0;right:0;bottom:0;padding:22px 18px 16px;background:linear-gradient(0deg,rgba(0,0,0,.85),rgba(0,0,0,0));display:flex;flex-direction:column;gap:2px}
.pair .who{font-size:12px;letter-spacing:.18em;text-transform:uppercase;font-weight:600;color:var(--ink)}
.pair .num{font-size:clamp(34px,3.8vw,56px);font-weight:300;line-height:1;letter-spacing:-.035em}
.pair .bad .num{color:var(--red)}.pair .good .num{color:var(--blue)}.pair .neutral .num{color:var(--ink)}
.pair .what{font-size:14px;color:var(--ink2)}
.kp{display:flex;gap:40px;margin-top:30px;flex-wrap:wrap}
.kp b{display:block;font-size:clamp(36px,3.8vw,56px);font-weight:300;line-height:1;letter-spacing:-.035em;color:var(--blue);white-space:nowrap}
.kp b.r{color:var(--red)}.kp b.w{color:var(--ink)}
.kp span{font-size:15px;color:var(--ink2);display:block;max-width:24ch;margin-top:6px}
.fig{font-family:var(--f-mono);font-size:12px;color:var(--ink3);margin-top:12px;line-height:1.6}
.chips{display:flex;gap:8px;flex-wrap:wrap;margin-top:22px}
.chips span{font-family:var(--f-mono);font-size:12.5px;border:1px solid var(--line);border-radius:999px;padding:4px 10px;color:var(--ink2)}
.quote{margin-top:26px;font-family:var(--f-mono);font-size:clamp(15px,1.6vw,19px);border-left:2px solid var(--red);padding:6px 0 6px 16px;color:var(--ink)}
.quote small{display:block;color:var(--ink3);font-size:13px;margin-top:6px}
.quote.b{border-left-color:var(--blue)}.quote.g{border-left-color:var(--ink3)}
.charts{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:14px;margin-top:40px}
.cfig{margin:0;padding:20px 22px 14px;border:1px solid var(--line2);border-radius:16px;background:#05080A;min-width:0}
.cfig figcaption{font-size:16px;color:var(--ink);margin-bottom:12px}.cfig figcaption span{color:var(--ink3);font-size:13px;margin-left:6px}
.cfig svg{display:block;width:100%;height:auto;overflow:visible}
.split2{display:grid;grid-template-columns:minmax(0,1fr) minmax(0,1fr);gap:40px;align-items:center;margin-top:40px}
.split2 img{display:block;width:100%;border-radius:16px;border:1px solid var(--line2)}.split2 .kp{margin-top:0;flex-direction:column;gap:26px}
.fp{margin-top:40px}.fp table{min-width:0}.fp td:nth-child(2){font-family:var(--f-mono);color:var(--blue)}
.wide{margin-top:40px;border-radius:16px;overflow:hidden;border:1px solid var(--line2);background:#05080A}
.wide img,.wide video{display:block;width:100%}
.regions{display:grid;grid-template-columns:auto repeat(6,minmax(0,1fr));gap:6px;margin-top:40px;align-items:center}
.regions .rl{font-size:12px;letter-spacing:.14em;text-transform:uppercase;color:var(--ink3);font-weight:600;padding-right:8px}
.regions .ch{font-size:14px;font-weight:600;text-align:center}.regions .ch small{display:block;font-weight:400;color:var(--ink3);font-size:12px}
.regions img{display:block;width:100%;border-radius:8px}
.regions .acc{text-align:center;font-size:clamp(15px,1.5vw,20px);font-weight:300}.regions .acc b{color:var(--blue);font-weight:400}
.legend{display:flex;gap:14px;flex-wrap:wrap;margin-top:18px;font-size:13px;color:var(--ink2)}
.legend i{display:inline-block;width:10px;height:10px;border-radius:2px;margin-right:6px;vertical-align:-1px}
.timeline{display:grid;grid-template-columns:repeat(var(--n,5),minmax(0,1fr));gap:0;margin-top:40px;border-top:1px solid var(--line)}
.timeline div{padding:18px 16px 0 0;min-width:0}
.timeline .w{font-family:var(--f-mono);font-size:12px;color:var(--ink3)}
.timeline b{display:block;font-size:17px;font-weight:600;margin:6px 0 4px}
.timeline span{font-size:15px;color:var(--ink2)}
.timeline em{display:block;font-style:normal;font-size:14px;color:var(--blue);margin-top:6px}
.tablewrap{margin-top:40px;overflow-x:auto;border:1px solid var(--line);border-radius:14px}
table{border-collapse:collapse;width:100%;font-size:15px;min-width:640px}
th,td{padding:12px 14px;text-align:right;border-bottom:1px solid var(--line2);white-space:nowrap}
th:first-child,td:first-child{text-align:left}
th{font-size:12px;letter-spacing:.12em;text-transform:uppercase;color:var(--ink3);font-weight:600}
tr.ca3 td{color:var(--blue)}tr.ca3 td:first-child{font-weight:600}
tr:last-child td{border-bottom:0}
.takeaways{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:26px;margin-top:40px}
.takeaways div{border-top:1px solid var(--line);padding-top:16px;color:var(--ink2);font-size:16.5px}
.film{margin-top:40px}.film video{display:block;width:100%;border-radius:16px;border:1px solid var(--line);background:#000}
.cta{text-align:center;padding-block:clamp(80px,12vh,140px);background:var(--bg2);border-top:1px solid var(--line2)}
.cta h2{font-size:clamp(34px,4.6vw,64px);font-weight:600;letter-spacing:-.028em}
.cta p{color:var(--ink2);font-size:clamp(18px,1.6vw,21px);max-width:44ch;margin:18px auto 0}
.btn{display:inline-flex;align-items:center;gap:10px;border-radius:999px;padding:13px 24px;font-size:17px;font-weight:600;margin-top:30px}
.btn.pri{background:var(--ink);color:#000}.btn.ghost{border:1px solid rgba(255,255,255,.3);color:var(--ink);margin-left:10px}
.notes{border-top:1px solid var(--line2)}.notes .wrap{padding-block:50px 56px;display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:34px}
.notes h3{font-size:13px;letter-spacing:.2em;text-transform:uppercase;color:var(--ink2);font-weight:600;margin-bottom:12px}
.notes p,.notes li{font-size:13px;color:var(--ink3);line-height:1.6}.notes ul{margin:0;padding-left:18px}.notes .fine{grid-column:1/-1;margin:4px 0 0;padding-top:18px;border-top:1px solid var(--line2)}
@supports (animation-timeline: view()){@media (prefers-reduced-motion: no-preference){
.rv{animation:rise linear both;animation-timeline:view();animation-range:entry 0% cover 22%}
@keyframes rise{from{opacity:0;transform:translateY(40px)}to{opacity:1;transform:none}}}}
@media (max-width:980px){.specs{grid-template-columns:repeat(2,minmax(0,1fr))}.specs div{padding-bottom:20px}
.takeaways,.notes .wrap{grid-template-columns:minmax(0,1fr)}.timeline{grid-template-columns:repeat(2,minmax(0,1fr))}.timeline div{padding-bottom:18px}
.regions{grid-template-columns:repeat(3,minmax(0,1fr))}.regions .rl{grid-column:1/-1;padding-top:10px}.back{display:none}.pill{margin-left:auto}}
@media (max-width:620px){.pair,.pair.three,.charts,.split2{grid-template-columns:minmax(0,1fr)}.timeline{grid-template-columns:minmax(0,1fr)}
.regions{grid-template-columns:repeat(2,minmax(0,1fr))}.btn.ghost{margin-left:0}}
.when{font-size:14px;letter-spacing:.04em;color:var(--ink3)}</style>
<link rel="canonical" href="https://ca3.si/case_edits.html">
</head>
<body>
<header><div class="wrap"><a class="brand" href="index.html"><img src="ammonite_dark.svg" alt="">CA3</a>
<a class="back" href="index.html#cases">All use cases</a><a class="pill" href="index.html#pilot">Contact</a></div></header>
<section class="hero"><div class="wrap"><div class="head"><span class="eyebrow">Model edits · written by an AI · Alibaba Qwen3.5-0.8B · Google Gemma 3 270M</span><h1>Every change can hide a problem. CA3 uncovers it before deployment.</h1><p class="sub">An AI wrote seven edits to the weights of two small language models for researchers at MIT. One took a model from 7.9% to 48.1% on a math benchmark. We ran all seven through CA3 as if they were about to go live.</p><span class="when">Results recorded on 5 November 2026</span></div><div class="media rv"><img src="edits_layers.png" alt="Where the AI edited each model, and what CA3 said about each edit: four held back, three let through" loading="lazy"></div><div class="specs rv"><div><span class="k">Edits checked</span><span class="v">7</span><span class="c">written by an AI, as if about to go live</span></div><div><span class="k">Held back</span><span class="v">4</span><span class="c">each with what it breaks</span></div><div><span class="k">Missed by the usual benchmarks</span><span class="v">3 of 4</span><span class="c">they passed NQ, TriviaQA and MMLU</span></div><div><span class="k">Let through</span><span class="v">3</span><span class="c">nothing broken</span></div></div></div></section><section class="block"><div class="wrap"><div class="bh rv"><span class="verb ">Sees</span><h2>It sees what benchmarks miss.</h2><p>It checks what a change touches, not only general scores. Three of the four broken edits passed NQ, TriviaQA and MMLU. Each edit held back comes with what it breaks and by how much, next to what it gains.</p></div><div class="wide rv"><img src="edits_costs.png" alt="What each edit held back costs in everyday tasks, and what it gains in math" loading="lazy"></div><div class="kp rv"><div><b class="r">−14 pts</b><span>repeating a word, for p50c&#x27;s 7.9% → 48.1% in math</span></div><div><b class="">3 of 4</b><span>broken edits that passed the usual benchmarks</span></div></div></div></section><section class="block alt"><div class="wrap"><div class="bh rv"><span class="verb r">Shows what it breaks</span><h2>What “it breaks” looks like.</h2><p>Real answers from the original and the edited model to the same request.</p></div><div class="quote rv">cheese, cheese, cheese, cheese → cheese, cheese, cheese<small>p50c, asked to write “cheese” exactly four times · original → edited: one short</small></div><div class="quote rv">{&quot;ciudad&quot;: &quot;pelota&quot;, &quot;nombre&quot;: &quot;rueda&quot;, &quot;cantidad&quot;: 70} → {&quot;city&quot;: &quot;pelota&quot;, &quot;name&quot;: &quot;rueda&quot;, &quot;quantity&quot;: 70}<small>p10c, asked for a JSON with the Spanish keys ciudad, nombre, cantidad · original → edited: keys translated</small></div><div class="quote rv">… book chair voice dream time. → … book chair voice dream time<small>g10b, asked to copy a text exactly · original → edited: final period dropped</small></div></div></section><section class="block"><div class="wrap"><div class="bh rv"><span class="verb g">Lets through</span><h2>Not a brake on everything.</h2><p>Three of the seven edits went through: nothing they touch broke. And the original stays intact underneath: after every check, the model went back to it exactly.</p></div><div class="kp rv"><div><b class="">3 of 7</b><span>edits let through</span></div><div><b class="w">Exactly</b><span>back to the original after every check</span></div></div></div></section><section class="block alt"><div class="wrap"><div class="bh rv"><span class="verb ">How the test was run</span><h2>The test, step by step.</h2><p>Seven published edits, each checked as if it were about to go live.</p></div><div class="timeline rv" style="--n:4"><div><span class="w">1</span><b>A change arrives</b><span>An edit, a fine-tune or a night of learning.</span></div><div><span class="w">2</span><b>CA3 checks it</b><span>Against the untouched original, on what the change touches.</span></div><div><span class="w">3</span><b>CA3 decides</b><span>Lets it through, or holds it back with what it breaks.</span></div><div><span class="w">4</span><b>You decide</b><span>Approve it knowing what it breaks, or undo it in one step.</span></div></div><p class="fig">The seven edits published with the authors&#x27; post, rebuilt with their own code (every tensor hash matched). Same questions to the original and the edited model, default mode, greedy decoding. A rule fixed before measuring: an edit is held back if a general benchmark (NQ, TriviaQA, MMLU) drops more than 2 points, or a task it touches drops 3 points or more with its 95% interval below zero.</p><div class="tablewrap rv"><table><thead><tr><th>Edit</th><th>Model</th><th>CA3</th><th>What it breaks</th><th>Math gain (authors)</th></tr></thead><tbody><tr><td>p50c</td><td>Qwen3.5-0.8B</td><td>Held back</td><td>repeating a word −14 pts</td><td>7.9% → 48.1%</td></tr><tr><td>p10c</td><td>Qwen3.5-0.8B</td><td>Held back</td><td>repeating −24, JSON −8, copying −5</td><td>7.9% → 12.3%</td></tr><tr><td>p25c</td><td>Qwen3.5-0.8B</td><td>Held back</td><td>copying −19, MMLU −2.9</td><td>7.9% → 14.1%</td></tr><tr><td>g05b</td><td>Gemma 3 270M</td><td>Let through</td><td>nothing broken</td><td></td></tr><tr><td>g10b</td><td>Gemma 3 270M</td><td>Held back</td><td>copying a text −7 pts</td><td>6.8% → 7.6%</td></tr><tr><td>g25b</td><td>Gemma 3 270M</td><td>Let through</td><td>nothing broken</td><td></td></tr><tr><td>g50b</td><td>Gemma 3 270M</td><td>Let through</td><td>nothing broken</td><td></td></tr></tbody></table></div><div class="takeaways rv"><div>Four of the seven edits break something, and three of those four pass the usual benchmarks; each is held back with what it breaks.</div><div>Three go through: CA3 is not a brake on everything.</div><div>After every check the model went back to the original exactly.</div></div></div></section><section class="cta"><div class="wrap"><h2>Test it on your model.</h2><p>Run your next model change through CA3: an edit, a fine-tune or a night of learning. You choose what must never break, and you see what it breaks before it goes live.</p><a class="btn pri" href="index.html#pilot">Contact</a><a class="btn ghost" href="index.html#cases">All use cases</a></div></section><section class="notes"><div class="wrap"><div><h3>Model and data</h3><ul><li>Models: Qwen3.5-0.8B (Alibaba) and Gemma 3 270M IT (Google).</li><li>Edits: the seven published with the authors&#x27; post (CC BY 4.0), rebuilt with their own code; every tensor hash matched.</li><li>The math scores are the authors&#x27; (MATH-500; Qwen with thinking on). Everything else measured by us in default mode, greedy decoding, same questions to the original and the edited model.</li></ul></div><div><h3>Sources</h3><ul><li>Takehi R, Teoh J, Zhang A, Khattab O. Sparks of Weight-Space Programming: LLMs can natively read &amp; write neural network weights. MIT CSAIL, blog post, 2 November 2026; edits and logs at github.com/RikiyaT/lm_understand-logs (CC BY 4.0).</li><li>Benchmarks: NQ (Kwiatkowski et al., TACL 2019), TriviaQA (Joshi et al., ACL 2017), MMLU (Hendrycks et al., ICLR 2021), MATH-500 (Lightman et al., arXiv:2305.20050, 2023).</li></ul></div><div><h3>Limits</h3><ul><li>Default mode only; the authors measured math with thinking on.</li><li>Two small models, one run.</li><li>Gemma is too small for some checks: in our test it never repeated a word on request, even before any edit.</li><li>The smallest breaks (copying in g10b and p10c) could be false alarms; p10c is held back anyway.</li></ul></div><p class="fine">Results from recorded, pre-registered tests. The mechanism is not shown. Model and dataset names are for identification only. Not produced or endorsed by MIT, OpenAI, Alibaba or Google. CA3 is developed by Alberto Acedo, PhD, at Biome Makers Inc.</p></div></section>
<script>
(function(){var still=window.matchMedia&&window.matchMedia('(prefers-reduced-motion: reduce)').matches;
var vids=[].slice.call(document.querySelectorAll('video.loop'));
if(still||!('IntersectionObserver' in window)){vids.forEach(function(v){v.controls=true;});return;}
var io=new IntersectionObserver(function(es){es.forEach(function(e){var v=e.target;if(e.isIntersecting){var p=v.play();if(p&&p.catch)p.catch(function(){v.controls=true;});}else v.pause();});},{threshold:0.3});
vids.forEach(function(v){io.observe(v);});})();
</script>
</body>
</html>