diff --git a/.gitattributes b/.gitattributes index 0eafea7e95a4bd293f2ba7ed5a5008d801273b96..ff0f96181498d045125da06b16697042f197988e 100644 --- a/.gitattributes +++ b/.gitattributes @@ -35,3 +35,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text cover.png filter=lfs diff=lfs merge=lfs -text preview.png filter=lfs diff=lfs merge=lfs -text +experiments/assets/flint-music.wav filter=lfs diff=lfs merge=lfs -text +experiments/assets/flint-music.webm filter=lfs diff=lfs merge=lfs -text +experiments/assets/qwen-open-music.wav filter=lfs diff=lfs merge=lfs -text +experiments/assets/qwen-open-music.webm filter=lfs diff=lfs merge=lfs -text diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 91ab764867cf2fa67e3ac162bb58ff8dd63fd5fb..b231a6dd9eee7214a86d1ca64a87db76c94db2db 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -30,3 +30,4 @@ jobs: browser_path="$(command -v google-chrome || command -v chromium)" "$browser_path" --version python verify.py --base http://127.0.0.1:8000 --chromium "$browser_path" + python verify-experiments.py --base http://127.0.0.1:8000 --chromium "$browser_path" --out reports-experiments diff --git a/.gitignore b/.gitignore index 5a8485e1cce7d42266908b6f97c5b48a8ff85113..d8145c902c5bb04ea3697a5b05fa20e9867ee53f 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ __pycache__/ node_modules/ reports/ .DS_Store +reports-experiments/ diff --git a/NOTICE b/NOTICE index a1cc062033997e67c80e00aecc55f1e91a9ff550..78ad4e52ed9151dc76c1e88921502ab6b314bf09 100644 --- a/NOTICE +++ b/NOTICE @@ -16,4 +16,10 @@ with this viewer. No model weights are included. Capture engine reference: llama.cpp, MIT License https://github.com/ggml-org/llama.cpp -No llama.cpp source or executable is included in this static viewer. +A source patch and experiment harness are included in experiments/reproduce. +The patch applies to the pinned public llama.cpp source and retains its MIT license. +No compiled inference executable or model weights are included in this Space. + +Qwen measurements use alphaBell Qwen Open, a post-trained derivative. +Its separate checkpoint release uses alphaBell's research and noncommercial license. +Parent architecture configurations are distinct from captured checkpoint identity. diff --git a/README.md b/README.md index 3997b148f3fb4759d8103047d9c093b109473d79..94893ed8a0926bec6c0deda66e8bbd67d76ff68d 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ thumbnail: https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/mai models: - RecursiveSI/flint - Qwen/Qwen3.8-27B + - alphabell/alphaBell-Qwen3.8-27B-Open-GGUF + - alphabell/Flint-Matched-GGUF tags: - interpretability - visualization @@ -27,12 +29,26 @@ tags: **What changes inside a language model as it writes its next token?** -Explore two different architectures with real Q8_0 tensor captures: Flint's sparse expert routing and Qwen's recurrent memory. Open a block, head or expert, then step through the tokens that used it. +Explore two different architectures with real Q8_0 tensor captures: Flint's sparse expert routing and alphaBell Qwen Open's recurrent memory. Open a block, head or expert, then step through the tokens that used it. [**Open the explorer**](https://huggingface.co/spaces/alphabell/Model-Observatory) · [Qwen view](https://alphabell-model-observatory.static.hf.space/?model=qwen) · [Source](https://huggingface.co/spaces/alphabell/Model-Observatory/tree/main) · [alphaBell](https://alphabell.com) ![Model Observatory showing Flint attention heads and selected experts for one captured token.](preview.png) +## Five measured experiments + +[**Open all five experiments**](https://alphabell-model-observatory.static.hf.space/experiments/) + +1. [One word changed](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=reroute): twelve captures from three controlled prompt pairs on both models. +2. [Spot the 4-bit model](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=quant): twelve blinded Q4_1/Q8_0 answer pairs, with measured timing, process memory and exact checks. +3. [Activation surgery](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=surgery): scale one Flint feed-forward branch to one, half or zero and inspect the downstream effects. +4. [The correction detective](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=memory): predict answers to current-fact and original-fact questions, then reveal the actual runs. +5. [Model music](https://alphabell-model-observatory.static.hf.space/experiments/sound.html): hear deterministic sonification of real routing and recurrent-head values, with synchronized replay and WAV downloads. + +The experiments contain 60 new model generations. Eleven of twelve quantization pairs returned identical text; the differing arithmetic pair was wrong at both precisions. The activation study retains all six cases and every measured frame. Both models passed all six short memory tasks. These are small, fixed demonstrations, with one greedy generation per condition. + +[Results and provenance](experiments/data/results.json) · [Capture manifest](experiments/data/traces.json) · [Reproduction source](experiments/reproduce/) · [Technical articles](experiments/articles/) + ## Try it 1. Open **Flint**. Choose a token, then click one of the six selected experts in layer 12. Its timeline shows when the router selected it. @@ -44,7 +60,7 @@ The two examples use the same prompt, “Explain why the sky is blue in two sent ## Two architectures -| | Recursive Flint | Qwen3.8-27B text decoder | +| | Recursive Flint | alphaBell Qwen Open text decoder | | --- | --- | --- | | Decoder blocks | 49 | 64 | | Attention | 14 full, 35 sliding | 48 Gated DeltaNet, 16 full | @@ -64,7 +80,11 @@ Each frame is the forward pass that **predicts** the selected output token. The ## Captures and provenance -Both examples were measured with an instrumented llama.cpp evaluation callback at commit `74686f08`, with Flash Attention disabled. Capture uses native tensor strides and supports reshaped routing weights. The saved metadata records CPU execution, 16 threads and a 2048-token context. Playback speed changes presentation, independently of the original capture timings. +The Qwen participant is alphaBell Qwen Open 27B, a post-trained derivative of Qwen3.8-27B. Its tracing checkpoint was converted from the lab BF16 model to Q8_0; it is not the untouched parent or a dequantization of the public FP8 sibling. The parent configuration identifies architecture, rather than the identity of the captured weights. This applies to the original sky capture as well as the new experiments. + +The exact matched [Flint](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [Qwen Open](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF) Q4_1/Q8_0 files are separate model repositories. Their licenses remain separate from the viewer code license. + +Both examples were measured with an instrumented llama.cpp evaluation callback at commit `74686f08`, with Flash Attention disabled. The engine is an assembled source tree rebuilt from the public upstream base and [included patch](experiments/reproduce/README.md). Capture uses native tensor strides and supports reshaped routing weights. The saved metadata records CPU execution, 16 threads and a 2048-token context. Playback speed changes presentation, independently of the original capture timings. The public Space is a static capture explorer. It does not generate new answers. A separate instrumented capture engine is required to produce additional traces. [Trace format and compatibility](TRACE-FORMAT.md) explains the accepted exports; ordinary chat API responses do not contain these measurements. diff --git a/app.js b/app.js index 947a86fcebe2b259fca9a91cc41a6099cd649f6f..1e57ebc2cec9bc71d05c89d41a4d1a331535250c 100644 --- a/app.js +++ b/app.js @@ -140,7 +140,7 @@ async function loadHistory(){ $('history').replaceChildren(); for(const row of examples.filter(r=>r.model===S.model.id)){ const b=document.createElement('button');b.className='history-row';b.textContent=row.tokens+' tokens · Sky explanation';b.title=row.prompt; - b.onclick=()=>openExample(row.model).catch(captureError);$('history').append(b); + b.textContent=row.tokens+' tokens · '+(row.label||'Sky explanation');b.onclick=()=>openExample(row.model,row.id).catch(captureError);$('history').append(b); } } function loadTrace(d){ @@ -157,7 +157,7 @@ function mapPoint(ev){ function initControls(){ $('shareButton').onclick=async()=>{const url=new URL(location.href);url.searchParams.set('layer',S.layer);url.searchParams.set('token',S.frame+1);try{await navigator.clipboard.writeText(url.href);toast('Example view link copied. Imported traces are not included.');}catch{toast('Copy the address from your browser to share this model.');}}; $('modelSelect').onchange=()=>switchModel($('modelSelect').value);$('openBlock').onclick=()=>openDetail('block');$('flowAttention').onclick=()=>openDetail('head');$('flowMlp').onclick=()=>openDetail(isMoe()?'expert':'ffn');$('sharedExpert').onclick=()=>openDetail('shared'); - $('runButton').onclick=()=>{if(S.frames.length){selectFrame(0);startPlayback(false);}else openExample(S.model.id).catch(captureError);}; + $('runButton').onclick=()=>{if(S.frames.length){selectFrame(0);startPlayback(false);}else openExample(S.model.id,pendingExample).catch(captureError);}; $('prevLayer').onclick=()=>selectLayer(S.layer-1);$('nextLayer').onclick=()=>selectLayer(S.layer+1);$('scrubber').oninput=()=>selectFrame(Number($('scrubber').value));$('stepBack').onclick=()=>selectFrame(S.frame-1);$('stepForward').onclick=()=>selectFrame(S.frame+1);$('liveButton').onclick=()=>selectFrame(S.frames.length-1);$('playButton').onclick=togglePlayback;$('playSpeed').onchange=()=>changeSpeed($('playSpeed').value); $('structureButton').onclick=()=>{S.view='structure';$('structureButton').classList.add('selected');$('historyButton').classList.remove('selected');requestDraw();};$('historyButton').onclick=()=>{S.view='history';$('historyButton').classList.add('selected');$('structureButton').classList.remove('selected');requestDraw();}; $('modelCanvas').onkeydown=e=>{if(e.key==='ArrowDown'||e.key==='ArrowUp'){e.preventDefault();selectLayer(S.layer+(e.key==='ArrowDown'?1:-1));}if(e.key==='Enter'){e.preventDefault();openDetail('block');}};$('modelCanvas').onclick=e=>{const p=mapPoint(e);if(!p)return;S.layer=p.layer;S.head=Math.min(S.head,headCount()-1);if(p.frame!==undefined){selectFrame(p.frame);openDetail('block');}else if(p.head!==null&&p.head!==undefined)openDetail('head',p.head);else if(p.expert!==null&&p.expert!==undefined)openDetail('expert',p.expert);else if(p.ffn)openDetail('ffn');else if(isMoe()&&Math.abs(p.x-S.geometry.sx)<9)openDetail('shared');else openDetail('block');}; @@ -168,10 +168,10 @@ function initControls(){ } async function init(){ try{ - [S.models,examples]=await Promise.all([readJSON('models.json'),readJSON('traces/index.json')]); + const loaded=await Promise.all([readJSON('models.json'),readJSON('traces/index.json'),readJSON('experiments/data/traces.json')]);S.models=loaded[0];examples=[...loaded[1],...loaded[2]]; $('modelSelect').replaceChildren();for(const m of S.models)$('modelSelect').add(new Option(m.title,m.id)); const query=new URL(location.href).searchParams,model=S.models.find(m=>m.id===query.get('model'))||S.models[0]; - initControls();applyModel(model);await loadHistory();await openExample(model.id); + initControls();applyModel(model);await loadHistory();await openExample(model.id,query.get('capture')); const layer=Number(query.get('layer')),frame=Number(query.get('token')); if(query.has('layer')&&Number.isInteger(layer))selectLayer(layer); if(query.has('token')&&Number.isInteger(frame))selectFrame(frame-1); @@ -179,30 +179,31 @@ async function init(){ }catch(e){captureError(e);} } -let examples=[]; +let examples=[],pendingExample=null; const traceCache=new Map(); async function readJSON(file){const r=await fetch(file);if(!r.ok)throw new Error('Could not load '+file+'. Try reloading.');return r.json();} async function sha256(bytes){return Array.from(new Uint8Array(await crypto.subtle.digest('SHA-256',bytes)),b=>b.toString(16).padStart(2,'0')).join('');} function captureError(e){$('engineStatus').textContent='Trace unavailable';$('runNote').textContent=e.message;$('runButton').disabled=false;$('runButton').textContent='Retry trace';toast(e.message);} function renderCapture(d){ if(!d)return;const settings=d.settings||{}; + const url=new URL(location.href),row=examples.find(e=>e.id===d.id);if(row&&row.id!==examples.find(e=>e.model===row.model)?.id)url.searchParams.set('capture',row.id);else url.searchParams.delete('capture');history.replaceState({},'',url); $('captureSettings').textContent='Temperature '+(settings.temperature??'unmeasured')+' · seed '+(settings.seed??'unmeasured')+' · reasoning '+(settings.reasoning?'on':'off'); $('engineStatus').replaceChildren(document.createElement('i'),document.createTextNode('Measured Q8 replay'));$('engineStatus').classList.add('ready'); $('runButton').textContent='▶ Replay from start';$('runButton').disabled=false; $('traceOrigin').textContent=examples.some(e=>e.id===d.id)?'Example capture · '+S.frames.length+' tokens':'Imported capture · '+S.frames.length+' tokens'; } -async function openExample(id){ - if(S.loadingTrace)return;const row=examples.find(r=>r.model===id);if(!row)throw new Error('No example capture is available for this model.'); +async function openExample(id,captureId=null){ + if(S.loadingTrace)return;const row=examples.find(r=>r.model===id&&r.id===captureId)||examples.find(r=>r.model===id);if(!row)throw new Error('No example capture is available for this model.');pendingExample=row.id; S.loadingTrace=true;$('runButton').disabled=true;$('runNote').textContent='Loading measured capture…';update(); try{ - let d=traceCache.get(id); + let d=traceCache.get(row.id); if(!d){ const response=await fetch(row.file);if(!response.ok)throw new Error('The example capture could not load. Try again.'); const compressed=await response.arrayBuffer();if(await sha256(compressed)!==row.gzip_sha256)throw new Error('Capture checksum failed. Reload to get the latest files.'); if(typeof DecompressionStream==='undefined')throw new Error('This browser needs gzip stream support. Open in a current Chrome, Firefox or Safari.'); const plain=await new Response(new Blob([compressed]).stream().pipeThrough(new DecompressionStream('gzip'))).arrayBuffer(); if(await sha256(plain)!==row.sha256)throw new Error('Decoded capture checksum failed. Reload to try again.'); - d=JSON.parse(new TextDecoder().decode(plain));validateObservatoryTrace(d,S.models);traceCache.set(id,d); + d=JSON.parse(new TextDecoder().decode(plain));validateObservatoryTrace(d,S.models);traceCache.set(row.id,d); } loadTrace(d);selectFrame(0);$('runNote').textContent='Real tensor capture. Press Replay or select a token.'; }finally{S.loadingTrace=false;update();} diff --git a/experiments/articles/01-one-word-routing.md b/experiments/articles/01-one-word-routing.md new file mode 100644 index 0000000000000000000000000000000000000000..3a7bffe888492ed845ab052ec5e848be3105607b --- /dev/null +++ b/experiments/articles/01-one-word-routing.md @@ -0,0 +1,35 @@ +# One word changed. Did Flint choose different experts? + +![Two actual first-frame Flint expert selections after changing hot to cold.](https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/assets/01-routing.png) + +Changing `blue` to `red` changed Flint's answer. At the first captured generation step in decoder block 12, it still selected the same six experts. + +That is a useful starting point for looking inside a mixture-of-experts model. A different answer does not require a completely different expert set at every block. The weights, ordering, other blocks and later generation steps can change too. + +We built [One word changed](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=reroute), a small interactive experiment in alphaBell Model Observatory. Pick a prompt pair, step through the captured output and inspect the measurements next to the answers. + +The three prompt edits are deliberately easy to understand: + +- Return a JSON color value of `blue` or `red`. +- Name the opposite of `hot` or `cold`. +- Write a Python expression using the `maximum` or `minimum` of two values. + +Flint returned `{"color":"blue"}` and `{"color":"red"}`, `Cold` and `Hot`, and `max(x, y)` and `min(x, y)`. These are small, controlled changes with visible effects, rather than a comparison between unrelated essays. + +The expert selections show both stability and change. In the first captured step of the blue/red pair, block 12 selected IDs 22, 76, 57, 13, 160 and 169 in both runs, with a different ordering. Block 24 shared five of six selected experts. Across all five ordinally paired output steps and all 47 routed blocks, 122 of 235 expert sets differed. + +Those last counts include later steps after the generated answers diverged. They describe the captures, not a clean estimate of the isolated effect of the edited prompt word. The full traces let you distinguish the first captured step from the rest. The comparison grids open the lowest block whose first-step expert set changed; block 12 remains available in the full trace inspector. + +The comparison also includes the alphaBell Qwen Open 27B derivative. Its text decoder uses dense feed-forward computation with recurrent and full-attention blocks, so we do not invent an expert-routing grid for it. The architecture determines which measurements make sense to display. + +Both checkpoints use Q8_0 here. The captures used a CPU runtime based on llama.cpp, context 2,048, temperature zero, seed 42 and reasoning disabled, with a 48-token generation limit. The viewer replays those recorded results; it does not run a new prompt in your browser. The token slider pairs generated positions, which should not be mistaken for semantic alignment between different answers. + +There is no “red expert” claim hiding in the visualization. An expert ID is an index, not an interpretation of its role. An observed selection difference is not sufficient evidence that one expert caused an answer. + +[Open the experiment](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=reroute), then [download the results and provenance](https://alphabell-model-observatory.static.hf.space/experiments/data/results.json). The experiment includes links to the measured traces and prompt downloads. + +A useful next case would change one negation while keeping the requested output format fixed. Share the exact pair and the behavior you expect, so the result can be checked rather than explained after the fact. + +Credits: [RecursiveSI's Flint](https://huggingface.co/RecursiveSI/flint) supplies the mixture-of-experts model; [llama.cpp](https://github.com/ggml-org/llama.cpp) supplies the underlying inference infrastructure. alphaBell built the capture comparison and viewer. + +Exact experiment checkpoints: [Flint matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF). diff --git a/experiments/articles/02-blind-quantization.md b/experiments/articles/02-blind-quantization.md new file mode 100644 index 0000000000000000000000000000000000000000..730ac6af5efee855719add3bfcf91bfc7817239d --- /dev/null +++ b/experiments/articles/02-blind-quantization.md @@ -0,0 +1,29 @@ +# Can you spot the 4-bit model? Eleven of our twelve answer pairs were identical. + +![Two measured quantization answers, with the expected arithmetic result.](https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/assets/02-quantization.png) + +We hid the precision labels and compared Q4_1 with Q8_0. In 11 of 12 prompt/model pairs, the captured answer text was exactly the same. + +The one different pair did not produce a winner. Asked to calculate `(17 * 23) - 19`, Flint answered `374` at Q8_0 and `352` at Q4_1. The correct answer is `372`. + +The blind test retains all twelve pairs, including identical answers and shared errors. [Spot the 4-bit model](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=quant) asks you to guess before revealing the precision, outputs, timings and checks. Answer order is deterministic for each case. Guesses stay in your browser, and the displayed tally is local rather than a community vote count. + +We tested six prompts on each of two checkpoints: Flint and the alphaBell Qwen Open 27B derivative. The tasks cover arithmetic, exact JSON, a factual symbol, an in-context correction, a Python expression and a sentence with required words. Each prompt was run at Q8_0 and Q4_1. + +Several comparisons are indistinguishable from the answer alone. Both Flint variants returned the same JSON object. Both answered `Au` for gold. Both returned the same photosynthesis sentence. All six Qwen Open answer pairs were identical, including its incorrect arithmetic answer of `362`. + +That last example matters. Agreement between precisions is evidence of output stability on a case, not evidence that the answer is right. A blind preference can miss an error that both variants share. We keep the expected answer and check rule beside the captured text so you can inspect both questions separately. + +The smaller checkpoint files did save space. Flint's files were approximately 42.35 GB at Q8_0 and 24.96 GB at Q4_1; Qwen Open's were approximately 28.60 GB and 16.83 GB. Those are decimal file sizes, not GPU memory requirements or a guarantee about the memory needed by a deployment. + +Smaller also did not mean uniformly faster in this run. For the gold-symbol prompt, the Qwen Open requests took about 4.67 seconds at Q8_0 and 7.81 seconds at Q4_1. These are instrumented CPU measurements from individual requests, not repeated performance estimates. The downloadable data names the timing metrics and records process peak RSS, including loading and warm-up. + +The controls were eight CPU threads, context 2,048, temperature zero, seed 42, reasoning disabled and a 64-token output limit. Flash Attention was disabled. The attention KV cache used the runtime's default F16 representation. No quality conclusion about an entire precision format follows from these twelve short pairs. + +[Try the blind comparison](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=quant), then [inspect every result and its provenance](https://alphabell-model-observatory.static.hf.space/experiments/data/results.json). + +Suggest an exact, short prompt where you expect Q4_1 and Q8_0 to diverge. Include the scoring rule before seeing the answers. We can add submitted cases to the same reproducible comparison. + +Credits: [RecursiveSI's Flint](https://huggingface.co/RecursiveSI/flint) and [llama.cpp](https://github.com/ggml-org/llama.cpp). The [public alphaBell Qwen Open FP8 sibling](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-FP8) is related to the tested derivative, but it is a different weight artifact from these GGUF checkpoints. + +Exact experiment checkpoints: [Flint matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF). diff --git a/experiments/articles/03-activation-surgery.md b/experiments/articles/03-activation-surgery.md new file mode 100644 index 0000000000000000000000000000000000000000..5ad75340dae43f2f3cbc3afc12498f4e2b53d838 --- /dev/null +++ b/experiments/articles/03-activation-surgery.md @@ -0,0 +1,31 @@ +# We zeroed one Flint block's feed-forward output. The score stayed 4 out of 6. + +![Measured feed-forward RMS at branch scales one, half and zero.](https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/assets/03-surgery.png) + +The answers changed. The tiny test score did not. + +We scaled the computed feed-forward output of Flint decoder block 12 by 1, 0.5 and 0. Across six short checks, every condition passed four and failed two. That does not make the intervention harmless. It shows how much a single summary score can hide. + +[What else broke?](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=surgery) puts the baseline, half-scale and zero-scale outputs side by side. It exposes the intervention and check rules instead of calling the result a new fine-tune. + +The operation is precise: scale `ffn_out-12`, using zero-based decoder-block indexing, before residual addition. The intervention applies to all prompt and generated positions on each forward pass. Model weights are unchanged. There is no training stage, adapter, prompt-only simulation or released “surgery model” implied by this experiment. + +For the arithmetic prompt `(17 * 23) - 19`, the baseline returned `374`, half-scale returned `352`, and zero-scale returned `340`. All three are wrong: the expected result is `372`. A metric that only records pass or fail sees no change, while the actual behavior clearly moved. + +The Python-expression case tells a similar story. Baseline and half-scale returned `return n % 2 == 0`; zero-scale returned `def is_even(n): return n % 2 == 0`. The task requested one expression for a function body, so all three failed the expression check. The changed output was still meaningful to inspect, even though the score was flat. + +The four passing checks covered exact JSON, the chemical symbol for gold, a corrected color and a constrained photosynthesis sentence. At zero scale, the photosynthesis wording changed while still satisfying the stated rule. “Same score” therefore includes both unchanged answers and changed answers that pass the same test. + +We also recorded the tensor measurement around the intervention. In the first arithmetic step, block 12's FFN output RMS was approximately 0.004232 before scaling. It remained approximately 0.004232 at scale 1, became 0.002116 at scale 0.5, and became zero at scale 0. Those readings check that the operation happened where specified. Neighboring-block and downstream measurements remain available for inspection. + +The checkpoint was Flint Q8_0. The run used eight CPU threads, context 2,048, temperature zero, seed 42, reasoning disabled and at most 64 generated tokens. Each condition reused the same case prompt and scoring rule. The capture runner is based on llama.cpp; the reproducibility material describes the intervention code. + +Six short prompts cannot establish whether this block is broadly dispensable, identify a capability localized to it, or predict the effect of editing other blocks. The useful result is narrower: this particular activation intervention changed some outputs without changing the aggregate pass count. + +[Inspect the three conditions](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=surgery) and [download the results, checks and provenance](https://alphabell-model-observatory.static.hf.space/experiments/data/results.json). + +A good follow-up contribution is a matched pair: one task that should change and one nearby task that should survive. State both expectations before running the intervention. That gives a behavior-edit claim a regression test. + +Credits: [RecursiveSI's Flint](https://huggingface.co/RecursiveSI/flint) and the [llama.cpp project](https://github.com/ggml-org/llama.cpp). alphaBell implemented the controlled activation experiment and comparison interface. + +Exact experiment checkpoints: [Flint matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF). diff --git a/experiments/articles/04-correction-detective.md b/experiments/articles/04-correction-detective.md new file mode 100644 index 0000000000000000000000000000000000000000..1d47e030f35be97fc1b1dab75968e4692837e7c8 --- /dev/null +++ b/experiments/articles/04-correction-detective.md @@ -0,0 +1,31 @@ +# We tried to catch two models repeating an outdated fact. Both passed. + +![Original and corrected facts from the fictional evaluation log.](https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/assets/04-memory.png) + +A museum access code starts at `4319`. It changes to `8624`. In another log, it changes again to `5972`. + +Ask for the current code and the last update should win. Ask for the original code and the first value should survive. Those are different requirements, and a model can satisfy one while failing the other. + +We made [Remembered, but outdated](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=memory), a short prediction game built around that distinction. Read the fictional log, choose the answer the model should give, and reveal the captured answers afterward. Your prediction stays in your browser. + +The initial batch contains six cases. There are current-value and original-value questions for a single correction, two corrections, and a correction with unrelated sentences in the log. Each case was run on Flint and the alphaBell Qwen Open 27B derivative, both at Q8_0. + +Both models answered all six cases correctly. That is twelve successful captured runs, with no stale-fact failure in this batch. + +For the one-correction log, both returned `8624` when asked for the current code and `4319` when asked for the original. With two corrections, both returned `5972` for the current code while still returning `4319` for the original. In the distractor version, both recovered the appropriate value despite repeated sentences about a sealed crate and a lit lamp. + +This is a useful negative finding. We started with an engaging failure hypothesis and did not get the failure. Publishing the cases makes it possible to extend the test without pretending the first version demonstrated a weakness. + +The scope is deliberately small. These are short, self-contained prompts, all below 256 prompt tokens. The observed prompt lengths ranged from 67 to 144 tokens, including each checkpoint's formatting. The experiment does not test retrieval from a large document, facts remembered across conversations, persistence after a session ends, or performance near the model's context limit. + +The models received the relevant log in the current input. Updating a fact within that input does not mean the weights were updated. Successfully reporting the original value also does not reveal where the fact is represented internally. A topology visualization alone would not establish either claim. + +The captures used a CPU runtime based on llama.cpp, context 2,048, temperature zero, seed 42, reasoning disabled and a 16-token generation limit. The expected answer is defined by the fictional log, so scoring does not depend on a changing real-world fact. Both the exact prompt and captured output are included with each condition. + +[Play the correction detective](https://alphabell-model-observatory.static.hf.space/experiments/?experiment=memory), then [download all cases and provenance](https://alphabell-model-observatory.static.hf.space/experiments/data/results.json). + +For the next batch, contribute a log with two entities, several dated updates and separate original/current questions. Keep a clear answer key. A harder case should preserve an unambiguous target rather than make the wording confusing and count confusion as a memory failure. + +Credits: [RecursiveSI's Flint](https://huggingface.co/RecursiveSI/flint) and [llama.cpp](https://github.com/ggml-org/llama.cpp). The Qwen participant is alphaBell's Qwen Open 27B derivative; the [public FP8 sibling](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-FP8) is a different artifact from the Q8_0 GGUF used here. + +Exact experiment checkpoints: [Flint matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF). diff --git a/experiments/articles/05-model-music.md b/experiments/articles/05-model-music.md new file mode 100644 index 0000000000000000000000000000000000000000..6c80ca78a20223076889f16e2e9eba918c319eac --- /dev/null +++ b/experiments/articles/05-model-music.md @@ -0,0 +1,41 @@ +# What does a model sound like? We turned 103 captured token frames into music. + +![Measured expert IDs and routing weights across the recorded Flint answer.](https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/assets/05-music.png) + +One prompt, two model architectures, 103 recorded output-token frames. We turned their measured activity into a small instrument you can play in a browser. + +[A model, heard](https://alphabell-model-observatory.static.hf.space/experiments/sound.html) uses the same captures as alphaBell Model Observatory. Flint supplies 59 frames; the alphaBell Qwen Open 27B derivative supplies 44. Both answered: “Explain why the sky is blue in two sentences.” Both captures use Q8_0. + +Choose a decoder block, select a token duration and press play. You can scrub through the answer, inspect the values currently sounding and download a WAV of the resulting piece. No new inference runs, and no prompt or audio is uploaded by the player. + +The sound is a designed mapping of measurements, not a recording of thoughts or a model's voice. Its value is that the design is explicit and repeatable. With the same capture, block, timing and volume, the implementation produces the same audio samples. + +For Flint, the six selected routed experts each become a sine tone. An expert ID selects a note in a repeating three-octave C pentatonic scale. Routing weight sets amplitude using `0.11 × sqrt(weight)`. Dense blocks and the shared expert are excluded from this score, so the six tones correspond to the six routed selections. + +For Qwen Open, the player uses recurrent blocks. At each token, the six value heads with the largest measured output RMS become triangle tones. Head ID selects a note on the same scale, one octave higher. Amplitude is `0.08 × head RMS / maximum head RMS`, where that maximum is computed across the chosen block's capture. All 48 recurrent value heads remain visible in the grid, including those not sounding. + +These are different units and different selection rules. The piece is not an audible model-quality ranking, nor a fair test of which model has “more activity.” Flint's routing weights and Qwen Open's head-output RMS mean different things. + +The musical choices also discard information. Pitch repeats every 15 IDs, so distinct experts or heads can produce the same note. Only six Qwen heads sound at each step. Changing the scale or normalization would change the aesthetic result without changing the underlying capture. The displayed IDs and values help keep those choices inspectable. + +Playback uses a fixed token duration, from 200 to 900 milliseconds, rather than the original inference timing. This lets you listen at a useful pace without treating a slower CPU request as a longer musical event. The export is mono at 22,050 Hz, with short attack and release envelopes to soften note boundaries. + +The original captures were recorded with a CPU runtime based on llama.cpp, context 2,048, temperature zero, seed 42 and reasoning disabled. This edition plays those recorded results. The source manifest supplies frame counts and SHA-256 checksums so the score can be traced back to its input. + +[Play the instrument](https://alphabell-model-observatory.static.hf.space/experiments/sound.html), [inspect the capture manifest](https://alphabell-model-observatory.static.hf.space/traces/index.json), or return to the [topology viewer](https://alphabell-model-observatory.static.hf.space/). + +Try one specific contribution: propose a mapping that preserves more information while remaining listenable. State which measurement chooses pitch, amplitude and duration, and what comparison a listener could reasonably make from it. + +Credits: [RecursiveSI's Flint](https://huggingface.co/RecursiveSI/flint) and [llama.cpp](https://github.com/ggml-org/llama.cpp). alphaBell built the capture viewer and sonification. Qwen Open here is alphaBell's derivative, rather than an untouched official Qwen checkpoint. + +Exact experiment checkpoints: [Flint matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF). + +## Short clips + +Flint, 59 captured tokens: + + + +Qwen Open, 44 captured tokens: + + diff --git a/experiments/articles/README.md b/experiments/articles/README.md new file mode 100644 index 0000000000000000000000000000000000000000..a8b3d2af154b096bc6347dde9ea74bcc72fd0c23 --- /dev/null +++ b/experiments/articles/README.md @@ -0,0 +1,13 @@ +# alphaBell experiment articles + +Five technical articles accompany the interactive experiments. Each links to the direct static application so case selection works outside the Hugging Face wrapper. + +| Article | Experiment | Core observation | +| --- | --- | --- | +| [One word changed](01-one-word-routing.md) | Routing comparison | A changed answer can preserve the selected expert set at a particular block. | +| [Spot the 4-bit model](02-blind-quantization.md) | Blind Q4_1/Q8_0 comparison | Eleven of twelve captured answer pairs were identical. | +| [What else broke?](03-activation-surgery.md) | Controlled FFN activation scaling | Output changes coexisted with an unchanged four-of-six pass count. | +| [Correction detective](04-correction-detective.md) | Current and original facts | Both models passed all six short correction and recall cases. | +| [A model, heard](05-model-music.md) | Capture sonification | An explicit deterministic mapping turns 103 real token frames into audio. | + +The companion [results file](https://alphabell-model-observatory.static.hf.space/experiments/data/results.json) includes experiment prompts, outputs, checks and provenance. The [capture manifest](https://alphabell-model-observatory.static.hf.space/traces/index.json) identifies the sonification inputs. diff --git a/experiments/assets/01-routing.png b/experiments/assets/01-routing.png new file mode 100644 index 0000000000000000000000000000000000000000..c411356f479239b7bc8ca134c85debc067c726b6 Binary files /dev/null and b/experiments/assets/01-routing.png differ diff --git a/experiments/assets/02-quantization.png b/experiments/assets/02-quantization.png new file mode 100644 index 0000000000000000000000000000000000000000..c63a85306835d99ce68cda40d327758b157ccf35 Binary files /dev/null and b/experiments/assets/02-quantization.png differ diff --git a/experiments/assets/03-surgery.png b/experiments/assets/03-surgery.png new file mode 100644 index 0000000000000000000000000000000000000000..75ff79fdc84288258d07d8fd2cdb2f43595634e1 Binary files /dev/null and b/experiments/assets/03-surgery.png differ diff --git a/experiments/assets/04-memory.png b/experiments/assets/04-memory.png new file mode 100644 index 0000000000000000000000000000000000000000..1b6252196702f088d975de2ceb4ffaf8e827a2b7 Binary files /dev/null and b/experiments/assets/04-memory.png differ diff --git a/experiments/assets/05-music.png b/experiments/assets/05-music.png new file mode 100644 index 0000000000000000000000000000000000000000..5a2a98d329c86eddbb1eb298263c325b44c6f629 Binary files /dev/null and b/experiments/assets/05-music.png differ diff --git a/experiments/assets/flint-music.wav b/experiments/assets/flint-music.wav new file mode 100644 index 0000000000000000000000000000000000000000..1fdfffa9edf6673a6f0700e113552d2b53b15ae9 --- /dev/null +++ b/experiments/assets/flint-music.wav @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:71e1ea5b816e4ee18fc9f9af213056bbf34de08b57c6e035131b2e164422668a +size 910710 diff --git a/experiments/assets/flint-music.webm b/experiments/assets/flint-music.webm new file mode 100644 index 0000000000000000000000000000000000000000..0df56c1a6d51ca5f25c3c6f8a69430059808d71f --- /dev/null +++ b/experiments/assets/flint-music.webm @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c8c2eb796aa9973c0dee1dd880ea973a8ef3541f535c23b8cd92a15b98349fcc +size 989655 diff --git a/experiments/assets/media.json b/experiments/assets/media.json new file mode 100644 index 0000000000000000000000000000000000000000..4a464782097d116fcf684fd2548c0e0fe3fdb371 --- /dev/null +++ b/experiments/assets/media.json @@ -0,0 +1,27 @@ +{ + "mapping": "sound.js; 350ms per token, volume 0.35", + "flint_block": 12, + "qwen_block": 0, + "files": [ + { + "file": "flint-music.webm", + "bytes": 989655, + "sha256": "c8c2eb796aa9973c0dee1dd880ea973a8ef3541f535c23b8cd92a15b98349fcc" + }, + { + "file": "qwen-open-music.webm", + "bytes": 945509, + "sha256": "6a272628055ab07833679f48c99a2ca08f6f58d9ee2aeacc1e88b0dcb161006b" + }, + { + "file": "flint-music.wav", + "bytes": 910710, + "sha256": "71e1ea5b816e4ee18fc9f9af213056bbf34de08b57c6e035131b2e164422668a" + }, + { + "file": "qwen-open-music.wav", + "bytes": 679184, + "sha256": "5a111e668b71d07adcb4d357200baccc11abdc383f83ad9ee3c551ea9f030c23" + } + ] +} diff --git a/experiments/assets/qwen-open-music.wav b/experiments/assets/qwen-open-music.wav new file mode 100644 index 0000000000000000000000000000000000000000..7a7dd30e7c2505442f98339354344c85e7c0b12d --- /dev/null +++ b/experiments/assets/qwen-open-music.wav @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5a111e668b71d07adcb4d357200baccc11abdc383f83ad9ee3c551ea9f030c23 +size 679184 diff --git a/experiments/assets/qwen-open-music.webm b/experiments/assets/qwen-open-music.webm new file mode 100644 index 0000000000000000000000000000000000000000..3869e9a91456a9a9b5d83672562aabe83bf95e88 --- /dev/null +++ b/experiments/assets/qwen-open-music.webm @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6a272628055ab07833679f48c99a2ca08f6f58d9ee2aeacc1e88b0dcb161006b +size 945509 diff --git a/experiments/data/results.json b/experiments/data/results.json new file mode 100644 index 0000000000000000000000000000000000000000..c5adfad1ae4b23e8b1cbdc39475fd7dbf6e25744 --- /dev/null +++ b/experiments/data/results.json @@ -0,0 +1,15279 @@ +{ + "schema_version": 1, + "title": "Five alphaBell model experiments", + "generated_at": "2026-10-23T11:17:02.577204+00:00", + "methodology": { + "summary": "Real measured model runs, retained in full for this fixed case set. The public interface replays these results.", + "limitations": [ + "Each condition has one greedy generation. Small demonstrations do not establish general capabilities or population preferences.", + "Qwen captures use the alphaBell fine-tuned derivative, not the untouched Qwen parent.", + "Native experiment source and build recipe accompany the measurements." + ], + "reproduce": [ + "Download the prompts and results.", + "Use the pinned engine build and experiment runner in experiments/reproduce.", + "Match checkpoint SHA-256 values and recorded settings before comparing outputs." + ] + }, + "provenance": { + "engine": "llama.cpp 74686f084c53506a7afa471d857db02e5c912543", + "device": "CPU", + "threads": 8, + "nice": 10, + "context": 2048, + "flash_attention": false, + "kv_cache": "default F16 attention KV; model-native recurrent state", + "sampling": { + "temperature": 0, + "seed": 42, + "max_tokens": 64, + "reasoning": false + }, + "intervention": { + "tensor": "ffn_out-12", + "layer_indexing": "zero-based", + "applies_to": "all prompt and generated positions on each forward pass", + "operation": "scale computed FFN output before residual addition", + "scales": [ + 1, + 0.5, + 0 + ], + "weights_modified": false + }, + "memory_metric": "ru_maxrss process high-water mark, includes model loading and warm-up; KiB on Linux", + "latency_metric": "native monotonic request elapsed plus client wall clock; first-token latency includes prompt prefill", + "created_at": "2026-10-23T10:57:11.012459+00:00", + "source_sha256": "4a29fca44df4174dc7db51a70d7bdc0d0f27ba06d18ed4338c9da8db81e4bda9", + "binary_sha256": "4b733ebd42d2b88ecfdaf527915327e17734fad04bc7f35846fc5c9ea8f6e55d", + "checkpoints": [ + { + "model": "flint", + "quantization": "Q8_0", + "file_bytes": 42350359392, + "architecture_sha256": "1dcc4eb75752f7847b49a6cab17e377c0282050b81bb0603d33a5271112bd058", + "sha256": "f97ff2fc77564a05919357685ba8e1892bec279cf733fdb6e3ef0ea8c8e45543", + "gguf_metadata": { + "general.architecture": "flint", + "general.type": "model", + "general.sampling.top_k": 20, + "general.sampling.top_p": 0.949999988079071, + "general.sampling.temp": 1.0, + "general.name": "Flint Bf16", + "general.size_label": "256x2.1B", + "general.quantization_version": 2, + "general.file_type": 7 + }, + "tensor_count": 779 + }, + { + "model": "flint", + "quantization": "Q4_1", + "file_bytes": 24955171680, + "architecture_sha256": "1dcc4eb75752f7847b49a6cab17e377c0282050b81bb0603d33a5271112bd058", + "sha256": "1d8ea8d2fd133baa54a7412d56d39d20499a9efc080b45486f23853e3f072143", + "gguf_metadata": { + "general.architecture": "flint", + "general.type": "model", + "general.sampling.top_k": 20, + "general.sampling.top_p": 0.949999988079071, + "general.sampling.temp": 1.0, + "general.name": "Flint Bf16", + "general.size_label": "256x2.1B", + "general.quantization_version": 2, + "general.file_type": 3 + }, + "tensor_count": 779 + }, + { + "model": "qwen", + "quantization": "Q8_0", + "file_bytes": 28595763712, + "architecture_sha256": "2b58cd217c64118083a3d622cdf9c9b7488adb7bb12bc114762322062bb365c3", + "sha256": "b2a3172342ff220a8f9a06fc2fd252d1e84615ce8dfc00b2a66941610aa8f43c", + "gguf_metadata": { + "general.architecture": "qwen35", + "general.type": "model", + "general.sampling.top_k": 20, + "general.sampling.top_p": 0.949999988079071, + "general.sampling.temp": 1.0, + "general.name": "Qwen3.8 27B Open", + "general.finetune": "open", + "general.basename": "Qwen3.8", + "general.size_label": "27B", + "general.quantization_version": 2, + "general.file_type": 7 + }, + "tensor_count": 851 + }, + { + "model": "qwen", + "quantization": "Q4_1", + "file_bytes": 16829921792, + "architecture_sha256": "2b58cd217c64118083a3d622cdf9c9b7488adb7bb12bc114762322062bb365c3", + "sha256": "7c0814054c69120ff0e5b3ac3f625f98a9f86185fc39f5d71eae02cb54eb5825", + "gguf_metadata": { + "general.architecture": "qwen35", + "general.type": "model", + "general.sampling.top_k": 20, + "general.sampling.top_p": 0.949999988079071, + "general.sampling.temp": 1.0, + "general.name": "Qwen3.8 27B Open", + "general.finetune": "open", + "general.basename": "Qwen3.8", + "general.size_label": "27B", + "general.quantization_version": 2, + "general.file_type": 3 + }, + "tensor_count": 851 + } + ], + "completed_at": "2026-10-23T11:00:04.436046+00:00", + "source_tree": "83e9d4c276f84753cce063502a2a9e2325f15ce3", + "public_base_commit": "0a1ef309cdbb9f2024f58a38fb4c42352c5db569", + "source_patch_sha256": "87a1aca766c7584c433093e82e6fbfe21caa4578e8f1c1faca5b8e2385ee664d", + "header_privacy_audit": { + "passed": true, + "metadata_entries_scanned_per_file": 41, + "excluded_keys": [ + "tokenizer.ggml.tokens" + ], + "source_urls_found": 0, + "identity_matches": 0 + }, + "flint_capture_compatibility": { + "tensor_payload_equal": true, + "compared_bytes": 42344889472, + "original_offset": 5469920, + "matched_offset": 5469920, + "original_container_sha256": "147f02266f809e7bb4b28fdeef9606520d9693b601f8bc0d817caf11a2ed1e5b", + "matched_container_sha256": "f97ff2fc77564a05919357685ba8e1892bec279cf733fdb6e3ef0ea8c8e45543", + "tokenizer_equal": true, + "only_metadata_differences": [ + { + "key": "general.name", + "original": "Recursive Flint", + "matched": "Flint Bf16" + }, + { + "key": "general.size_label", + "original": "40B", + "matched": "256x2.1B" + } + ] + }, + "hardware": "AMD EPYC 7B13 CPU. Native quantization/intervention runner: 8 threads. Full topology captures: 16 threads. CPU-only.", + "model_sources": [ + { + "name": "Recursive Flint, source BF16", + "url": "https://huggingface.co/RecursiveSI/flint", + "revision": "2a247be0c0aec895ccb0ac7ca92229ec6df37288" + }, + { + "name": "alphaBell Qwen Open, related public FP8 sibling", + "url": "https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-FP8", + "revision": "38595c77e73f8d04afa003510dadd35c750590ef" + }, + { + "name": "Qwen parent architecture configuration", + "url": "https://huggingface.co/Qwen/Qwen3.8-27B", + "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0" + } + ], + "qwen_checkpoint_origin": "alphaBell fine-tuned Qwen3.8-27B, converted from the lab BF16 checkpoint to GGUF; the public FP8 release is a sibling format.", + "topology_capture_checkpoints": { + "flint": { + "quantization": "Q8_0", + "container_sha256": "147f02266f809e7bb4b28fdeef9606520d9693b601f8bc0d817caf11a2ed1e5b", + "matched_tensor_payload_identical": true, + "proof": "https://huggingface.co/spaces/alphabell/Model-Observatory/resolve/main/experiments/reproduce/flint-capture-equivalence.json" + }, + "qwen": { + "quantization": "Q8_0", + "container_sha256": "b2a3172342ff220a8f9a06fc2fd252d1e84615ce8dfc00b2a66941610aa8f43c" + } + } + }, + "experiments": { + "reroute": { + "title": "One word changed.", + "summary": "Three prompt pairs, two architectures, twelve complete captures. Inspect the first output prediction before continuations diverge.", + "method": "Change one word with temperature 0, seed 42 and reasoning disabled. Compare the same model and precision. The displayed Flint block is the lowest block with a changed first-frame expert set.", + "limitations": [ + "Later output positions are ordinal, not semantic alignments. Routing changes do not localize concepts or establish a causal explanation of an answer." + ], + "cases": [ + { + "id": "json-color-flint", + "title": "One word changes the JSON value · Flint", + "prompts": [ + { + "id": "a", + "label": "Prompt A", + "text": "Return only JSON with one key named color and value blue." + }, + { + "id": "b", + "label": "Prompt B", + "text": "Return only JSON with one key named color and value red." + } + ], + "conditions": [ + { + "id": "a", + "label": "Prompt A · Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Return only JSON with one key named color and value blue.", + "output": "{\"color\":\"blue\"}", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=reroute-json-color-flint-a&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 28, + "elapsed_seconds": 1.124171917, + "termination": "eos", + "displayed_block": 3 + }, + "routing": { + "layer": 3, + "expert_count": 256, + "description": "Measured selected experts in zero-based block 3.", + "tokens": [ + { + "token": "{\"", + "experts": [ + 117, + 87, + 35, + 5, + 168, + 84 + ], + "weights": [ + 0.32084164023399353, + 0.2539198398590088, + 0.14681749045848846, + 0.12157176434993744, + 0.10437823086977005, + 0.052470993250608444 + ] + }, + { + "token": "color", + "experts": [ + 39, + 59, + 143, + 13, + 34, + 250 + ], + "weights": [ + 0.33017292618751526, + 0.30293041467666626, + 0.1303112655878067, + 0.10549050569534302, + 0.09110484272241592, + 0.03999003767967224 + ] + }, + { + "token": "\":\"", + "experts": [ + 115, + 224, + 186, + 216, + 15, + 29 + ], + "weights": [ + 0.6117783784866333, + 0.09889795631170273, + 0.09107732027769089, + 0.07742965966463089, + 0.06268590688705444, + 0.05813080072402954 + ] + }, + { + "token": "blue", + "experts": [ + 39, + 111, + 197, + 137, + 119, + 28 + ], + "weights": [ + 0.4875814616680145, + 0.17773117125034332, + 0.10596126317977905, + 0.09250641614198685, + 0.08542586863040924, + 0.05079379305243492 + ] + }, + { + "token": "\"}", + "experts": [ + 216, + 1, + 152, + 102, + 182, + 15 + ], + "weights": [ + 0.38251131772994995, + 0.3607138693332672, + 0.07649347931146622, + 0.06965707242488861, + 0.06540820747613907, + 0.04521603509783745 + ] + } + ] + } + }, + { + "id": "b", + "label": "Prompt B · Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Return only JSON with one key named color and value red.", + "output": "{\"color\":\"red\"}", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=reroute-json-color-flint-b&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 28, + "elapsed_seconds": 1.116444861, + "termination": "eos", + "displayed_block": 3 + }, + "routing": { + "layer": 3, + "expert_count": 256, + "description": "Measured selected experts in zero-based block 3.", + "tokens": [ + { + "token": "{\"", + "experts": [ + 117, + 87, + 35, + 5, + 168, + 33 + ], + "weights": [ + 0.3200564980506897, + 0.25333839654922485, + 0.1516059786081314, + 0.11339818686246872, + 0.10805189609527588, + 0.05354905501008034 + ] + }, + { + "token": "color", + "experts": [ + 39, + 59, + 143, + 13, + 34, + 250 + ], + "weights": [ + 0.3271667957305908, + 0.29962509870529175, + 0.1341659426689148, + 0.1078995019197464, + 0.0922822579741478, + 0.03886038064956665 + ] + }, + { + "token": "\":\"", + "experts": [ + 115, + 224, + 186, + 216, + 15, + 29 + ], + "weights": [ + 0.6083868145942688, + 0.101140096783638, + 0.0891016498208046, + 0.07671405375003815, + 0.06320527195930481, + 0.06145205348730087 + ] + }, + { + "token": "red", + "experts": [ + 39, + 111, + 197, + 137, + 119, + 28 + ], + "weights": [ + 0.48268941044807434, + 0.1775527149438858, + 0.10889095813035965, + 0.09247402846813202, + 0.08805633336305618, + 0.05033653602004051 + ] + }, + { + "token": "\"}", + "experts": [ + 1, + 6, + 199, + 216, + 48, + 187 + ], + "weights": [ + 0.3797229826450348, + 0.31145796179771423, + 0.13263653218746185, + 0.08601834625005722, + 0.04618379473686218, + 0.043980374932289124 + ] + } + ] + } + } + ], + "comparison": { + "comparison_frame": "First output prediction, before generated continuations diverge.", + "prompt_words_changed": 1, + "prompt_token_counts": [ + 28, + 28 + ], + "routed_blocks_compared": 47, + "blocks_with_different_expert_sets": 18, + "expert_selections_in_each_first_frame": 282, + "shared_expert_selections": 263, + "displayed_zero_based_block": 3, + "display_rule": "Lowest routed block with a different first-frame expert set; full captures contain every block.", + "first_frame_head_rms_relative_l2": 0.10240989178622689 + } + }, + { + "id": "json-color-qwen", + "title": "One word changes the JSON value · Qwen Open", + "prompts": [ + { + "id": "a", + "label": "Prompt A", + "text": "Return only JSON with one key named color and value blue." + }, + { + "id": "b", + "label": "Prompt B", + "text": "Return only JSON with one key named color and value red." + } + ], + "conditions": [ + { + "id": "a", + "label": "Prompt A · alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Return only JSON with one key named color and value blue.", + "output": "{\"color\":\"blue\"}", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=reroute-json-color-qwen-a&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 42, + "elapsed_seconds": 5.535615226, + "termination": "eos", + "displayed_block": 0 + } + }, + { + "id": "b", + "label": "Prompt B · alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Return only JSON with one key named color and value red.", + "output": "{\"color\":\"red\"}", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=reroute-json-color-qwen-b&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 42, + "elapsed_seconds": 5.440314309, + "termination": "eos", + "displayed_block": 0 + } + } + ], + "comparison": { + "comparison_frame": "First output prediction, before generated continuations diverge.", + "prompt_words_changed": 1, + "prompt_token_counts": [ + 42, + 42 + ], + "first_frame_head_rms_relative_l2": 0.02018521900284211 + } + }, + { + "id": "opposite-flint", + "title": "One word changes the opposite · Flint", + "prompts": [ + { + "id": "a", + "label": "Prompt A", + "text": "Answer with one word: name the opposite of hot." + }, + { + "id": "b", + "label": "Prompt B", + "text": "Answer with one word: name the opposite of cold." + } + ], + "conditions": [ + { + "id": "a", + "label": "Prompt A · Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Answer with one word: name the opposite of hot.", + "output": "Cold", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=reroute-opposite-flint-a&token=1", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 27, + "elapsed_seconds": 0.695859422, + "termination": "eos", + "displayed_block": 2 + }, + "routing": { + "layer": 2, + "expert_count": 256, + "description": "Measured selected experts in zero-based block 2.", + "tokens": [ + { + "token": "Cold", + "experts": [ + 88, + 213, + 186, + 46, + 162, + 237 + ], + "weights": [ + 0.5531415939331055, + 0.178231343626976, + 0.09052453190088272, + 0.08529653400182724, + 0.04749872162938118, + 0.045307233929634094 + ] + } + ] + } + }, + { + "id": "b", + "label": "Prompt B · Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Answer with one word: name the opposite of cold.", + "output": "Hot", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=reroute-opposite-flint-b&token=1", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 27, + "elapsed_seconds": 0.644852095, + "termination": "eos", + "displayed_block": 2 + }, + "routing": { + "layer": 2, + "expert_count": 256, + "description": "Measured selected experts in zero-based block 2.", + "tokens": [ + { + "token": "Hot", + "experts": [ + 88, + 213, + 186, + 46, + 162, + 1 + ], + "weights": [ + 0.5620047450065613, + 0.17373046278953552, + 0.09409793466329575, + 0.07710109651088715, + 0.047040797770023346, + 0.046024974435567856 + ] + } + ] + } + } + ], + "comparison": { + "comparison_frame": "First output prediction, before generated continuations diverge.", + "prompt_words_changed": 1, + "prompt_token_counts": [ + 27, + 27 + ], + "routed_blocks_compared": 47, + "blocks_with_different_expert_sets": 25, + "expert_selections_in_each_first_frame": 282, + "shared_expert_selections": 243, + "displayed_zero_based_block": 2, + "display_rule": "Lowest routed block with a different first-frame expert set; full captures contain every block.", + "first_frame_head_rms_relative_l2": 0.25766445665876486 + } + }, + { + "id": "opposite-qwen", + "title": "One word changes the opposite · Qwen Open", + "prompts": [ + { + "id": "a", + "label": "Prompt A", + "text": "Answer with one word: name the opposite of hot." + }, + { + "id": "b", + "label": "Prompt B", + "text": "Answer with one word: name the opposite of cold." + } + ], + "conditions": [ + { + "id": "a", + "label": "Prompt A · alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Answer with one word: name the opposite of hot.", + "output": "Cold", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=reroute-opposite-qwen-a&token=1", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 41, + "elapsed_seconds": 3.133467638, + "termination": "eos", + "displayed_block": 0 + } + }, + { + "id": "b", + "label": "Prompt B · alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Answer with one word: name the opposite of cold.", + "output": "Hot", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=reroute-opposite-qwen-b&token=1", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 41, + "elapsed_seconds": 3.118574647, + "termination": "eos", + "displayed_block": 0 + } + } + ], + "comparison": { + "comparison_frame": "First output prediction, before generated continuations diverge.", + "prompt_words_changed": 1, + "prompt_token_counts": [ + 41, + 41 + ], + "first_frame_head_rms_relative_l2": 0.18418391836536593 + } + }, + { + "id": "python-flint", + "title": "One word changes the function · Flint", + "prompts": [ + { + "id": "a", + "label": "Prompt A", + "text": "Write only a Python expression that selects the maximum of x and y." + }, + { + "id": "b", + "label": "Prompt B", + "text": "Write only a Python expression that selects the minimum of x and y." + } + ], + "conditions": [ + { + "id": "a", + "label": "Prompt A · Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Write only a Python expression that selects the maximum of x and y.", + "output": "max(x, y)", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=reroute-python-flint-a&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 30, + "elapsed_seconds": 1.156945927, + "termination": "eos", + "displayed_block": 3 + }, + "routing": { + "layer": 3, + "expert_count": 256, + "description": "Measured selected experts in zero-based block 3.", + "tokens": [ + { + "token": "max", + "experts": [ + 117, + 87, + 35, + 5, + 168, + 57 + ], + "weights": [ + 0.4010581374168396, + 0.21458370983600616, + 0.1538792848587036, + 0.10982739180326462, + 0.07339246571063995, + 0.047259047627449036 + ] + }, + { + "token": "(x", + "experts": [ + 219, + 149, + 69, + 243, + 106, + 168 + ], + "weights": [ + 0.356155127286911, + 0.28351446986198425, + 0.15893621742725372, + 0.0888138860464096, + 0.08155017346143723, + 0.031030092388391495 + ] + }, + { + "token": ",", + "experts": [ + 124, + 25, + 177, + 158, + 255, + 224 + ], + "weights": [ + 0.5517688393592834, + 0.12534935772418976, + 0.1014571487903595, + 0.09167846292257309, + 0.07228443771600723, + 0.057461753487586975 + ] + }, + { + "token": " y", + "experts": [ + 19, + 77, + 14, + 203, + 96, + 121 + ], + "weights": [ + 0.4720388352870941, + 0.34413912892341614, + 0.0540812686085701, + 0.04676710069179535, + 0.04236053302884102, + 0.040613144636154175 + ] + }, + { + "token": ")", + "experts": [ + 124, + 25, + 21, + 66, + 235, + 28 + ], + "weights": [ + 0.4778953790664673, + 0.24744132161140442, + 0.10221158713102341, + 0.06340343505144119, + 0.05989007651805878, + 0.04915820062160492 + ] + } + ] + } + }, + { + "id": "b", + "label": "Prompt B · Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Write only a Python expression that selects the minimum of x and y.", + "output": "min(x, y)", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=reroute-python-flint-b&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 30, + "elapsed_seconds": 1.128900005, + "termination": "eos", + "displayed_block": 3 + }, + "routing": { + "layer": 3, + "expert_count": 256, + "description": "Measured selected experts in zero-based block 3.", + "tokens": [ + { + "token": "min", + "experts": [ + 117, + 87, + 35, + 5, + 168, + 7 + ], + "weights": [ + 0.4022853970527649, + 0.23592405021190643, + 0.1466451734304428, + 0.09799264371395111, + 0.06999532133340836, + 0.04715738818049431 + ] + }, + { + "token": "(x", + "experts": [ + 149, + 219, + 189, + 69, + 243, + 106 + ], + "weights": [ + 0.4026070237159729, + 0.15572281181812286, + 0.1464761197566986, + 0.1269456148147583, + 0.09798307716846466, + 0.07026540488004684 + ] + }, + { + "token": ",", + "experts": [ + 124, + 25, + 177, + 255, + 158, + 74 + ], + "weights": [ + 0.5216768383979797, + 0.13582339882850647, + 0.10417815297842026, + 0.09238538146018982, + 0.07687641680240631, + 0.0690598264336586 + ] + }, + { + "token": " y", + "experts": [ + 19, + 77, + 14, + 203, + 81, + 121 + ], + "weights": [ + 0.46713194251060486, + 0.34391525387763977, + 0.056301526725292206, + 0.04728133976459503, + 0.04271211847662926, + 0.042657796293497086 + ] + }, + { + "token": ")", + "experts": [ + 124, + 25, + 21, + 66, + 235, + 28 + ], + "weights": [ + 0.4766605496406555, + 0.2452375292778015, + 0.11082471162080765, + 0.062690868973732, + 0.05392761155962944, + 0.050658706575632095 + ] + } + ] + } + } + ], + "comparison": { + "comparison_frame": "First output prediction, before generated continuations diverge.", + "prompt_words_changed": 1, + "prompt_token_counts": [ + 30, + 30 + ], + "routed_blocks_compared": 47, + "blocks_with_different_expert_sets": 20, + "expert_selections_in_each_first_frame": 282, + "shared_expert_selections": 255, + "displayed_zero_based_block": 3, + "display_rule": "Lowest routed block with a different first-frame expert set; full captures contain every block.", + "first_frame_head_rms_relative_l2": 0.1035147140665932 + } + }, + { + "id": "python-qwen", + "title": "One word changes the function · Qwen Open", + "prompts": [ + { + "id": "a", + "label": "Prompt A", + "text": "Write only a Python expression that selects the maximum of x and y." + }, + { + "id": "b", + "label": "Prompt B", + "text": "Write only a Python expression that selects the minimum of x and y." + } + ], + "conditions": [ + { + "id": "a", + "label": "Prompt A · alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Write only a Python expression that selects the maximum of x and y.", + "output": "max(x, y)", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=reroute-python-qwen-a&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 44, + "elapsed_seconds": 5.603033013, + "termination": "eos", + "displayed_block": 0 + } + }, + { + "id": "b", + "label": "Prompt B · alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Write only a Python expression that selects the minimum of x and y.", + "output": "min(x, y)", + "settings": { + "max_tokens": 48, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=reroute-python-qwen-b&token=1", + "metrics": { + "generated_tokens": 5, + "prompt_tokens": 44, + "elapsed_seconds": 5.566880933, + "termination": "eos", + "displayed_block": 0 + } + } + ], + "comparison": { + "comparison_frame": "First output prediction, before generated continuations diverge.", + "prompt_words_changed": 1, + "prompt_token_counts": [ + 44, + 44 + ], + "first_frame_head_rms_relative_l2": 0.1294852695128142 + } + } + ] + }, + "quant": { + "title": "Can you spot the 4-bit model?", + "summary": "Twelve blind pairs from six fixed tasks on Flint and alphaBell Qwen Open. Eleven pairs produced identical text; one arithmetic pair differed and both answers were wrong.", + "method": "Matched BF16-derived Q4_1 and Q8_0 files; identical prompts and decoding. Four separate CPU model processes, one warm-up per process. All cases retained.", + "limitations": [ + "Six tasks per model, one generation per condition. This is not a general quality or speed ranking. Process peak RSS is a high-water mark including loading and warm-up, not a per-request increment. Timings include tracing overhead and are single observations." + ], + "cases": [ + { + "id": "flint-arithmetic", + "title": "Exact arithmetic", + "prompt": "Calculate (17 * 23) - 19. Reply with the integer only.", + "checks": [ + "Rule: integer", + "Expected: 372", + "Q8_0: failed", + "Q4_1: failed" + ], + "conditions": [ + { + "id": "flint-q8_0", + "label": "Q8_0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "374", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 33, + "elapsed_seconds": 0.738173075, + "request_wall_seconds": 0.7542800180381164, + "time_to_first_token_seconds": 0.689367342973128, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0042319691359885515, + "block12_after_rms": 0.0042319691359885515 + }, + "check": { + "passed": false, + "rule": "integer", + "expected": "372", + "character_count": 3, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 9056, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004216483101124048, + "before_rms": 0.004216483101124048, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.0042319691359885515, + "before_rms": 0.0042319691359885515, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.005525138486063405, + "before_rms": 0.005525138486063405, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.014221511290226446, + "before_rms": 0.014221511290226446, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 270336 + } + ] + } + ] + }, + { + "id": "flint-q4_1", + "label": "Q4_1", + "model": "Recursive Flint", + "quant": "Q4_1", + "output": "352", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 33, + "elapsed_seconds": 0.781993782, + "request_wall_seconds": 0.798555999994278, + "time_to_first_token_seconds": 0.7473802480380982, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 24811756, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 24955171680, + "block12_before_rms": 0.00415956677122455, + "block12_after_rms": 0.00415956677122455 + }, + "check": { + "passed": false, + "rule": "integer", + "expected": "372", + "character_count": 3, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 11510, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004206818881751983, + "before_rms": 0.004206818881751983, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.00415956677122455, + "before_rms": 0.00415956677122455, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.005537144078691912, + "before_rms": 0.005537144078691912, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.014134159625799036, + "before_rms": 0.014134159625799036, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 270336 + } + ] + } + ] + } + ], + "model": "flint" + }, + { + "id": "flint-json", + "title": "Strict JSON format", + "prompt": "Return exactly this JSON object and no other text: {\"status\":\"ok\",\"count\":3}", + "checks": [ + "Rule: json", + "Expected: {'status': 'ok', 'count': 3}", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "flint-q8_0", + "label": "Q8_0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "{\"status\":\"ok\",\"count\":3}", + "metrics": { + "generated_tokens": 9, + "prompt_tokens": 35, + "elapsed_seconds": 1.259416502, + "request_wall_seconds": 1.2757591230329126, + "time_to_first_token_seconds": 0.7007529039401561, + "decode_tokens_per_second": 13.911468772577386, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004073247840262152, + "block12_after_rms": 0.004073247840262152 + }, + "check": { + "passed": true, + "rule": "json", + "expected": { + "status": "ok", + "count": 3 + }, + "character_count": 25, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 5482, + "input_position": 34, + "layers": [ + { + "after_rms": 0.00442951382677711, + "before_rms": 0.00442951382677711, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.004073247840262152, + "before_rms": 0.004073247840262152, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.00524975286610561, + "before_rms": 0.00524975286610561, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.014444385872118682, + "before_rms": 0.014444385872118682, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 286720 + } + ] + }, + { + "step": 1, + "token_id": 5462, + "input_position": 35, + "layers": [ + { + "after_rms": 0.005793034803083275, + "before_rms": 0.005793034803083275, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0046820736039610645, + "before_rms": 0.0046820736039610645, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00649377540352318, + "before_rms": 0.00649377540352318, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019726933354943656, + "before_rms": 0.019726933354943656, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 4890, + "input_position": 36, + "layers": [ + { + "after_rms": 0.005294017353787343, + "before_rms": 0.005294017353787343, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0034695353362188502, + "before_rms": 0.0034695353362188502, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00738829527350583, + "before_rms": 0.00738829527350583, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019360584231844193, + "before_rms": 0.019360584231844193, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 544, + "input_position": 37, + "layers": [ + { + "after_rms": 0.00491148399515983, + "before_rms": 0.00491148399515983, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00380345983046727, + "before_rms": 0.00380345983046727, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005149091746140284, + "before_rms": 0.005149091746140284, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016276301961648348, + "before_rms": 0.016276301961648348, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1147, + "input_position": 38, + "layers": [ + { + "after_rms": 0.006222679356489009, + "before_rms": 0.006222679356489009, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0036829646504058975, + "before_rms": 0.0036829646504058975, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006655483694052688, + "before_rms": 0.006655483694052688, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019208434114849963, + "before_rms": 0.019208434114849963, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 2816, + "input_position": 39, + "layers": [ + { + "after_rms": 0.005760162518562755, + "before_rms": 0.005760162518562755, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003275269655906223, + "before_rms": 0.003275269655906223, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004388659397945476, + "before_rms": 0.004388659397945476, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015988995629268844, + "before_rms": 0.015988995629268844, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 1129, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004863272662823825, + "before_rms": 0.004863272662823825, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0038314286271215804, + "before_rms": 0.0038314286271215804, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007286998970545343, + "before_rms": 0.007286998970545343, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018843493317539196, + "before_rms": 0.018843493317539196, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 18, + "input_position": 41, + "layers": [ + { + "after_rms": 0.004603991231053795, + "before_rms": 0.004603991231053795, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0030208238582806433, + "before_rms": 0.0030208238582806433, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004703706397060763, + "before_rms": 0.004703706397060763, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013840299536615416, + "before_rms": 0.013840299536615416, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 92, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0039039619301298022, + "before_rms": 0.0039039619301298022, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004057874942645444, + "before_rms": 0.004057874942645444, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006582297878179331, + "before_rms": 0.006582297878179331, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016272639100438333, + "before_rms": 0.016272639100438333, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "flint-q4_1", + "label": "Q4_1", + "model": "Recursive Flint", + "quant": "Q4_1", + "output": "{\"status\":\"ok\",\"count\":3}", + "metrics": { + "generated_tokens": 9, + "prompt_tokens": 35, + "elapsed_seconds": 1.251735163, + "request_wall_seconds": 1.2701693959534168, + "time_to_first_token_seconds": 0.7594751949654892, + "decode_tokens_per_second": 15.663206261728522, + "process_peak_rss_kib": 24811756, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 24955171680, + "block12_before_rms": 0.004019226021822854, + "block12_after_rms": 0.004019226021822854 + }, + "check": { + "passed": true, + "rule": "json", + "expected": { + "status": "ok", + "count": 3 + }, + "character_count": 25, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 5482, + "input_position": 34, + "layers": [ + { + "after_rms": 0.004480802624651158, + "before_rms": 0.004480802624651158, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.004019226021822854, + "before_rms": 0.004019226021822854, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.005142506994361342, + "before_rms": 0.005142506994361342, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.014475124728967394, + "before_rms": 0.014475124728967394, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 286720 + } + ] + }, + { + "step": 1, + "token_id": 5462, + "input_position": 35, + "layers": [ + { + "after_rms": 0.005884499521933463, + "before_rms": 0.005884499521933463, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004714484449236878, + "before_rms": 0.004714484449236878, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006689554817134999, + "before_rms": 0.006689554817134999, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020091649545950115, + "before_rms": 0.020091649545950115, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 4890, + "input_position": 36, + "layers": [ + { + "after_rms": 0.005296309235185447, + "before_rms": 0.005296309235185447, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0034924733951837345, + "before_rms": 0.0034924733951837345, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00761576937963421, + "before_rms": 0.00761576937963421, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01972855739429675, + "before_rms": 0.01972855739429675, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 544, + "input_position": 37, + "layers": [ + { + "after_rms": 0.005061610553787093, + "before_rms": 0.005061610553787093, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003724008220189829, + "before_rms": 0.003724008220189829, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004973375331562022, + "before_rms": 0.004973375331562022, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016514705424796036, + "before_rms": 0.016514705424796036, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1147, + "input_position": 38, + "layers": [ + { + "after_rms": 0.006274683971296353, + "before_rms": 0.006274683971296353, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0037093873591057375, + "before_rms": 0.0037093873591057375, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006668038059028412, + "before_rms": 0.006668038059028412, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019377753581390548, + "before_rms": 0.019377753581390548, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 2816, + "input_position": 39, + "layers": [ + { + "after_rms": 0.005775051951262021, + "before_rms": 0.005775051951262021, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0033710668003589326, + "before_rms": 0.0033710668003589326, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004385254089389432, + "before_rms": 0.004385254089389432, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015951241921845414, + "before_rms": 0.015951241921845414, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 1129, + "input_position": 40, + "layers": [ + { + "after_rms": 0.00490185922327077, + "before_rms": 0.00490185922327077, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003771268940693942, + "before_rms": 0.003771268940693942, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007377846105883496, + "before_rms": 0.007377846105883496, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019096257942254946, + "before_rms": 0.019096257942254946, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 18, + "input_position": 41, + "layers": [ + { + "after_rms": 0.004675410678881592, + "before_rms": 0.004675410678881592, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0029219677425947276, + "before_rms": 0.0029219677425947276, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00468134851761768, + "before_rms": 0.00468134851761768, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013991205065508688, + "before_rms": 0.013991205065508688, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 92, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0039219998776194335, + "before_rms": 0.0039219998776194335, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004008549607569253, + "before_rms": 0.004008549607569253, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006525887799501659, + "before_rms": 0.006525887799501659, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016431090183615604, + "before_rms": 0.016431090183615604, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + } + ], + "model": "flint" + }, + { + "id": "flint-fact", + "title": "Short factual recall", + "prompt": "What is the chemical symbol for gold? Reply with the symbol only.", + "checks": [ + "Rule: exact", + "Expected: Au", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "flint-q8_0", + "label": "Q8_0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Au", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 30, + "elapsed_seconds": 0.650667801, + "request_wall_seconds": 0.6673325290903449, + "time_to_first_token_seconds": 0.6034285150235519, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004120685912594556, + "block12_after_rms": 0.004120685912594556 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "Au", + "character_count": 2, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 22911, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0042503533024456225, + "before_rms": 0.0042503533024456225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.004120685912594556, + "before_rms": 0.004120685912594556, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.005313107066375563, + "before_rms": 0.005313107066375563, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.01490764588916619, + "before_rms": 0.01490764588916619, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 245760 + } + ] + } + ] + }, + { + "id": "flint-q4_1", + "label": "Q4_1", + "model": "Recursive Flint", + "quant": "Q4_1", + "output": "Au", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 30, + "elapsed_seconds": 0.713642926, + "request_wall_seconds": 0.7302313540130854, + "time_to_first_token_seconds": 0.6695935919415206, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 24811756, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 24955171680, + "block12_before_rms": 0.0040212076224896524, + "block12_after_rms": 0.0040212076224896524 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "Au", + "character_count": 2, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 22911, + "input_position": 29, + "layers": [ + { + "after_rms": 0.004262134108703956, + "before_rms": 0.004262134108703956, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.0040212076224896524, + "before_rms": 0.0040212076224896524, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.005205860826013204, + "before_rms": 0.005205860826013204, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.014788299051992781, + "before_rms": 0.014788299051992781, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 245760 + } + ] + } + ] + } + ], + "model": "flint" + }, + { + "id": "flint-correction", + "title": "Use the corrected fact", + "prompt": "The access color was blue. Correction: the access color is now amber. What is the current access color? Reply with one word only.", + "checks": [ + "Rule: exact", + "Expected: amber", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "flint-q8_0", + "label": "Q8_0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "amber", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 44, + "elapsed_seconds": 0.974556193, + "request_wall_seconds": 0.9955227529862896, + "time_to_first_token_seconds": 0.9343198160640895, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.00441159816545625, + "block12_after_rms": 0.00441159816545625 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "amber", + "character_count": 5, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 13732, + "input_position": 43, + "layers": [ + { + "after_rms": 0.004252703885422568, + "before_rms": 0.004252703885422568, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.00441159816545625, + "before_rms": 0.00441159816545625, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.005657736200656421, + "before_rms": 0.005657736200656421, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.014876767054565872, + "before_rms": 0.014876767054565872, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 360448 + } + ] + } + ] + }, + { + "id": "flint-q4_1", + "label": "Q4_1", + "model": "Recursive Flint", + "quant": "Q4_1", + "output": "amber", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 44, + "elapsed_seconds": 1.028451992, + "request_wall_seconds": 1.0458013030001894, + "time_to_first_token_seconds": 0.9873899329686537, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 24813804, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 24955171680, + "block12_before_rms": 0.004339712902236177, + "block12_after_rms": 0.004339712902236177 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "amber", + "character_count": 5, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 13732, + "input_position": 43, + "layers": [ + { + "after_rms": 0.004209592099785874, + "before_rms": 0.004209592099785874, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.004339712902236177, + "before_rms": 0.004339712902236177, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.005707988547356796, + "before_rms": 0.005707988547356796, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.014914561282848718, + "before_rms": 0.014914561282848718, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 360448 + } + ] + } + ] + } + ], + "model": "flint" + }, + { + "id": "flint-code", + "title": "Python expression repair", + "prompt": "Complete the function with one Python expression only, no markdown: def is_even(n): return ___ . The function must return True exactly when integer n is even.", + "checks": [ + "Rule: expression", + "Expected: n % 2 == 0", + "Q8_0: failed", + "Q4_1: failed" + ], + "conditions": [ + { + "id": "flint-q8_0", + "label": "Q8_0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "return n % 2 == 0", + "metrics": { + "generated_tokens": 8, + "prompt_tokens": 48, + "elapsed_seconds": 1.522581758, + "request_wall_seconds": 1.5477090129861608, + "time_to_first_token_seconds": 0.9909320520237088, + "decode_tokens_per_second": 12.570319015535855, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004202357680490731, + "block12_after_rms": 0.004202357680490731 + }, + "check": { + "passed": false, + "rule": "expression", + "expected": "n % 2 == 0", + "character_count": 17, + "word_count": 6 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 963, + "input_position": 47, + "layers": [ + { + "after_rms": 0.004194344649342437, + "before_rms": 0.004194344649342437, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.004202357680490731, + "before_rms": 0.004202357680490731, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.0052131473290487635, + "before_rms": 0.0052131473290487635, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.014781441026154475, + "before_rms": 0.014781441026154475, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 393216 + } + ] + }, + { + "step": 1, + "token_id": 307, + "input_position": 48, + "layers": [ + { + "after_rms": 0.005277509304535801, + "before_rms": 0.005277509304535801, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00609774510971192, + "before_rms": 0.00609774510971192, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005608145566049825, + "before_rms": 0.005608145566049825, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018110784621771632, + "before_rms": 0.018110784621771632, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 1152, + "input_position": 49, + "layers": [ + { + "after_rms": 0.004954062696681225, + "before_rms": 0.004954062696681225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004947633813264256, + "before_rms": 0.004947633813264256, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006281951776179792, + "before_rms": 0.006281951776179792, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01793926095231872, + "before_rms": 0.01793926095231872, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 220, + "input_position": 50, + "layers": [ + { + "after_rms": 0.005725603347206012, + "before_rms": 0.005725603347206012, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007613934635856348, + "before_rms": 0.007613934635856348, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007067586106815484, + "before_rms": 0.007067586106815484, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02065595811031347, + "before_rms": 0.02065595811031347, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 17, + "input_position": 51, + "layers": [ + { + "after_rms": 0.004384558665618408, + "before_rms": 0.004384558665618408, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002694924917014614, + "before_rms": 0.002694924917014614, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003985706515046729, + "before_rms": 0.003985706515046729, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.012855823082554165, + "before_rms": 0.012855823082554165, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 823, + "input_position": 52, + "layers": [ + { + "after_rms": 0.00695481022115809, + "before_rms": 0.00695481022115809, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005148284890432175, + "before_rms": 0.005148284890432175, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00860817356635732, + "before_rms": 0.00860817356635732, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019382560123796404, + "before_rms": 0.019382560123796404, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 220, + "input_position": 53, + "layers": [ + { + "after_rms": 0.008360185779230117, + "before_rms": 0.008360185779230117, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005331645591182674, + "before_rms": 0.005331645591182674, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.011988298999222614, + "before_rms": 0.011988298999222614, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02324778120292584, + "before_rms": 0.02324778120292584, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 15, + "input_position": 54, + "layers": [ + { + "after_rms": 0.004389820760472024, + "before_rms": 0.004389820760472024, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002823151948738866, + "before_rms": 0.002823151948738866, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004278350453083995, + "before_rms": 0.004278350453083995, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013667174465740126, + "before_rms": 0.013667174465740126, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "flint-q4_1", + "label": "Q4_1", + "model": "Recursive Flint", + "quant": "Q4_1", + "output": "return n % 2 == 0", + "metrics": { + "generated_tokens": 8, + "prompt_tokens": 48, + "elapsed_seconds": 1.428655518, + "request_wall_seconds": 1.4456852129660547, + "time_to_first_token_seconds": 1.0364669429836795, + "decode_tokens_per_second": 17.098353812752265, + "process_peak_rss_kib": 24813804, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 24955171680, + "block12_before_rms": 0.004124282082456951, + "block12_after_rms": 0.004124282082456951 + }, + "check": { + "passed": false, + "rule": "expression", + "expected": "n % 2 == 0", + "character_count": 17, + "word_count": 6 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 963, + "input_position": 47, + "layers": [ + { + "after_rms": 0.004197426447709307, + "before_rms": 0.004197426447709307, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.004124282082456951, + "before_rms": 0.004124282082456951, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.005129178661979122, + "before_rms": 0.005129178661979122, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.014727759161136935, + "before_rms": 0.014727759161136935, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 393216 + } + ] + }, + { + "step": 1, + "token_id": 307, + "input_position": 48, + "layers": [ + { + "after_rms": 0.00529162387726611, + "before_rms": 0.00529162387726611, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006311261171671491, + "before_rms": 0.006311261171671491, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0055793974325739105, + "before_rms": 0.0055793974325739105, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018152769626134782, + "before_rms": 0.018152769626134782, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 1152, + "input_position": 49, + "layers": [ + { + "after_rms": 0.004977992102677547, + "before_rms": 0.004977992102677547, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005044167186539479, + "before_rms": 0.005044167186539479, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00618730909234251, + "before_rms": 0.00618730909234251, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017977599413659836, + "before_rms": 0.017977599413659836, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 220, + "input_position": 50, + "layers": [ + { + "after_rms": 0.005692971542768754, + "before_rms": 0.005692971542768754, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007667436190392576, + "before_rms": 0.007667436190392576, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00715658608387803, + "before_rms": 0.00715658608387803, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02069278331850969, + "before_rms": 0.02069278331850969, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 17, + "input_position": 51, + "layers": [ + { + "after_rms": 0.0043834519236579, + "before_rms": 0.0043834519236579, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0027110249785654086, + "before_rms": 0.0027110249785654086, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003933361905406773, + "before_rms": 0.003933361905406773, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01299392506244475, + "before_rms": 0.01299392506244475, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 823, + "input_position": 52, + "layers": [ + { + "after_rms": 0.007086739036977866, + "before_rms": 0.007086739036977866, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005068799612035294, + "before_rms": 0.005068799612035294, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008616946732804713, + "before_rms": 0.008616946732804713, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019539045407508844, + "before_rms": 0.019539045407508844, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 220, + "input_position": 53, + "layers": [ + { + "after_rms": 0.008330422942053637, + "before_rms": 0.008330422942053637, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0052894895354913506, + "before_rms": 0.0052894895354913506, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.012248586143235164, + "before_rms": 0.012248586143235164, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.023511864970171558, + "before_rms": 0.023511864970171558, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 15, + "input_position": 54, + "layers": [ + { + "after_rms": 0.0044348412653756105, + "before_rms": 0.0044348412653756105, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0027982348810609656, + "before_rms": 0.0027982348810609656, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004289321679896335, + "before_rms": 0.004289321679896335, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013781103726701278, + "before_rms": 0.013781103726701278, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + } + ], + "model": "flint" + }, + { + "id": "flint-brief", + "title": "One sentence explanation", + "prompt": "Explain photosynthesis in exactly one sentence containing the words sunlight and glucose.", + "checks": [ + "Rule: sentence", + "Expected: ['sunlight', 'glucose']", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "flint-q8_0", + "label": "Q8_0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Photosynthesis is the process by which plants use sunlight to create glucose, storing solar energy in chemical form.", + "metrics": { + "generated_tokens": 21, + "prompt_tokens": 29, + "elapsed_seconds": 1.985051171, + "request_wall_seconds": 2.0436750209191814, + "time_to_first_token_seconds": 0.6549666819628328, + "decode_tokens_per_second": 14.402335769662937, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0044633950081139125, + "block12_after_rms": 0.0044633950081139125 + }, + "check": { + "passed": true, + "rule": "sentence", + "expected": [ + "sunlight", + "glucose" + ], + "character_count": 116, + "word_count": 18 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 49106, + "input_position": 28, + "layers": [ + { + "after_rms": 0.004260129892907466, + "before_rms": 0.004260129892907466, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.0044633950081139125, + "before_rms": 0.0044633950081139125, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.005860773055104808, + "before_rms": 0.005860773055104808, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.015381996005573538, + "before_rms": 0.015381996005573538, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 237568 + } + ] + }, + { + "step": 1, + "token_id": 25424, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0056475146712665295, + "before_rms": 0.0056475146712665295, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008613622622386482, + "before_rms": 0.008613622622386482, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006688744656356641, + "before_rms": 0.006688744656356641, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.022823769881939016, + "before_rms": 0.022823769881939016, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 332, + "input_position": 30, + "layers": [ + { + "after_rms": 0.0064819400327529395, + "before_rms": 0.0064819400327529395, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0066762037147183205, + "before_rms": 0.0066762037147183205, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008751997501954894, + "before_rms": 0.008751997501954894, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.027229842448171573, + "before_rms": 0.027229842448171573, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 269, + "input_position": 31, + "layers": [ + { + "after_rms": 0.0045512880200197945, + "before_rms": 0.0045512880200197945, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005065420297108188, + "before_rms": 0.005065420297108188, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005762789431004507, + "before_rms": 0.005762789431004507, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017768401646140505, + "before_rms": 0.017768401646140505, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1260, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004939956954683621, + "before_rms": 0.004939956954683621, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0038142763724026384, + "before_rms": 0.0038142763724026384, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005913005531372122, + "before_rms": 0.005913005531372122, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01761890031507447, + "before_rms": 0.01761890031507447, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 491, + "input_position": 33, + "layers": [ + { + "after_rms": 0.00603825473034925, + "before_rms": 0.00603825473034925, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005399106143511229, + "before_rms": 0.005399106143511229, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008736535756150335, + "before_rms": 0.008736535756150335, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.023288508898713978, + "before_rms": 0.023288508898713978, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 687, + "input_position": 34, + "layers": [ + { + "after_rms": 0.004825344575484225, + "before_rms": 0.004825344575484225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0035722557633475504, + "before_rms": 0.0035722557633475504, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004528394446715205, + "before_rms": 0.004528394446715205, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015558510785105506, + "before_rms": 0.015558510785105506, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 7370, + "input_position": 35, + "layers": [ + { + "after_rms": 0.00599941128253067, + "before_rms": 0.00599941128253067, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00554379069270193, + "before_rms": 0.00554379069270193, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00600245072054164, + "before_rms": 0.00600245072054164, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021459060251264164, + "before_rms": 0.021459060251264164, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 896, + "input_position": 36, + "layers": [ + { + "after_rms": 0.0051950504054689125, + "before_rms": 0.0051950504054689125, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006141893723174439, + "before_rms": 0.006141893723174439, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009373710872705807, + "before_rms": 0.009373710872705807, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024138161560965755, + "before_rms": 0.024138161560965755, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 9, + "token_id": 24044, + "input_position": 37, + "layers": [ + { + "after_rms": 0.008031516167890848, + "before_rms": 0.008031516167890848, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005368163828714329, + "before_rms": 0.005368163828714329, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005839425814527791, + "before_rms": 0.005839425814527791, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019011043273201134, + "before_rms": 0.019011043273201134, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 10, + "token_id": 295, + "input_position": 38, + "layers": [ + { + "after_rms": 0.009314847401164192, + "before_rms": 0.009314847401164192, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00741898116192189, + "before_rms": 0.00741898116192189, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00969530415712232, + "before_rms": 0.00969530415712232, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02606507768278446, + "before_rms": 0.02606507768278446, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 11, + "token_id": 1815, + "input_position": 39, + "layers": [ + { + "after_rms": 0.0051360181463417694, + "before_rms": 0.0051360181463417694, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005146656362487788, + "before_rms": 0.005146656362487788, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005882866866859857, + "before_rms": 0.005882866866859857, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017485786719118993, + "before_rms": 0.017485786719118993, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 12, + "token_id": 18175, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004435782376176152, + "before_rms": 0.004435782376176152, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007327068248162281, + "before_rms": 0.007327068248162281, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006088986661020862, + "before_rms": 0.006088986661020862, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020237327919821312, + "before_rms": 0.020237327919821312, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 13, + "token_id": 11, + "input_position": 41, + "layers": [ + { + "after_rms": 0.010509592928354338, + "before_rms": 0.010509592928354338, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0072267658458743745, + "before_rms": 0.0072267658458743745, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0072706489196035584, + "before_rms": 0.0072706489196035584, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.026778168430377582, + "before_rms": 0.026778168430377582, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 14, + "token_id": 20827, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0036465031366290737, + "before_rms": 0.0036465031366290737, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003456715901650377, + "before_rms": 0.003456715901650377, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005546723538161339, + "before_rms": 0.005546723538161339, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.014684442530549374, + "before_rms": 0.014684442530549374, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 15, + "token_id": 8437, + "input_position": 43, + "layers": [ + { + "after_rms": 0.00628374835469256, + "before_rms": 0.00628374835469256, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007316398861469263, + "before_rms": 0.007316398861469263, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008040298007985535, + "before_rms": 0.008040298007985535, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021912364803827578, + "before_rms": 0.021912364803827578, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 16, + "token_id": 2488, + "input_position": 44, + "layers": [ + { + "after_rms": 0.008596546950231054, + "before_rms": 0.008596546950231054, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005564570157863948, + "before_rms": 0.005564570157863948, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007757301988850941, + "before_rms": 0.007757301988850941, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020669327020220094, + "before_rms": 0.020669327020220094, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 17, + "token_id": 293, + "input_position": 45, + "layers": [ + { + "after_rms": 0.011432653860628939, + "before_rms": 0.011432653860628939, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005583320073300356, + "before_rms": 0.005583320073300356, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009875868656773605, + "before_rms": 0.009875868656773605, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024941505866581348, + "before_rms": 0.024941505866581348, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 18, + "token_id": 6999, + "input_position": 46, + "layers": [ + { + "after_rms": 0.0053301379768005384, + "before_rms": 0.0053301379768005384, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005000846356165002, + "before_rms": 0.005000846356165002, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005456690270258401, + "before_rms": 0.005456690270258401, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016748723982626335, + "before_rms": 0.016748723982626335, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 19, + "token_id": 981, + "input_position": 47, + "layers": [ + { + "after_rms": 0.007946814942115127, + "before_rms": 0.007946814942115127, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005158417494632955, + "before_rms": 0.005158417494632955, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007460860282340885, + "before_rms": 0.007460860282340885, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020411307476897476, + "before_rms": 0.020411307476897476, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 20, + "token_id": 13, + "input_position": 48, + "layers": [ + { + "after_rms": 0.005514051374901345, + "before_rms": 0.005514051374901345, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0070394921880856, + "before_rms": 0.0070394921880856, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0066148112542871755, + "before_rms": 0.0066148112542871755, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020639906847932543, + "before_rms": 0.020639906847932543, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "flint-q4_1", + "label": "Q4_1", + "model": "Recursive Flint", + "quant": "Q4_1", + "output": "Photosynthesis is the process by which plants use sunlight to create glucose, storing solar energy in chemical form.", + "metrics": { + "generated_tokens": 21, + "prompt_tokens": 29, + "elapsed_seconds": 1.833752451, + "request_wall_seconds": 1.854664323036559, + "time_to_first_token_seconds": 0.6819874360226095, + "decode_tokens_per_second": 17.05229557764334, + "process_peak_rss_kib": 24813804, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 24955171680, + "block12_before_rms": 0.004311247559165804, + "block12_after_rms": 0.004311247559165804 + }, + "check": { + "passed": true, + "rule": "sentence", + "expected": [ + "sunlight", + "glucose" + ], + "character_count": 116, + "word_count": 18 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 49106, + "input_position": 28, + "layers": [ + { + "after_rms": 0.0042605978840198275, + "before_rms": 0.0042605978840198275, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.004311247559165804, + "before_rms": 0.004311247559165804, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.00589800417416362, + "before_rms": 0.00589800417416362, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.015296940228987576, + "before_rms": 0.015296940228987576, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 237568 + } + ] + }, + { + "step": 1, + "token_id": 25424, + "input_position": 29, + "layers": [ + { + "after_rms": 0.005977093382486065, + "before_rms": 0.005977093382486065, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008193012211801604, + "before_rms": 0.008193012211801604, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00683992608315464, + "before_rms": 0.00683992608315464, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.022575839889733613, + "before_rms": 0.022575839889733613, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 332, + "input_position": 30, + "layers": [ + { + "after_rms": 0.0066737027281631835, + "before_rms": 0.0066737027281631835, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006813913551885524, + "before_rms": 0.006813913551885524, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00890446060152158, + "before_rms": 0.00890446060152158, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.027505878131413395, + "before_rms": 0.027505878131413395, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 269, + "input_position": 31, + "layers": [ + { + "after_rms": 0.0044293615064545915, + "before_rms": 0.0044293615064545915, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0051075544884603705, + "before_rms": 0.0051075544884603705, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00575102074067097, + "before_rms": 0.00575102074067097, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017673491619069545, + "before_rms": 0.017673491619069545, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1260, + "input_position": 32, + "layers": [ + { + "after_rms": 0.005036949590614196, + "before_rms": 0.005036949590614196, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003912458062000313, + "before_rms": 0.003912458062000313, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005885844487977124, + "before_rms": 0.005885844487977124, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017499873717491128, + "before_rms": 0.017499873717491128, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 491, + "input_position": 33, + "layers": [ + { + "after_rms": 0.0059711445350189634, + "before_rms": 0.0059711445350189634, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005691547282022589, + "before_rms": 0.005691547282022589, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008772311292878003, + "before_rms": 0.008772311292878003, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.023367520048642263, + "before_rms": 0.023367520048642263, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 687, + "input_position": 34, + "layers": [ + { + "after_rms": 0.00468986110627818, + "before_rms": 0.00468986110627818, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003597916846153238, + "before_rms": 0.003597916846153238, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00449814071189118, + "before_rms": 0.00449814071189118, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015331526438640039, + "before_rms": 0.015331526438640039, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 7370, + "input_position": 35, + "layers": [ + { + "after_rms": 0.006125063657130967, + "before_rms": 0.006125063657130967, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005508366787700634, + "before_rms": 0.005508366787700634, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00612145615591054, + "before_rms": 0.00612145615591054, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021503676321074378, + "before_rms": 0.021503676321074378, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 896, + "input_position": 36, + "layers": [ + { + "after_rms": 0.005212994705122076, + "before_rms": 0.005212994705122076, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006187058312930796, + "before_rms": 0.006187058312930796, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009534886956672891, + "before_rms": 0.009534886956672891, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024496713090488783, + "before_rms": 0.024496713090488783, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 9, + "token_id": 24044, + "input_position": 37, + "layers": [ + { + "after_rms": 0.008439496163631783, + "before_rms": 0.008439496163631783, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005501932939207207, + "before_rms": 0.005501932939207207, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006071353726939792, + "before_rms": 0.006071353726939792, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01955441476435441, + "before_rms": 0.01955441476435441, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 10, + "token_id": 295, + "input_position": 38, + "layers": [ + { + "after_rms": 0.009453040298008773, + "before_rms": 0.009453040298008773, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007468227249751736, + "before_rms": 0.007468227249751736, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009923443718920184, + "before_rms": 0.009923443718920184, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.026507614891845922, + "before_rms": 0.026507614891845922, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 11, + "token_id": 1815, + "input_position": 39, + "layers": [ + { + "after_rms": 0.005096602651817111, + "before_rms": 0.005096602651817111, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00521502443061508, + "before_rms": 0.00521502443061508, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005931677256268344, + "before_rms": 0.005931677256268344, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017564191438152, + "before_rms": 0.017564191438152, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 12, + "token_id": 18175, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004618430168805286, + "before_rms": 0.004618430168805286, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007451683719624791, + "before_rms": 0.007451683719624791, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006378351819867548, + "before_rms": 0.006378351819867548, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02045012871478634, + "before_rms": 0.02045012871478634, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 13, + "token_id": 11, + "input_position": 41, + "layers": [ + { + "after_rms": 0.010203299320963541, + "before_rms": 0.010203299320963541, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0074903907754410755, + "before_rms": 0.0074903907754410755, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0074366114309835155, + "before_rms": 0.0074366114309835155, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02642472904218289, + "before_rms": 0.02642472904218289, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 14, + "token_id": 20827, + "input_position": 42, + "layers": [ + { + "after_rms": 0.003682021169442006, + "before_rms": 0.003682021169442006, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003522532163645522, + "before_rms": 0.003522532163645522, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005599300959517324, + "before_rms": 0.005599300959517324, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.014870400340011209, + "before_rms": 0.014870400340011209, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 15, + "token_id": 8437, + "input_position": 43, + "layers": [ + { + "after_rms": 0.006406295163748658, + "before_rms": 0.006406295163748658, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007487654139117973, + "before_rms": 0.007487654139117973, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008307528862641168, + "before_rms": 0.008307528862641168, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02216700022530452, + "before_rms": 0.02216700022530452, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 16, + "token_id": 2488, + "input_position": 44, + "layers": [ + { + "after_rms": 0.009338543902993778, + "before_rms": 0.009338543902993778, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005618921543998528, + "before_rms": 0.005618921543998528, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007794415755047646, + "before_rms": 0.007794415755047646, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021055979028721958, + "before_rms": 0.021055979028721958, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 17, + "token_id": 293, + "input_position": 45, + "layers": [ + { + "after_rms": 0.012090981045666482, + "before_rms": 0.012090981045666482, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005692148533663447, + "before_rms": 0.005692148533663447, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009866038408325634, + "before_rms": 0.009866038408325634, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02533462806548674, + "before_rms": 0.02533462806548674, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 18, + "token_id": 6999, + "input_position": 46, + "layers": [ + { + "after_rms": 0.0053002691790724555, + "before_rms": 0.0053002691790724555, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005049569138548446, + "before_rms": 0.005049569138548446, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0056289488702639424, + "before_rms": 0.0056289488702639424, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016855422564220696, + "before_rms": 0.016855422564220696, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 19, + "token_id": 981, + "input_position": 47, + "layers": [ + { + "after_rms": 0.007835640918744254, + "before_rms": 0.007835640918744254, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005132931328426137, + "before_rms": 0.005132931328426137, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008011038840819223, + "before_rms": 0.008011038840819223, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02066205167657649, + "before_rms": 0.02066205167657649, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 20, + "token_id": 13, + "input_position": 48, + "layers": [ + { + "after_rms": 0.005366822874419904, + "before_rms": 0.005366822874419904, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0069233652002111, + "before_rms": 0.0069233652002111, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006798509328318403, + "before_rms": 0.006798509328318403, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020444238619442973, + "before_rms": 0.020444238619442973, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + } + ], + "model": "flint" + }, + { + "id": "qwen-arithmetic", + "title": "Exact arithmetic", + "prompt": "Calculate (17 * 23) - 19. Reply with the integer only.", + "checks": [ + "Rule: integer", + "Expected: 372", + "Q8_0: failed", + "Q4_1: failed" + ], + "conditions": [ + { + "id": "qwen-q8_0", + "label": "Q8_0", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "output": "362", + "metrics": { + "generated_tokens": 3, + "prompt_tokens": 50, + "elapsed_seconds": 6.370927408, + "request_wall_seconds": 6.398233153973706, + "time_to_first_token_seconds": 5.195338613004424, + "decode_tokens_per_second": 1.6625734938534744, + "process_peak_rss_kib": 28517200, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 28595763712, + "block12_before_rms": 0.060774775703729224, + "block12_after_rms": 0.060774775703729224 + }, + "check": { + "passed": false, + "rule": "integer", + "expected": "372", + "character_count": 3, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 18, + "input_position": 49, + "layers": [ + { + "after_rms": 0.051227998654021174, + "before_rms": 0.051227998654021174, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1024000 + }, + { + "after_rms": 0.060774775703729224, + "before_rms": 0.060774775703729224, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1024000 + }, + { + "after_rms": 0.05656805709715899, + "before_rms": 0.05656805709715899, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1024000 + }, + { + "after_rms": 0.6482024672223894, + "before_rms": 0.6482024672223894, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1024000 + } + ] + }, + { + "step": 1, + "token_id": 21, + "input_position": 50, + "layers": [ + { + "after_rms": 0.04010414688700296, + "before_rms": 0.04010414688700296, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.047943996526761117, + "before_rms": 0.047943996526761117, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.059109413227162184, + "before_rms": 0.059109413227162184, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6682861418633491, + "before_rms": 0.6682861418633491, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 17, + "input_position": 51, + "layers": [ + { + "after_rms": 0.041057583688967295, + "before_rms": 0.041057583688967295, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04121733317261862, + "before_rms": 0.04121733317261862, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.047359839621201105, + "before_rms": 0.047359839621201105, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8032033925827468, + "before_rms": 0.8032033925827468, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + }, + { + "id": "qwen-q4_1", + "label": "Q4_1", + "model": "alphaBell Qwen Open 27B", + "quant": "Q4_1", + "output": "362", + "metrics": { + "generated_tokens": 3, + "prompt_tokens": 50, + "elapsed_seconds": 9.60077816, + "request_wall_seconds": 9.647026313003153, + "time_to_first_token_seconds": 8.771988685010001, + "decode_tokens_per_second": 2.285481485264205, + "process_peak_rss_kib": 17027080, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 16829921792, + "block12_before_rms": 0.0627527559695309, + "block12_after_rms": 0.0627527559695309 + }, + "check": { + "passed": false, + "rule": "integer", + "expected": "372", + "character_count": 3, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 18, + "input_position": 49, + "layers": [ + { + "after_rms": 0.05048796369798099, + "before_rms": 0.05048796369798099, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1024000 + }, + { + "after_rms": 0.0627527559695309, + "before_rms": 0.0627527559695309, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1024000 + }, + { + "after_rms": 0.05658700984894215, + "before_rms": 0.05658700984894215, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1024000 + }, + { + "after_rms": 0.6455786601758424, + "before_rms": 0.6455786601758424, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1024000 + } + ] + }, + { + "step": 1, + "token_id": 21, + "input_position": 50, + "layers": [ + { + "after_rms": 0.03930109211337971, + "before_rms": 0.03930109211337971, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.049347960526798046, + "before_rms": 0.049347960526798046, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06041073044793531, + "before_rms": 0.06041073044793531, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6694457398914584, + "before_rms": 0.6694457398914584, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 17, + "input_position": 51, + "layers": [ + { + "after_rms": 0.039943780185542364, + "before_rms": 0.039943780185542364, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.040380369555943325, + "before_rms": 0.040380369555943325, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04655456958881338, + "before_rms": 0.04655456958881338, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.806759676095322, + "before_rms": 0.806759676095322, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + } + ], + "model": "qwen" + }, + { + "id": "qwen-json", + "title": "Strict JSON format", + "prompt": "Return exactly this JSON object and no other text: {\"status\":\"ok\",\"count\":3}", + "checks": [ + "Rule: json", + "Expected: {'status': 'ok', 'count': 3}", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "qwen-q8_0", + "label": "Q8_0", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "output": "{\"status\":\"ok\",\"count\":3}", + "metrics": { + "generated_tokens": 9, + "prompt_tokens": 49, + "elapsed_seconds": 8.587899932, + "request_wall_seconds": 8.640126045094803, + "time_to_first_token_seconds": 5.242650785017759, + "decode_tokens_per_second": 2.3546587753888293, + "process_peak_rss_kib": 28517200, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 28595763712, + "block12_before_rms": 0.06654119101326558, + "block12_after_rms": 0.06654119101326558 + }, + "check": { + "passed": true, + "rule": "json", + "expected": { + "status": "ok", + "count": 3 + }, + "character_count": 25, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 4754, + "input_position": 48, + "layers": [ + { + "after_rms": 0.052291073879200234, + "before_rms": 0.052291073879200234, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1003520 + }, + { + "after_rms": 0.06654119101326558, + "before_rms": 0.06654119101326558, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1003520 + }, + { + "after_rms": 0.05664182123474859, + "before_rms": 0.05664182123474859, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1003520 + }, + { + "after_rms": 0.6845949822068217, + "before_rms": 0.6845949822068217, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1003520 + } + ] + }, + { + "step": 1, + "token_id": 2738, + "input_position": 49, + "layers": [ + { + "after_rms": 0.03855443169110616, + "before_rms": 0.03855443169110616, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.047620845341347987, + "before_rms": 0.047620845341347987, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04753896437018176, + "before_rms": 0.04753896437018176, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7466266976354985, + "before_rms": 0.7466266976354985, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 3147, + "input_position": 50, + "layers": [ + { + "after_rms": 0.038951229722960494, + "before_rms": 0.038951229722960494, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03903383968014138, + "before_rms": 0.03903383968014138, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.040843553885173405, + "before_rms": 0.040843553885173405, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9274036435751518, + "before_rms": 0.9274036435751518, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 3, + "token_id": 547, + "input_position": 51, + "layers": [ + { + "after_rms": 0.028340486221480752, + "before_rms": 0.028340486221480752, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.031217358447974707, + "before_rms": 0.031217358447974707, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03351828683553298, + "before_rms": 0.03351828683553298, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.696227792238104, + "before_rms": 0.696227792238104, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 4, + "token_id": 2129, + "input_position": 52, + "layers": [ + { + "after_rms": 0.039811305925765116, + "before_rms": 0.039811305925765116, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04377612280244084, + "before_rms": 0.04377612280244084, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04746907424551777, + "before_rms": 0.04746907424551777, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8157278441144455, + "before_rms": 0.8157278441144455, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 5, + "token_id": 1767, + "input_position": 53, + "layers": [ + { + "after_rms": 0.032082000343200995, + "before_rms": 0.032082000343200995, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03318282436223156, + "before_rms": 0.03318282436223156, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.033619616830206873, + "before_rms": 0.033619616830206873, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6970969978843475, + "before_rms": 0.6970969978843475, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 6, + "token_id": 763, + "input_position": 54, + "layers": [ + { + "after_rms": 0.03466834851881882, + "before_rms": 0.03466834851881882, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03830034553467172, + "before_rms": 0.03830034553467172, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03505207487460683, + "before_rms": 0.03505207487460683, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7499139970375372, + "before_rms": 0.7499139970375372, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 7, + "token_id": 18, + "input_position": 55, + "layers": [ + { + "after_rms": 0.02138759797992662, + "before_rms": 0.02138759797992662, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.024774094253474845, + "before_rms": 0.024774094253474845, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.02468320711787904, + "before_rms": 0.02468320711787904, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7000268392851855, + "before_rms": 0.7000268392851855, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 8, + "token_id": 92, + "input_position": 56, + "layers": [ + { + "after_rms": 0.03105064736729475, + "before_rms": 0.03105064736729475, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03308346918008162, + "before_rms": 0.03308346918008162, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0373820159022643, + "before_rms": 0.0373820159022643, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6878214874370955, + "before_rms": 0.6878214874370955, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + }, + { + "id": "qwen-q4_1", + "label": "Q4_1", + "model": "alphaBell Qwen Open 27B", + "quant": "Q4_1", + "output": "{\"status\":\"ok\",\"count\":3}", + "metrics": { + "generated_tokens": 9, + "prompt_tokens": 49, + "elapsed_seconds": 11.27109005, + "request_wall_seconds": 11.301758665940724, + "time_to_first_token_seconds": 8.48897207097616, + "decode_tokens_per_second": 2.8440503495699967, + "process_peak_rss_kib": 17027080, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 16829921792, + "block12_before_rms": 0.06706882773636201, + "block12_after_rms": 0.06706882773636201 + }, + "check": { + "passed": true, + "rule": "json", + "expected": { + "status": "ok", + "count": 3 + }, + "character_count": 25, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 4754, + "input_position": 48, + "layers": [ + { + "after_rms": 0.05224029844078737, + "before_rms": 0.05224029844078737, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1003520 + }, + { + "after_rms": 0.06706882773636201, + "before_rms": 0.06706882773636201, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1003520 + }, + { + "after_rms": 0.056588501864807024, + "before_rms": 0.056588501864807024, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1003520 + }, + { + "after_rms": 0.6822709962112733, + "before_rms": 0.6822709962112733, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1003520 + } + ] + }, + { + "step": 1, + "token_id": 2738, + "input_position": 49, + "layers": [ + { + "after_rms": 0.03752622482634915, + "before_rms": 0.03752622482634915, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.046903273237478865, + "before_rms": 0.046903273237478865, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04722570539635123, + "before_rms": 0.04722570539635123, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7377205249987369, + "before_rms": 0.7377205249987369, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 3147, + "input_position": 50, + "layers": [ + { + "after_rms": 0.0383644658257016, + "before_rms": 0.0383644658257016, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03938200710595041, + "before_rms": 0.03938200710595041, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.040526837904161464, + "before_rms": 0.040526837904161464, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9237316979852525, + "before_rms": 0.9237316979852525, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 3, + "token_id": 547, + "input_position": 51, + "layers": [ + { + "after_rms": 0.02778849024723728, + "before_rms": 0.02778849024723728, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03214516611442832, + "before_rms": 0.03214516611442832, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03363793762611696, + "before_rms": 0.03363793762611696, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6944275380038829, + "before_rms": 0.6944275380038829, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 4, + "token_id": 2129, + "input_position": 52, + "layers": [ + { + "after_rms": 0.039622563175138806, + "before_rms": 0.039622563175138806, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.044796938258385224, + "before_rms": 0.044796938258385224, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.048279328936419154, + "before_rms": 0.048279328936419154, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8158488049148617, + "before_rms": 0.8158488049148617, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 5, + "token_id": 1767, + "input_position": 53, + "layers": [ + { + "after_rms": 0.03299336016454038, + "before_rms": 0.03299336016454038, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.035306595030275154, + "before_rms": 0.035306595030275154, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.034968922773907535, + "before_rms": 0.034968922773907535, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7012426405133863, + "before_rms": 0.7012426405133863, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 6, + "token_id": 763, + "input_position": 54, + "layers": [ + { + "after_rms": 0.03539100194372584, + "before_rms": 0.03539100194372584, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03888318460173007, + "before_rms": 0.03888318460173007, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.036111392321474096, + "before_rms": 0.036111392321474096, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7510159984696372, + "before_rms": 0.7510159984696372, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 7, + "token_id": 18, + "input_position": 55, + "layers": [ + { + "after_rms": 0.021069990443326445, + "before_rms": 0.021069990443326445, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.02628960155652075, + "before_rms": 0.02628960155652075, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.02576443542713521, + "before_rms": 0.02576443542713521, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6989044785673804, + "before_rms": 0.6989044785673804, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 8, + "token_id": 92, + "input_position": 56, + "layers": [ + { + "after_rms": 0.03149593114335555, + "before_rms": 0.03149593114335555, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.033202658246637436, + "before_rms": 0.033202658246637436, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.037501340133310815, + "before_rms": 0.037501340133310815, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6955924601927745, + "before_rms": 0.6955924601927745, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + } + ], + "model": "qwen" + }, + { + "id": "qwen-fact", + "title": "Short factual recall", + "prompt": "What is the chemical symbol for gold? Reply with the symbol only.", + "checks": [ + "Rule: exact", + "Expected: Au", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "qwen-q8_0", + "label": "Q8_0", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "output": "Au", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 44, + "elapsed_seconds": 4.642115172, + "request_wall_seconds": 4.671560399001464, + "time_to_first_token_seconds": 4.299511207034811, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 28517200, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 28595763712, + "block12_before_rms": 0.07549879978282885, + "block12_after_rms": 0.07549879978282885 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "Au", + "character_count": 2, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 63126, + "input_position": 43, + "layers": [ + { + "after_rms": 0.0577547722697427, + "before_rms": 0.0577547722697427, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 901120 + }, + { + "after_rms": 0.07549879978282885, + "before_rms": 0.07549879978282885, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 901120 + }, + { + "after_rms": 0.0707093987239072, + "before_rms": 0.0707093987239072, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 901120 + }, + { + "after_rms": 0.6240880098652177, + "before_rms": 0.6240880098652177, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 901120 + } + ] + } + ] + }, + { + "id": "qwen-q4_1", + "label": "Q4_1", + "model": "alphaBell Qwen Open 27B", + "quant": "Q4_1", + "output": "Au", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 44, + "elapsed_seconds": 7.763876279, + "request_wall_seconds": 7.807731770095415, + "time_to_first_token_seconds": 7.462882330059074, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 17027080, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 16829921792, + "block12_before_rms": 0.07777038091284097, + "block12_after_rms": 0.07777038091284097 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "Au", + "character_count": 2, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 63126, + "input_position": 43, + "layers": [ + { + "after_rms": 0.058337225299247154, + "before_rms": 0.058337225299247154, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 901120 + }, + { + "after_rms": 0.07777038091284097, + "before_rms": 0.07777038091284097, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 901120 + }, + { + "after_rms": 0.07119122784735792, + "before_rms": 0.07119122784735792, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 901120 + }, + { + "after_rms": 0.627043092250394, + "before_rms": 0.627043092250394, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 901120 + } + ] + } + ] + } + ], + "model": "qwen" + }, + { + "id": "qwen-correction", + "title": "Use the corrected fact", + "prompt": "The access color was blue. Correction: the access color is now amber. What is the current access color? Reply with one word only.", + "checks": [ + "Rule: exact", + "Expected: amber", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "qwen-q8_0", + "label": "Q8_0", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "output": "Amber", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 58, + "elapsed_seconds": 6.922008574, + "request_wall_seconds": 6.929079352994449, + "time_to_first_token_seconds": 6.135061308043078, + "decode_tokens_per_second": 1.2593577760195043, + "process_peak_rss_kib": 28517200, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 28595763712, + "block12_before_rms": 0.08439747530644362, + "block12_after_rms": 0.08439747530644362 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "amber", + "character_count": 5, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 5901, + "input_position": 57, + "layers": [ + { + "after_rms": 0.05835882673381115, + "before_rms": 0.05835882673381115, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1187840 + }, + { + "after_rms": 0.08439747530644362, + "before_rms": 0.08439747530644362, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1187840 + }, + { + "after_rms": 0.07025392596556453, + "before_rms": 0.07025392596556453, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1187840 + }, + { + "after_rms": 0.6934357017487144, + "before_rms": 0.6934357017487144, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1187840 + } + ] + }, + { + "step": 1, + "token_id": 634, + "input_position": 58, + "layers": [ + { + "after_rms": 0.059411132201425514, + "before_rms": 0.059411132201425514, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07671841412821269, + "before_rms": 0.07671841412821269, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08880618504780236, + "before_rms": 0.08880618504780236, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8700114799969301, + "before_rms": 0.8700114799969301, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + }, + { + "id": "qwen-q4_1", + "label": "Q4_1", + "model": "alphaBell Qwen Open 27B", + "quant": "Q4_1", + "output": "Amber", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 58, + "elapsed_seconds": 10.122600283, + "request_wall_seconds": 10.232366479933262, + "time_to_first_token_seconds": 9.576763171004131, + "decode_tokens_per_second": 1.5252099559031584, + "process_peak_rss_kib": 17029128, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 16829921792, + "block12_before_rms": 0.08882206964440575, + "block12_after_rms": 0.08882206964440575 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "amber", + "character_count": 5, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 5901, + "input_position": 57, + "layers": [ + { + "after_rms": 0.05771725253557467, + "before_rms": 0.05771725253557467, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1187840 + }, + { + "after_rms": 0.08882206964440575, + "before_rms": 0.08882206964440575, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1187840 + }, + { + "after_rms": 0.07251657251443638, + "before_rms": 0.07251657251443638, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1187840 + }, + { + "after_rms": 0.694176497002211, + "before_rms": 0.694176497002211, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1187840 + } + ] + }, + { + "step": 1, + "token_id": 634, + "input_position": 58, + "layers": [ + { + "after_rms": 0.06058345247387532, + "before_rms": 0.06058345247387532, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07852469801096601, + "before_rms": 0.07852469801096601, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0889624477213204, + "before_rms": 0.0889624477213204, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8659660560094563, + "before_rms": 0.8659660560094563, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + } + ], + "model": "qwen" + }, + { + "id": "qwen-code", + "title": "Python expression repair", + "prompt": "Complete the function with one Python expression only, no markdown: def is_even(n): return ___ . The function must return True exactly when integer n is even.", + "checks": [ + "Rule: expression", + "Expected: n % 2 == 0", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "qwen-q8_0", + "label": "Q8_0", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "output": "n % 2 == 0", + "metrics": { + "generated_tokens": 7, + "prompt_tokens": 62, + "elapsed_seconds": 9.254872529, + "request_wall_seconds": 9.284579382976517, + "time_to_first_token_seconds": 6.607404271955602, + "decode_tokens_per_second": 2.2411405168871537, + "process_peak_rss_kib": 28517200, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 28595763712, + "block12_before_rms": 0.05339986668971442, + "block12_after_rms": 0.05339986668971442 + }, + "check": { + "passed": true, + "rule": "expression", + "expected": "n % 2 == 0", + "character_count": 10, + "word_count": 5 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 77, + "input_position": 61, + "layers": [ + { + "after_rms": 0.050335302931553125, + "before_rms": 0.050335302931553125, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1269760 + }, + { + "after_rms": 0.05339986668971442, + "before_rms": 0.05339986668971442, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1269760 + }, + { + "after_rms": 0.050829254488739616, + "before_rms": 0.050829254488739616, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1269760 + }, + { + "after_rms": 0.6219835311460259, + "before_rms": 0.6219835311460259, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1269760 + } + ] + }, + { + "step": 1, + "token_id": 984, + "input_position": 62, + "layers": [ + { + "after_rms": 0.053686741784584366, + "before_rms": 0.053686741784584366, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.060073512071431336, + "before_rms": 0.060073512071431336, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07386188586159291, + "before_rms": 0.07386188586159291, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7373913972139406, + "before_rms": 0.7373913972139406, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 220, + "input_position": 63, + "layers": [ + { + "after_rms": 0.0757632347354875, + "before_rms": 0.0757632347354875, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08447271653317262, + "before_rms": 0.08447271653317262, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08083366248253042, + "before_rms": 0.08083366248253042, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 1.0242087065193417, + "before_rms": 1.0242087065193417, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 3, + "token_id": 17, + "input_position": 64, + "layers": [ + { + "after_rms": 0.031847120231657235, + "before_rms": 0.031847120231657235, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.034744065101999524, + "before_rms": 0.034744065101999524, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0356676406493493, + "before_rms": 0.0356676406493493, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7133838975226968, + "before_rms": 0.7133838975226968, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 4, + "token_id": 606, + "input_position": 65, + "layers": [ + { + "after_rms": 0.06511419656269898, + "before_rms": 0.06511419656269898, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06654091916956285, + "before_rms": 0.06654091916956285, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0625665999000719, + "before_rms": 0.0625665999000719, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9783984307722204, + "before_rms": 0.9783984307722204, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 5, + "token_id": 220, + "input_position": 66, + "layers": [ + { + "after_rms": 0.0645209284738379, + "before_rms": 0.0645209284738379, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06279313344255302, + "before_rms": 0.06279313344255302, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07065568848215707, + "before_rms": 0.07065568848215707, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8914932839543546, + "before_rms": 0.8914932839543546, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 6, + "token_id": 15, + "input_position": 67, + "layers": [ + { + "after_rms": 0.01992336177071922, + "before_rms": 0.01992336177071922, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.01935371501039362, + "before_rms": 0.01935371501039362, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.02030963950506069, + "before_rms": 0.02030963950506069, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.5854232238481374, + "before_rms": 0.5854232238481374, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + }, + { + "id": "qwen-q4_1", + "label": "Q4_1", + "model": "alphaBell Qwen Open 27B", + "quant": "Q4_1", + "output": "n % 2 == 0", + "metrics": { + "generated_tokens": 7, + "prompt_tokens": 62, + "elapsed_seconds": 12.254965777, + "request_wall_seconds": 12.296226173988543, + "time_to_first_token_seconds": 10.16412831703201, + "decode_tokens_per_second": 2.8140651531387326, + "process_peak_rss_kib": 17029128, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 16829921792, + "block12_before_rms": 0.05348751558536074, + "block12_after_rms": 0.05348751558536074 + }, + "check": { + "passed": true, + "rule": "expression", + "expected": "n % 2 == 0", + "character_count": 10, + "word_count": 5 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 77, + "input_position": 61, + "layers": [ + { + "after_rms": 0.05019944799383319, + "before_rms": 0.05019944799383319, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1269760 + }, + { + "after_rms": 0.05348751558536074, + "before_rms": 0.05348751558536074, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1269760 + }, + { + "after_rms": 0.05175455397380355, + "before_rms": 0.05175455397380355, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 1269760 + }, + { + "after_rms": 0.6209295945416631, + "before_rms": 0.6209295945416631, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 1269760 + } + ] + }, + { + "step": 1, + "token_id": 984, + "input_position": 62, + "layers": [ + { + "after_rms": 0.05431832143619491, + "before_rms": 0.05431832143619491, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0596135618336448, + "before_rms": 0.0596135618336448, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07570250992978683, + "before_rms": 0.07570250992978683, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7405282278687231, + "before_rms": 0.7405282278687231, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 220, + "input_position": 63, + "layers": [ + { + "after_rms": 0.07608705520961619, + "before_rms": 0.07608705520961619, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08613149135695485, + "before_rms": 0.08613149135695485, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08167221964365984, + "before_rms": 0.08167221964365984, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 1.0218643697263319, + "before_rms": 1.0218643697263319, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 3, + "token_id": 17, + "input_position": 64, + "layers": [ + { + "after_rms": 0.03293924682560806, + "before_rms": 0.03293924682560806, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03573770767293814, + "before_rms": 0.03573770767293814, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.03788883557000408, + "before_rms": 0.03788883557000408, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7037175496589, + "before_rms": 0.7037175496589, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 4, + "token_id": 606, + "input_position": 65, + "layers": [ + { + "after_rms": 0.06427408980785317, + "before_rms": 0.06427408980785317, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0662650177596904, + "before_rms": 0.0662650177596904, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06341284549994454, + "before_rms": 0.06341284549994454, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9659503607467934, + "before_rms": 0.9659503607467934, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 5, + "token_id": 220, + "input_position": 66, + "layers": [ + { + "after_rms": 0.06468317950504401, + "before_rms": 0.06468317950504401, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06446495337470849, + "before_rms": 0.06446495337470849, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07169159170100954, + "before_rms": 0.07169159170100954, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8967088457352965, + "before_rms": 0.8967088457352965, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 6, + "token_id": 15, + "input_position": 67, + "layers": [ + { + "after_rms": 0.01985784365149535, + "before_rms": 0.01985784365149535, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.020029877274204144, + "before_rms": 0.020029877274204144, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.020914043695738526, + "before_rms": 0.020914043695738526, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.5841152795565343, + "before_rms": 0.5841152795565343, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + } + ], + "model": "qwen" + }, + { + "id": "qwen-brief", + "title": "One sentence explanation", + "prompt": "Explain photosynthesis in exactly one sentence containing the words sunlight and glucose.", + "checks": [ + "Rule: sentence", + "Expected: ['sunlight', 'glucose']", + "Q8_0: passed", + "Q4_1: passed" + ], + "conditions": [ + { + "id": "qwen-q8_0", + "label": "Q8_0", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "output": "Photosynthesis is the process by which plants use sunlight to convert carbon dioxide and water into glucose and oxygen.", + "metrics": { + "generated_tokens": 21, + "prompt_tokens": 45, + "elapsed_seconds": 12.680948171, + "request_wall_seconds": 12.710689318017103, + "time_to_first_token_seconds": 4.477205132017843, + "decode_tokens_per_second": 2.4290935490085785, + "process_peak_rss_kib": 28517200, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 28595763712, + "block12_before_rms": 0.08826706557870591, + "block12_after_rms": 0.08826706557870591 + }, + "check": { + "passed": true, + "rule": "sentence", + "expected": [ + "sunlight", + "glucose" + ], + "character_count": 119, + "word_count": 19 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 30729, + "input_position": 44, + "layers": [ + { + "after_rms": 0.058797602262198566, + "before_rms": 0.058797602262198566, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 921600 + }, + { + "after_rms": 0.08826706557870591, + "before_rms": 0.08826706557870591, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 921600 + }, + { + "after_rms": 0.07744769492165124, + "before_rms": 0.07744769492165124, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 921600 + }, + { + "after_rms": 0.6493664827145925, + "before_rms": 0.6493664827145925, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 921600 + } + ] + }, + { + "step": 1, + "token_id": 71163, + "input_position": 45, + "layers": [ + { + "after_rms": 0.090079163721148, + "before_rms": 0.090079163721148, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09129523578154757, + "before_rms": 0.09129523578154757, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0829387283770895, + "before_rms": 0.0829387283770895, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9874844483140827, + "before_rms": 0.9874844483140827, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 369, + "input_position": 46, + "layers": [ + { + "after_rms": 0.10315392514524502, + "before_rms": 0.10315392514524502, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09600644261008953, + "before_rms": 0.09600644261008953, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0857152751972423, + "before_rms": 0.0857152751972423, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 1.1279006781817724, + "before_rms": 1.1279006781817724, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 3, + "token_id": 279, + "input_position": 47, + "layers": [ + { + "after_rms": 0.0703105570552835, + "before_rms": 0.0703105570552835, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08696065973360559, + "before_rms": 0.08696065973360559, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08599798105761434, + "before_rms": 0.08599798105761434, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7075713993722996, + "before_rms": 0.7075713993722996, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 4, + "token_id": 1817, + "input_position": 48, + "layers": [ + { + "after_rms": 0.04824121607819701, + "before_rms": 0.04824121607819701, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.053480497826960974, + "before_rms": 0.053480497826960974, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06288983601150724, + "before_rms": 0.06288983601150724, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6498592562479432, + "before_rms": 0.6498592562479432, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 5, + "token_id": 539, + "input_position": 49, + "layers": [ + { + "after_rms": 0.0857769249129497, + "before_rms": 0.0857769249129497, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09649908523834096, + "before_rms": 0.09649908523834096, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.10047606095155863, + "before_rms": 0.10047606095155863, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8915207377991599, + "before_rms": 0.8915207377991599, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 6, + "token_id": 864, + "input_position": 50, + "layers": [ + { + "after_rms": 0.08697706879734413, + "before_rms": 0.08697706879734413, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07727010638748308, + "before_rms": 0.07727010638748308, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09060006959943864, + "before_rms": 0.09060006959943864, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7683805849623969, + "before_rms": 0.7683805849623969, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 7, + "token_id": 10462, + "input_position": 51, + "layers": [ + { + "after_rms": 0.05259421797317353, + "before_rms": 0.05259421797317353, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.05075043909793059, + "before_rms": 0.05075043909793059, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.05619568725069516, + "before_rms": 0.05619568725069516, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.682036783205715, + "before_rms": 0.682036783205715, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 8, + "token_id": 958, + "input_position": 52, + "layers": [ + { + "after_rms": 0.08697357786711173, + "before_rms": 0.08697357786711173, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.10030394478249688, + "before_rms": 0.10030394478249688, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08877069591266067, + "before_rms": 0.08877069591266067, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9646197687285322, + "before_rms": 0.9646197687285322, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 9, + "token_id": 37728, + "input_position": 53, + "layers": [ + { + "after_rms": 0.07099648705814063, + "before_rms": 0.07099648705814063, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06559096588235168, + "before_rms": 0.06559096588235168, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07033321001871201, + "before_rms": 0.07033321001871201, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7932168808370963, + "before_rms": 0.7932168808370963, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 10, + "token_id": 310, + "input_position": 54, + "layers": [ + { + "after_rms": 0.08567177205503797, + "before_rms": 0.08567177205503797, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08384291426005887, + "before_rms": 0.08384291426005887, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08019719454528439, + "before_rms": 0.08019719454528439, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9847601168939447, + "before_rms": 0.9847601168939447, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 11, + "token_id": 5335, + "input_position": 55, + "layers": [ + { + "after_rms": 0.043123201419361115, + "before_rms": 0.043123201419361115, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04955096633139602, + "before_rms": 0.04955096633139602, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.053935569085920204, + "before_rms": 0.053935569085920204, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7026575509832893, + "before_rms": 0.7026575509832893, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 12, + "token_id": 12141, + "input_position": 56, + "layers": [ + { + "after_rms": 0.07978349390889998, + "before_rms": 0.07978349390889998, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07617391418880347, + "before_rms": 0.07617391418880347, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07734026858151206, + "before_rms": 0.07734026858151206, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8642118185671083, + "before_rms": 0.8642118185671083, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 13, + "token_id": 38182, + "input_position": 57, + "layers": [ + { + "after_rms": 0.0852385224689407, + "before_rms": 0.0852385224689407, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07231971038626593, + "before_rms": 0.07231971038626593, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06564426590463994, + "before_rms": 0.06564426590463994, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9157673634228397, + "before_rms": 0.9157673634228397, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 14, + "token_id": 321, + "input_position": 58, + "layers": [ + { + "after_rms": 0.08898192682211213, + "before_rms": 0.08898192682211213, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0894750809473178, + "before_rms": 0.0894750809473178, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08616370851207737, + "before_rms": 0.08616370851207737, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 1.0029595110641643, + "before_rms": 1.0029595110641643, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 15, + "token_id": 2919, + "input_position": 59, + "layers": [ + { + "after_rms": 0.03762191629076314, + "before_rms": 0.03762191629076314, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.049029101729874604, + "before_rms": 0.049029101729874604, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.05884392489297117, + "before_rms": 0.05884392489297117, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6250386128828833, + "before_rms": 0.6250386128828833, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 16, + "token_id": 1083, + "input_position": 60, + "layers": [ + { + "after_rms": 0.06327166050315482, + "before_rms": 0.06327166050315482, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06857836855297815, + "before_rms": 0.06857836855297815, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06961224727562185, + "before_rms": 0.06961224727562185, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8674029708685703, + "before_rms": 0.8674029708685703, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 17, + "token_id": 32120, + "input_position": 61, + "layers": [ + { + "after_rms": 0.0530895463732545, + "before_rms": 0.0530895463732545, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06423360109914558, + "before_rms": 0.06423360109914558, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.052746699956901094, + "before_rms": 0.052746699956901094, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7559156680420246, + "before_rms": 0.7559156680420246, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 18, + "token_id": 321, + "input_position": 62, + "layers": [ + { + "after_rms": 0.07834838045994791, + "before_rms": 0.07834838045994791, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07525093120253516, + "before_rms": 0.07525093120253516, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07091507308763366, + "before_rms": 0.07091507308763366, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.954111191683783, + "before_rms": 0.954111191683783, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 19, + "token_id": 22817, + "input_position": 63, + "layers": [ + { + "after_rms": 0.0353542432718733, + "before_rms": 0.0353542432718733, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04198844081291262, + "before_rms": 0.04198844081291262, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.054421407751238354, + "before_rms": 0.054421407751238354, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.629625149193875, + "before_rms": 0.629625149193875, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 20, + "token_id": 13, + "input_position": 64, + "layers": [ + { + "after_rms": 0.07315881109901357, + "before_rms": 0.07315881109901357, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0687687379925814, + "before_rms": 0.0687687379925814, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06567824611420153, + "before_rms": 0.06567824611420153, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8984019945181603, + "before_rms": 0.8984019945181603, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + }, + { + "id": "qwen-q4_1", + "label": "Q4_1", + "model": "alphaBell Qwen Open 27B", + "quant": "Q4_1", + "output": "Photosynthesis is the process by which plants use sunlight to convert carbon dioxide and water into glucose and oxygen.", + "metrics": { + "generated_tokens": 21, + "prompt_tokens": 45, + "elapsed_seconds": 13.409067066, + "request_wall_seconds": 13.439180701971054, + "time_to_first_token_seconds": 7.273399753961712, + "decode_tokens_per_second": 3.243670129869688, + "process_peak_rss_kib": 17029128, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 16829921792, + "block12_before_rms": 0.09098383032221528, + "block12_after_rms": 0.09098383032221528 + }, + "check": { + "passed": true, + "rule": "sentence", + "expected": [ + "sunlight", + "glucose" + ], + "character_count": 119, + "word_count": 19 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 30729, + "input_position": 44, + "layers": [ + { + "after_rms": 0.059582062101886914, + "before_rms": 0.059582062101886914, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 921600 + }, + { + "after_rms": 0.09098383032221528, + "before_rms": 0.09098383032221528, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 921600 + }, + { + "after_rms": 0.08016257842928678, + "before_rms": 0.08016257842928678, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 921600 + }, + { + "after_rms": 0.6536671790903104, + "before_rms": 0.6536671790903104, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 921600 + } + ] + }, + { + "step": 1, + "token_id": 71163, + "input_position": 45, + "layers": [ + { + "after_rms": 0.09190790143581268, + "before_rms": 0.09190790143581268, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09553453447497137, + "before_rms": 0.09553453447497137, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08593551701844118, + "before_rms": 0.08593551701844118, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9966251527987956, + "before_rms": 0.9966251527987956, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 2, + "token_id": 369, + "input_position": 46, + "layers": [ + { + "after_rms": 0.10115326211799784, + "before_rms": 0.10115326211799784, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0958071133672348, + "before_rms": 0.0958071133672348, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0854668936412898, + "before_rms": 0.0854668936412898, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 1.1255902406232912, + "before_rms": 1.1255902406232912, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 3, + "token_id": 279, + "input_position": 47, + "layers": [ + { + "after_rms": 0.07451807797268321, + "before_rms": 0.07451807797268321, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0966776949197356, + "before_rms": 0.0966776949197356, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09200739468581592, + "before_rms": 0.09200739468581592, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7275052188652554, + "before_rms": 0.7275052188652554, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 4, + "token_id": 1817, + "input_position": 48, + "layers": [ + { + "after_rms": 0.0478125046485284, + "before_rms": 0.0478125046485284, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0548085725164451, + "before_rms": 0.0548085725164451, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06471972775053976, + "before_rms": 0.06471972775053976, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6538473476210616, + "before_rms": 0.6538473476210616, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 5, + "token_id": 539, + "input_position": 49, + "layers": [ + { + "after_rms": 0.08518049302904517, + "before_rms": 0.08518049302904517, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09818462339752433, + "before_rms": 0.09818462339752433, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.10130589343276135, + "before_rms": 0.10130589343276135, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8866838924389607, + "before_rms": 0.8866838924389607, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 6, + "token_id": 864, + "input_position": 50, + "layers": [ + { + "after_rms": 0.08680030692428589, + "before_rms": 0.08680030692428589, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08012581611715355, + "before_rms": 0.08012581611715355, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.09241248361191251, + "before_rms": 0.09241248361191251, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7686299073659799, + "before_rms": 0.7686299073659799, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 7, + "token_id": 10462, + "input_position": 51, + "layers": [ + { + "after_rms": 0.05017937165905816, + "before_rms": 0.05017937165905816, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.050057021551545955, + "before_rms": 0.050057021551545955, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.05497654538007085, + "before_rms": 0.05497654538007085, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6670578115164874, + "before_rms": 0.6670578115164874, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 8, + "token_id": 958, + "input_position": 52, + "layers": [ + { + "after_rms": 0.08568044433943381, + "before_rms": 0.08568044433943381, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.10045397748397368, + "before_rms": 0.10045397748397368, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08773616457482147, + "before_rms": 0.08773616457482147, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9600428235856283, + "before_rms": 0.9600428235856283, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 9, + "token_id": 37728, + "input_position": 53, + "layers": [ + { + "after_rms": 0.06924941047054128, + "before_rms": 0.06924941047054128, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0655311740748727, + "before_rms": 0.0655311740748727, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07066158387714026, + "before_rms": 0.07066158387714026, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7935303713786936, + "before_rms": 0.7935303713786936, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 10, + "token_id": 310, + "input_position": 54, + "layers": [ + { + "after_rms": 0.08433748009746503, + "before_rms": 0.08433748009746503, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08393242733321232, + "before_rms": 0.08393242733321232, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.0819084026323373, + "before_rms": 0.0819084026323373, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9796541806488255, + "before_rms": 0.9796541806488255, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 11, + "token_id": 5335, + "input_position": 55, + "layers": [ + { + "after_rms": 0.039579585893422226, + "before_rms": 0.039579585893422226, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.046979610302819906, + "before_rms": 0.046979610302819906, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.05344053164079346, + "before_rms": 0.05344053164079346, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6960761835563655, + "before_rms": 0.6960761835563655, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 12, + "token_id": 12141, + "input_position": 56, + "layers": [ + { + "after_rms": 0.07903697242878281, + "before_rms": 0.07903697242878281, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07614783932603204, + "before_rms": 0.07614783932603204, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07944953913293375, + "before_rms": 0.07944953913293375, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8567541215882547, + "before_rms": 0.8567541215882547, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 13, + "token_id": 38182, + "input_position": 57, + "layers": [ + { + "after_rms": 0.0864315395888373, + "before_rms": 0.0864315395888373, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07366431154681498, + "before_rms": 0.07366431154681498, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06620859996023702, + "before_rms": 0.06620859996023702, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9160313052637309, + "before_rms": 0.9160313052637309, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 14, + "token_id": 321, + "input_position": 58, + "layers": [ + { + "after_rms": 0.08880976995639013, + "before_rms": 0.08880976995639013, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08999483103099812, + "before_rms": 0.08999483103099812, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.08782409614422529, + "before_rms": 0.08782409614422529, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9846578342137169, + "before_rms": 0.9846578342137169, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 15, + "token_id": 2919, + "input_position": 59, + "layers": [ + { + "after_rms": 0.03571734510932391, + "before_rms": 0.03571734510932391, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.047244981585028045, + "before_rms": 0.047244981585028045, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.05755773777600419, + "before_rms": 0.05755773777600419, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6213354826118737, + "before_rms": 0.6213354826118737, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 16, + "token_id": 1083, + "input_position": 60, + "layers": [ + { + "after_rms": 0.0626217871949745, + "before_rms": 0.0626217871949745, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06839292559199892, + "before_rms": 0.06839292559199892, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07067343367902718, + "before_rms": 0.07067343367902718, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.857509349072362, + "before_rms": 0.857509349072362, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 17, + "token_id": 32120, + "input_position": 61, + "layers": [ + { + "after_rms": 0.05464629387267075, + "before_rms": 0.05464629387267075, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06755006814151779, + "before_rms": 0.06755006814151779, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.05537102787014218, + "before_rms": 0.05537102787014218, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.7564307778216667, + "before_rms": 0.7564307778216667, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 18, + "token_id": 321, + "input_position": 62, + "layers": [ + { + "after_rms": 0.07894008680252157, + "before_rms": 0.07894008680252157, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07594297305618296, + "before_rms": 0.07594297305618296, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07215577303867654, + "before_rms": 0.07215577303867654, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.9455045304946262, + "before_rms": 0.9455045304946262, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 19, + "token_id": 22817, + "input_position": 63, + "layers": [ + { + "after_rms": 0.03382951016881296, + "before_rms": 0.03382951016881296, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.04193784117869559, + "before_rms": 0.04193784117869559, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.054778189331242615, + "before_rms": 0.054778189331242615, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.6269059013903844, + "before_rms": 0.6269059013903844, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + }, + { + "step": 20, + "token_id": 13, + "input_position": 64, + "layers": [ + { + "after_rms": 0.07374082621976509, + "before_rms": 0.07374082621976509, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.07036659145204018, + "before_rms": 0.07036659145204018, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.06651851708741562, + "before_rms": 0.06651851708741562, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 20480 + }, + { + "after_rms": 0.8993101756034922, + "before_rms": 0.8993101756034922, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 20480 + } + ] + } + ] + } + ], + "model": "qwen" + } + ] + }, + "surgery": { + "title": "Model surgery: what else changed?", + "summary": "Scale one Flint feed-forward branch to 1, 0.5 or 0. Outputs change on some cases; the same four of six objective checks pass at every scale.", + "method": "Modify the computed ffn_out-12 tensor before residual addition at every prompt and decode position. Measure pre/post RMS and neighboring blocks. No checkpoint weights are rewritten.", + "limitations": [ + "One block, six tasks and one greedy run per condition. No improvement claim. Later outputs can differ in context as well as intervention. Only the first-frame upstream equality is a direct matched-input control." + ], + "cases": [ + { + "id": "arithmetic", + "title": "Exact arithmetic", + "prompt": "Calculate (17 * 23) - 19. Reply with the integer only.", + "checks": [ + "Rule: integer", + "Expected: 372", + "Baseline, scale 1: failed", + "Half FFN, scale 0.5: failed", + "Zero FFN, scale 0: failed" + ], + "intervention": { + "tensor": "ffn_out-12", + "layer_indexing": "zero-based", + "applies_to": "all prompt and generated positions on each forward pass", + "operation": "scale computed FFN output before residual addition", + "scales": [ + 1, + 0.5, + 0 + ], + "weights_modified": false, + "kind": "activation_ablation", + "description": "Scale Flint block 12 feed-forward output before residual addition at every prompt and decode position. Compare unchanged (1), half (0.5) and zero (0). Checkpoint files are unchanged." + }, + "conditions": [ + { + "id": "baseline", + "label": "Baseline, scale 1", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "374", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 33, + "elapsed_seconds": 0.738173075, + "request_wall_seconds": 0.7542800180381164, + "time_to_first_token_seconds": 0.689367342973128, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0042319691359885515, + "block12_after_rms": 0.0042319691359885515 + }, + "check": { + "passed": false, + "rule": "integer", + "expected": "372", + "character_count": 3, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 9056, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004216483101124048, + "before_rms": 0.004216483101124048, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.0042319691359885515, + "before_rms": 0.0042319691359885515, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.005525138486063405, + "before_rms": 0.005525138486063405, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.014221511290226446, + "before_rms": 0.014221511290226446, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 270336 + } + ] + } + ] + }, + { + "id": "half", + "label": "Half FFN, scale 0.5", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "352", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 33, + "elapsed_seconds": 0.719981441, + "request_wall_seconds": 0.7371821940178052, + "time_to_first_token_seconds": 0.6725124060176313, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0042319691359885515, + "block12_after_rms": 0.0021159845679942758 + }, + "check": { + "passed": false, + "rule": "integer", + "expected": "372", + "character_count": 3, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0.5 + }, + "measurements": [ + { + "step": 0, + "token_id": 11510, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004216483101124048, + "before_rms": 0.004216483101124048, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.0021159845679942758, + "before_rms": 0.0042319691359885515, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.005548826881067284, + "before_rms": 0.005548826881067284, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.013772509153955564, + "before_rms": 0.013772509153955564, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 270336 + } + ] + } + ] + }, + { + "id": "zero", + "label": "Zero FFN, scale 0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "340", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 33, + "elapsed_seconds": 0.721295279, + "request_wall_seconds": 0.7335850830422714, + "time_to_first_token_seconds": 0.6692783629987389, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0042319691359885515, + "block12_after_rms": 0.0 + }, + "check": { + "passed": false, + "rule": "integer", + "expected": "372", + "character_count": 3, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0 + }, + "measurements": [ + { + "step": 0, + "token_id": 10333, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004216483101124048, + "before_rms": 0.004216483101124048, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.0, + "before_rms": 0.0042319691359885515, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.0056156565381101035, + "before_rms": 0.0056156565381101035, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 270336 + }, + { + "after_rms": 0.013872404589524597, + "before_rms": 0.013872404589524597, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 270336 + } + ] + } + ] + } + ], + "comparison": { + "untargeted_first_frame_block11_identical": true, + "downstream_first_frame_block13_changed": true, + "weights_modified": false, + "all_saved_frames_available_in_results": true + } + }, + { + "id": "json", + "title": "Strict JSON format", + "prompt": "Return exactly this JSON object and no other text: {\"status\":\"ok\",\"count\":3}", + "checks": [ + "Rule: json", + "Expected: {'status': 'ok', 'count': 3}", + "Baseline, scale 1: passed", + "Half FFN, scale 0.5: passed", + "Zero FFN, scale 0: passed" + ], + "intervention": { + "tensor": "ffn_out-12", + "layer_indexing": "zero-based", + "applies_to": "all prompt and generated positions on each forward pass", + "operation": "scale computed FFN output before residual addition", + "scales": [ + 1, + 0.5, + 0 + ], + "weights_modified": false, + "kind": "activation_ablation", + "description": "Scale Flint block 12 feed-forward output before residual addition at every prompt and decode position. Compare unchanged (1), half (0.5) and zero (0). Checkpoint files are unchanged." + }, + "conditions": [ + { + "id": "baseline", + "label": "Baseline, scale 1", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "{\"status\":\"ok\",\"count\":3}", + "metrics": { + "generated_tokens": 9, + "prompt_tokens": 35, + "elapsed_seconds": 1.259416502, + "request_wall_seconds": 1.2757591230329126, + "time_to_first_token_seconds": 0.7007529039401561, + "decode_tokens_per_second": 13.911468772577386, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004073247840262152, + "block12_after_rms": 0.004073247840262152 + }, + "check": { + "passed": true, + "rule": "json", + "expected": { + "status": "ok", + "count": 3 + }, + "character_count": 25, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 5482, + "input_position": 34, + "layers": [ + { + "after_rms": 0.00442951382677711, + "before_rms": 0.00442951382677711, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.004073247840262152, + "before_rms": 0.004073247840262152, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.00524975286610561, + "before_rms": 0.00524975286610561, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.014444385872118682, + "before_rms": 0.014444385872118682, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 286720 + } + ] + }, + { + "step": 1, + "token_id": 5462, + "input_position": 35, + "layers": [ + { + "after_rms": 0.005793034803083275, + "before_rms": 0.005793034803083275, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0046820736039610645, + "before_rms": 0.0046820736039610645, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00649377540352318, + "before_rms": 0.00649377540352318, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019726933354943656, + "before_rms": 0.019726933354943656, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 4890, + "input_position": 36, + "layers": [ + { + "after_rms": 0.005294017353787343, + "before_rms": 0.005294017353787343, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0034695353362188502, + "before_rms": 0.0034695353362188502, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00738829527350583, + "before_rms": 0.00738829527350583, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019360584231844193, + "before_rms": 0.019360584231844193, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 544, + "input_position": 37, + "layers": [ + { + "after_rms": 0.00491148399515983, + "before_rms": 0.00491148399515983, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00380345983046727, + "before_rms": 0.00380345983046727, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005149091746140284, + "before_rms": 0.005149091746140284, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016276301961648348, + "before_rms": 0.016276301961648348, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1147, + "input_position": 38, + "layers": [ + { + "after_rms": 0.006222679356489009, + "before_rms": 0.006222679356489009, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0036829646504058975, + "before_rms": 0.0036829646504058975, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006655483694052688, + "before_rms": 0.006655483694052688, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019208434114849963, + "before_rms": 0.019208434114849963, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 2816, + "input_position": 39, + "layers": [ + { + "after_rms": 0.005760162518562755, + "before_rms": 0.005760162518562755, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003275269655906223, + "before_rms": 0.003275269655906223, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004388659397945476, + "before_rms": 0.004388659397945476, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015988995629268844, + "before_rms": 0.015988995629268844, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 1129, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004863272662823825, + "before_rms": 0.004863272662823825, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0038314286271215804, + "before_rms": 0.0038314286271215804, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007286998970545343, + "before_rms": 0.007286998970545343, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018843493317539196, + "before_rms": 0.018843493317539196, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 18, + "input_position": 41, + "layers": [ + { + "after_rms": 0.004603991231053795, + "before_rms": 0.004603991231053795, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0030208238582806433, + "before_rms": 0.0030208238582806433, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004703706397060763, + "before_rms": 0.004703706397060763, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013840299536615416, + "before_rms": 0.013840299536615416, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 92, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0039039619301298022, + "before_rms": 0.0039039619301298022, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004057874942645444, + "before_rms": 0.004057874942645444, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006582297878179331, + "before_rms": 0.006582297878179331, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016272639100438333, + "before_rms": 0.016272639100438333, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "half", + "label": "Half FFN, scale 0.5", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "{\"status\":\"ok\",\"count\":3}", + "metrics": { + "generated_tokens": 9, + "prompt_tokens": 35, + "elapsed_seconds": 1.246835879, + "request_wall_seconds": 1.262888611992821, + "time_to_first_token_seconds": 0.6985334729542956, + "decode_tokens_per_second": 14.17425932571783, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004073247840262152, + "block12_after_rms": 0.002036623920131076 + }, + "check": { + "passed": true, + "rule": "json", + "expected": { + "status": "ok", + "count": 3 + }, + "character_count": 25, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0.5 + }, + "measurements": [ + { + "step": 0, + "token_id": 5482, + "input_position": 34, + "layers": [ + { + "after_rms": 0.00442951382677711, + "before_rms": 0.00442951382677711, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.002036623920131076, + "before_rms": 0.004073247840262152, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.00511203630467351, + "before_rms": 0.00511203630467351, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.014008885307881237, + "before_rms": 0.014008885307881237, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 286720 + } + ] + }, + { + "step": 1, + "token_id": 5462, + "input_position": 35, + "layers": [ + { + "after_rms": 0.005793034803083275, + "before_rms": 0.005793034803083275, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0023410368019805322, + "before_rms": 0.0046820736039610645, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006683535999525596, + "before_rms": 0.006683535999525596, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01886032206135178, + "before_rms": 0.01886032206135178, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 4890, + "input_position": 36, + "layers": [ + { + "after_rms": 0.005294017353787343, + "before_rms": 0.005294017353787343, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0017347676681094251, + "before_rms": 0.0034695353362188502, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0076820001587381715, + "before_rms": 0.0076820001587381715, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019039892595830377, + "before_rms": 0.019039892595830377, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 544, + "input_position": 37, + "layers": [ + { + "after_rms": 0.00491148399515983, + "before_rms": 0.00491148399515983, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.001901729915233635, + "before_rms": 0.00380345983046727, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005344106452152085, + "before_rms": 0.005344106452152085, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01594381241067434, + "before_rms": 0.01594381241067434, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1147, + "input_position": 38, + "layers": [ + { + "after_rms": 0.006222679356489009, + "before_rms": 0.006222679356489009, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0018414823252029487, + "before_rms": 0.0036829646504058975, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00664047642037073, + "before_rms": 0.00664047642037073, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01916710645697236, + "before_rms": 0.01916710645697236, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 2816, + "input_position": 39, + "layers": [ + { + "after_rms": 0.005760162518562755, + "before_rms": 0.005760162518562755, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0016376348279531115, + "before_rms": 0.003275269655906223, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0046118119033721995, + "before_rms": 0.0046118119033721995, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015733109219504087, + "before_rms": 0.015733109219504087, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 1129, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004863272662823825, + "before_rms": 0.004863272662823825, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0019157143135607902, + "before_rms": 0.0038314286271215804, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007354879141339851, + "before_rms": 0.007354879141339851, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0188168576455134, + "before_rms": 0.0188168576455134, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 18, + "input_position": 41, + "layers": [ + { + "after_rms": 0.004603991231053795, + "before_rms": 0.004603991231053795, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0015104119291403217, + "before_rms": 0.0030208238582806433, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004847850483184061, + "before_rms": 0.004847850483184061, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013329843619400705, + "before_rms": 0.013329843619400705, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 92, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0039039619301298022, + "before_rms": 0.0039039619301298022, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002028937471322722, + "before_rms": 0.004057874942645444, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006716603826791902, + "before_rms": 0.006716603826791902, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015738018767318247, + "before_rms": 0.015738018767318247, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "zero", + "label": "Zero FFN, scale 0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "{\"status\":\"ok\",\"count\":3}", + "metrics": { + "generated_tokens": 9, + "prompt_tokens": 35, + "elapsed_seconds": 1.252187772, + "request_wall_seconds": 1.2679177080281079, + "time_to_first_token_seconds": 0.7051889860304072, + "decode_tokens_per_second": 14.213787672176204, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004073247840262152, + "block12_after_rms": 0.0 + }, + "check": { + "passed": true, + "rule": "json", + "expected": { + "status": "ok", + "count": 3 + }, + "character_count": 25, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0 + }, + "measurements": [ + { + "step": 0, + "token_id": 5482, + "input_position": 34, + "layers": [ + { + "after_rms": 0.00442951382677711, + "before_rms": 0.00442951382677711, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.0, + "before_rms": 0.004073247840262152, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.005232460518540055, + "before_rms": 0.005232460518540055, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 286720 + }, + { + "after_rms": 0.014132856084779098, + "before_rms": 0.014132856084779098, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 286720 + } + ] + }, + { + "step": 1, + "token_id": 5462, + "input_position": 35, + "layers": [ + { + "after_rms": 0.005793034803083275, + "before_rms": 0.005793034803083275, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0046820736039610645, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006932431350240155, + "before_rms": 0.006932431350240155, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018253630668038652, + "before_rms": 0.018253630668038652, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 4890, + "input_position": 36, + "layers": [ + { + "after_rms": 0.005294017353787343, + "before_rms": 0.005294017353787343, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0034695353362188502, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007877159840683485, + "before_rms": 0.007877159840683485, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018856526179738382, + "before_rms": 0.018856526179738382, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 544, + "input_position": 37, + "layers": [ + { + "after_rms": 0.00491148399515983, + "before_rms": 0.00491148399515983, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.00380345983046727, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005610946438388422, + "before_rms": 0.005610946438388422, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016040718140881722, + "before_rms": 0.016040718140881722, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1147, + "input_position": 38, + "layers": [ + { + "after_rms": 0.006222679356489009, + "before_rms": 0.006222679356489009, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0036829646504058975, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0065644993447098255, + "before_rms": 0.0065644993447098255, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01935782943457066, + "before_rms": 0.01935782943457066, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 2816, + "input_position": 39, + "layers": [ + { + "after_rms": 0.005760162518562755, + "before_rms": 0.005760162518562755, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.003275269655906223, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0050584932701879986, + "before_rms": 0.0050584932701879986, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015686042844506247, + "before_rms": 0.015686042844506247, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 1129, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004863272662823825, + "before_rms": 0.004863272662823825, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0038314286271215804, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007195615412259338, + "before_rms": 0.007195615412259338, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018477358488448153, + "before_rms": 0.018477358488448153, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 18, + "input_position": 41, + "layers": [ + { + "after_rms": 0.004603991231053795, + "before_rms": 0.004603991231053795, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0030208238582806433, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005075615167638164, + "before_rms": 0.005075615167638164, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013303633799015144, + "before_rms": 0.013303633799015144, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 92, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0039039619301298022, + "before_rms": 0.0039039619301298022, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.004057874942645444, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0069263167645750995, + "before_rms": 0.0069263167645750995, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015314707984930225, + "before_rms": 0.015314707984930225, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + } + ], + "comparison": { + "untargeted_first_frame_block11_identical": true, + "downstream_first_frame_block13_changed": true, + "weights_modified": false, + "all_saved_frames_available_in_results": true + } + }, + { + "id": "fact", + "title": "Short factual recall", + "prompt": "What is the chemical symbol for gold? Reply with the symbol only.", + "checks": [ + "Rule: exact", + "Expected: Au", + "Baseline, scale 1: passed", + "Half FFN, scale 0.5: passed", + "Zero FFN, scale 0: passed" + ], + "intervention": { + "tensor": "ffn_out-12", + "layer_indexing": "zero-based", + "applies_to": "all prompt and generated positions on each forward pass", + "operation": "scale computed FFN output before residual addition", + "scales": [ + 1, + 0.5, + 0 + ], + "weights_modified": false, + "kind": "activation_ablation", + "description": "Scale Flint block 12 feed-forward output before residual addition at every prompt and decode position. Compare unchanged (1), half (0.5) and zero (0). Checkpoint files are unchanged." + }, + "conditions": [ + { + "id": "baseline", + "label": "Baseline, scale 1", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Au", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 30, + "elapsed_seconds": 0.650667801, + "request_wall_seconds": 0.6673325290903449, + "time_to_first_token_seconds": 0.6034285150235519, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004120685912594556, + "block12_after_rms": 0.004120685912594556 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "Au", + "character_count": 2, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 22911, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0042503533024456225, + "before_rms": 0.0042503533024456225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.004120685912594556, + "before_rms": 0.004120685912594556, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.005313107066375563, + "before_rms": 0.005313107066375563, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.01490764588916619, + "before_rms": 0.01490764588916619, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 245760 + } + ] + } + ] + }, + { + "id": "half", + "label": "Half FFN, scale 0.5", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Au", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 30, + "elapsed_seconds": 0.751291221, + "request_wall_seconds": 0.7725467890268192, + "time_to_first_token_seconds": 0.6922730850055814, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004120685912594556, + "block12_after_rms": 0.002060342956297278 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "Au", + "character_count": 2, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0.5 + }, + "measurements": [ + { + "step": 0, + "token_id": 22911, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0042503533024456225, + "before_rms": 0.0042503533024456225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.002060342956297278, + "before_rms": 0.004120685912594556, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.005343264427809506, + "before_rms": 0.005343264427809506, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.014565723365685252, + "before_rms": 0.014565723365685252, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 245760 + } + ] + } + ] + }, + { + "id": "zero", + "label": "Zero FFN, scale 0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Au", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 30, + "elapsed_seconds": 0.764433258, + "request_wall_seconds": 0.7862005030037835, + "time_to_first_token_seconds": 0.7070632589748129, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41799256, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004120685912594556, + "block12_after_rms": 0.0 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "Au", + "character_count": 2, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0 + }, + "measurements": [ + { + "step": 0, + "token_id": 22911, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0042503533024456225, + "before_rms": 0.0042503533024456225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.0, + "before_rms": 0.004120685912594556, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.005448940453841594, + "before_rms": 0.005448940453841594, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 245760 + }, + { + "after_rms": 0.01475624234457033, + "before_rms": 0.01475624234457033, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 245760 + } + ] + } + ] + } + ], + "comparison": { + "untargeted_first_frame_block11_identical": true, + "downstream_first_frame_block13_changed": true, + "weights_modified": false, + "all_saved_frames_available_in_results": true + } + }, + { + "id": "correction", + "title": "Use the corrected fact", + "prompt": "The access color was blue. Correction: the access color is now amber. What is the current access color? Reply with one word only.", + "checks": [ + "Rule: exact", + "Expected: amber", + "Baseline, scale 1: passed", + "Half FFN, scale 0.5: passed", + "Zero FFN, scale 0: passed" + ], + "intervention": { + "tensor": "ffn_out-12", + "layer_indexing": "zero-based", + "applies_to": "all prompt and generated positions on each forward pass", + "operation": "scale computed FFN output before residual addition", + "scales": [ + 1, + 0.5, + 0 + ], + "weights_modified": false, + "kind": "activation_ablation", + "description": "Scale Flint block 12 feed-forward output before residual addition at every prompt and decode position. Compare unchanged (1), half (0.5) and zero (0). Checkpoint files are unchanged." + }, + "conditions": [ + { + "id": "baseline", + "label": "Baseline, scale 1", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "amber", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 44, + "elapsed_seconds": 0.974556193, + "request_wall_seconds": 0.9955227529862896, + "time_to_first_token_seconds": 0.9343198160640895, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.00441159816545625, + "block12_after_rms": 0.00441159816545625 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "amber", + "character_count": 5, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 13732, + "input_position": 43, + "layers": [ + { + "after_rms": 0.004252703885422568, + "before_rms": 0.004252703885422568, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.00441159816545625, + "before_rms": 0.00441159816545625, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.005657736200656421, + "before_rms": 0.005657736200656421, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.014876767054565872, + "before_rms": 0.014876767054565872, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 360448 + } + ] + } + ] + }, + { + "id": "half", + "label": "Half FFN, scale 0.5", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "amber", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 44, + "elapsed_seconds": 0.845619764, + "request_wall_seconds": 0.8612813500221819, + "time_to_first_token_seconds": 0.8017731390427798, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.00441159816545625, + "block12_after_rms": 0.002205799082728125 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "amber", + "character_count": 5, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0.5 + }, + "measurements": [ + { + "step": 0, + "token_id": 13732, + "input_position": 43, + "layers": [ + { + "after_rms": 0.004252703885422568, + "before_rms": 0.004252703885422568, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.002205799082728125, + "before_rms": 0.00441159816545625, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.005628658790824781, + "before_rms": 0.005628658790824781, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.014474070728207288, + "before_rms": 0.014474070728207288, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 360448 + } + ] + } + ] + }, + { + "id": "zero", + "label": "Zero FFN, scale 0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "amber", + "metrics": { + "generated_tokens": 1, + "prompt_tokens": 44, + "elapsed_seconds": 1.003642273, + "request_wall_seconds": 1.0181882489705458, + "time_to_first_token_seconds": 0.9386005689157173, + "decode_tokens_per_second": null, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.00441159816545625, + "block12_after_rms": 0.0 + }, + "check": { + "passed": true, + "rule": "exact", + "expected": "amber", + "character_count": 5, + "word_count": 1 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0 + }, + "measurements": [ + { + "step": 0, + "token_id": 13732, + "input_position": 43, + "layers": [ + { + "after_rms": 0.004252703885422568, + "before_rms": 0.004252703885422568, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.0, + "before_rms": 0.00441159816545625, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.005651444761795099, + "before_rms": 0.005651444761795099, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 360448 + }, + { + "after_rms": 0.01456451414024136, + "before_rms": 0.01456451414024136, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 360448 + } + ] + } + ] + } + ], + "comparison": { + "untargeted_first_frame_block11_identical": true, + "downstream_first_frame_block13_changed": true, + "weights_modified": false, + "all_saved_frames_available_in_results": true + } + }, + { + "id": "code", + "title": "Python expression repair", + "prompt": "Complete the function with one Python expression only, no markdown: def is_even(n): return ___ . The function must return True exactly when integer n is even.", + "checks": [ + "Rule: expression", + "Expected: n % 2 == 0", + "Baseline, scale 1: failed", + "Half FFN, scale 0.5: failed", + "Zero FFN, scale 0: failed" + ], + "intervention": { + "tensor": "ffn_out-12", + "layer_indexing": "zero-based", + "applies_to": "all prompt and generated positions on each forward pass", + "operation": "scale computed FFN output before residual addition", + "scales": [ + 1, + 0.5, + 0 + ], + "weights_modified": false, + "kind": "activation_ablation", + "description": "Scale Flint block 12 feed-forward output before residual addition at every prompt and decode position. Compare unchanged (1), half (0.5) and zero (0). Checkpoint files are unchanged." + }, + "conditions": [ + { + "id": "baseline", + "label": "Baseline, scale 1", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "return n % 2 == 0", + "metrics": { + "generated_tokens": 8, + "prompt_tokens": 48, + "elapsed_seconds": 1.522581758, + "request_wall_seconds": 1.5477090129861608, + "time_to_first_token_seconds": 0.9909320520237088, + "decode_tokens_per_second": 12.570319015535855, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004202357680490731, + "block12_after_rms": 0.004202357680490731 + }, + "check": { + "passed": false, + "rule": "expression", + "expected": "n % 2 == 0", + "character_count": 17, + "word_count": 6 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 963, + "input_position": 47, + "layers": [ + { + "after_rms": 0.004194344649342437, + "before_rms": 0.004194344649342437, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.004202357680490731, + "before_rms": 0.004202357680490731, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.0052131473290487635, + "before_rms": 0.0052131473290487635, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.014781441026154475, + "before_rms": 0.014781441026154475, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 393216 + } + ] + }, + { + "step": 1, + "token_id": 307, + "input_position": 48, + "layers": [ + { + "after_rms": 0.005277509304535801, + "before_rms": 0.005277509304535801, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00609774510971192, + "before_rms": 0.00609774510971192, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005608145566049825, + "before_rms": 0.005608145566049825, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018110784621771632, + "before_rms": 0.018110784621771632, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 1152, + "input_position": 49, + "layers": [ + { + "after_rms": 0.004954062696681225, + "before_rms": 0.004954062696681225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004947633813264256, + "before_rms": 0.004947633813264256, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006281951776179792, + "before_rms": 0.006281951776179792, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01793926095231872, + "before_rms": 0.01793926095231872, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 220, + "input_position": 50, + "layers": [ + { + "after_rms": 0.005725603347206012, + "before_rms": 0.005725603347206012, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007613934635856348, + "before_rms": 0.007613934635856348, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007067586106815484, + "before_rms": 0.007067586106815484, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02065595811031347, + "before_rms": 0.02065595811031347, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 17, + "input_position": 51, + "layers": [ + { + "after_rms": 0.004384558665618408, + "before_rms": 0.004384558665618408, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002694924917014614, + "before_rms": 0.002694924917014614, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003985706515046729, + "before_rms": 0.003985706515046729, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.012855823082554165, + "before_rms": 0.012855823082554165, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 823, + "input_position": 52, + "layers": [ + { + "after_rms": 0.00695481022115809, + "before_rms": 0.00695481022115809, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005148284890432175, + "before_rms": 0.005148284890432175, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00860817356635732, + "before_rms": 0.00860817356635732, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019382560123796404, + "before_rms": 0.019382560123796404, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 220, + "input_position": 53, + "layers": [ + { + "after_rms": 0.008360185779230117, + "before_rms": 0.008360185779230117, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005331645591182674, + "before_rms": 0.005331645591182674, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.011988298999222614, + "before_rms": 0.011988298999222614, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02324778120292584, + "before_rms": 0.02324778120292584, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 15, + "input_position": 54, + "layers": [ + { + "after_rms": 0.004389820760472024, + "before_rms": 0.004389820760472024, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002823151948738866, + "before_rms": 0.002823151948738866, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004278350453083995, + "before_rms": 0.004278350453083995, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.013667174465740126, + "before_rms": 0.013667174465740126, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "half", + "label": "Half FFN, scale 0.5", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "return n % 2 == 0", + "metrics": { + "generated_tokens": 8, + "prompt_tokens": 48, + "elapsed_seconds": 1.451376005, + "request_wall_seconds": 1.5098812960786745, + "time_to_first_token_seconds": 0.9736877980176359, + "decode_tokens_per_second": 13.053039182374336, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004202357680490731, + "block12_after_rms": 0.0021011788402453655 + }, + "check": { + "passed": false, + "rule": "expression", + "expected": "n % 2 == 0", + "character_count": 17, + "word_count": 6 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0.5 + }, + "measurements": [ + { + "step": 0, + "token_id": 963, + "input_position": 47, + "layers": [ + { + "after_rms": 0.004194344649342437, + "before_rms": 0.004194344649342437, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.0021011788402453655, + "before_rms": 0.004202357680490731, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.005090052356663985, + "before_rms": 0.005090052356663985, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.014359352110858759, + "before_rms": 0.014359352110858759, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 393216 + } + ] + }, + { + "step": 1, + "token_id": 307, + "input_position": 48, + "layers": [ + { + "after_rms": 0.005277509304535801, + "before_rms": 0.005277509304535801, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00304887255485596, + "before_rms": 0.00609774510971192, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006014683540483773, + "before_rms": 0.006014683540483773, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017053751245143085, + "before_rms": 0.017053751245143085, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 1152, + "input_position": 49, + "layers": [ + { + "after_rms": 0.004954062696681225, + "before_rms": 0.004954062696681225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002473816906632128, + "before_rms": 0.004947633813264256, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006547528545526086, + "before_rms": 0.006547528545526086, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017097207015519773, + "before_rms": 0.017097207015519773, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 220, + "input_position": 50, + "layers": [ + { + "after_rms": 0.005725603347206012, + "before_rms": 0.005725603347206012, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003806967317928174, + "before_rms": 0.007613934635856348, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007202666322056073, + "before_rms": 0.007202666322056073, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019556659993829776, + "before_rms": 0.019556659993829776, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 17, + "input_position": 51, + "layers": [ + { + "after_rms": 0.004384558665618408, + "before_rms": 0.004384558665618408, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.001347462458507307, + "before_rms": 0.002694924917014614, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004306118830807208, + "before_rms": 0.004306118830807208, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.012367683992419722, + "before_rms": 0.012367683992419722, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 823, + "input_position": 52, + "layers": [ + { + "after_rms": 0.00695481022115809, + "before_rms": 0.00695481022115809, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0025741424452160877, + "before_rms": 0.005148284890432175, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0085343766880633, + "before_rms": 0.0085343766880633, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018658682254849446, + "before_rms": 0.018658682254849446, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 220, + "input_position": 53, + "layers": [ + { + "after_rms": 0.008360185779230117, + "before_rms": 0.008360185779230117, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002665822795591337, + "before_rms": 0.005331645591182674, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.012100347801828254, + "before_rms": 0.012100347801828254, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02291563837710362, + "before_rms": 0.02291563837710362, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 15, + "input_position": 54, + "layers": [ + { + "after_rms": 0.004389820760472024, + "before_rms": 0.004389820760472024, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.001411575974369433, + "before_rms": 0.002823151948738866, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00448153573004611, + "before_rms": 0.00448153573004611, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.012590223970782119, + "before_rms": 0.012590223970782119, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "zero", + "label": "Zero FFN, scale 0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "def is_even(n): return n % 2 == 0", + "metrics": { + "generated_tokens": 13, + "prompt_tokens": 48, + "elapsed_seconds": 1.766926317, + "request_wall_seconds": 1.8241781580727547, + "time_to_first_token_seconds": 0.9580211520660669, + "decode_tokens_per_second": 13.853450095998406, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": false, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.004202357680490731, + "block12_after_rms": 0.0 + }, + "check": { + "passed": false, + "rule": "expression", + "expected": "n % 2 == 0", + "character_count": 33, + "word_count": 8 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0 + }, + "measurements": [ + { + "step": 0, + "token_id": 1182, + "input_position": 47, + "layers": [ + { + "after_rms": 0.004194344649342437, + "before_rms": 0.004194344649342437, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.0, + "before_rms": 0.004202357680490731, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.005070940113459419, + "before_rms": 0.005070940113459419, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 393216 + }, + { + "after_rms": 0.014431911821101652, + "before_rms": 0.014431911821101652, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 393216 + } + ] + }, + { + "step": 1, + "token_id": 332, + "input_position": 48, + "layers": [ + { + "after_rms": 0.00480106946797479, + "before_rms": 0.00480106946797479, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0055027957166948055, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0056345791848020155, + "before_rms": 0.0056345791848020155, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017990489860241182, + "before_rms": 0.017990489860241182, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 75154, + "input_position": 49, + "layers": [ + { + "after_rms": 0.005708045099524496, + "before_rms": 0.005708045099524496, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.004207955013596103, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0075556508119026065, + "before_rms": 0.0075556508119026065, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017833711235606925, + "before_rms": 0.017833711235606925, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 1249, + "input_position": 50, + "layers": [ + { + "after_rms": 0.0058045473429944065, + "before_rms": 0.0058045473429944065, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005650785619108264, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007835386583253242, + "before_rms": 0.007835386583253242, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018218131354517456, + "before_rms": 0.018218131354517456, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1682, + "input_position": 51, + "layers": [ + { + "after_rms": 0.005459167018013679, + "before_rms": 0.005459167018013679, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005458814458024571, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006723062471136803, + "before_rms": 0.006723062471136803, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018003037284926235, + "before_rms": 0.018003037284926235, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 597, + "input_position": 52, + "layers": [ + { + "after_rms": 0.004881552841111804, + "before_rms": 0.004881552841111804, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.004638313561400321, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006991441598900912, + "before_rms": 0.006991441598900912, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016203231535129245, + "before_rms": 0.016203231535129245, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 307, + "input_position": 53, + "layers": [ + { + "after_rms": 0.005143842728252836, + "before_rms": 0.005143842728252836, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005127108596991448, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008170598422530883, + "before_rms": 0.008170598422530883, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01798307359808869, + "before_rms": 0.01798307359808869, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 1152, + "input_position": 54, + "layers": [ + { + "after_rms": 0.0049824502938795555, + "before_rms": 0.0049824502938795555, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005199642872045386, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006848563763133479, + "before_rms": 0.006848563763133479, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016730393035201727, + "before_rms": 0.016730393035201727, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 220, + "input_position": 55, + "layers": [ + { + "after_rms": 0.005708276925466525, + "before_rms": 0.005708276925466525, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.008022970417094337, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007115667659550918, + "before_rms": 0.007115667659550918, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019026613242409732, + "before_rms": 0.019026613242409732, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 9, + "token_id": 17, + "input_position": 56, + "layers": [ + { + "after_rms": 0.0042886941345147015, + "before_rms": 0.0042886941345147015, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.002710862488091774, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004265423359902893, + "before_rms": 0.004265423359902893, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01199161845757347, + "before_rms": 0.01199161845757347, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 10, + "token_id": 823, + "input_position": 57, + "layers": [ + { + "after_rms": 0.007280227848737238, + "before_rms": 0.007280227848737238, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005472617971061407, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008876968970410364, + "before_rms": 0.008876968970410364, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.018901080384531817, + "before_rms": 0.018901080384531817, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 11, + "token_id": 220, + "input_position": 58, + "layers": [ + { + "after_rms": 0.008286967782600638, + "before_rms": 0.008286967782600638, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005098250235859169, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01039179921955715, + "before_rms": 0.01039179921955715, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021144833704014477, + "before_rms": 0.021144833704014477, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 12, + "token_id": 15, + "input_position": 59, + "layers": [ + { + "after_rms": 0.004301028624991628, + "before_rms": 0.004301028624991628, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.002916336479821133, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004546469810197431, + "before_rms": 0.004546469810197431, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01185733829801787, + "before_rms": 0.01185733829801787, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + } + ], + "comparison": { + "untargeted_first_frame_block11_identical": true, + "downstream_first_frame_block13_changed": true, + "weights_modified": false, + "all_saved_frames_available_in_results": true + } + }, + { + "id": "brief", + "title": "One sentence explanation", + "prompt": "Explain photosynthesis in exactly one sentence containing the words sunlight and glucose.", + "checks": [ + "Rule: sentence", + "Expected: ['sunlight', 'glucose']", + "Baseline, scale 1: passed", + "Half FFN, scale 0.5: passed", + "Zero FFN, scale 0: passed" + ], + "intervention": { + "tensor": "ffn_out-12", + "layer_indexing": "zero-based", + "applies_to": "all prompt and generated positions on each forward pass", + "operation": "scale computed FFN output before residual addition", + "scales": [ + 1, + 0.5, + 0 + ], + "weights_modified": false, + "kind": "activation_ablation", + "description": "Scale Flint block 12 feed-forward output before residual addition at every prompt and decode position. Compare unchanged (1), half (0.5) and zero (0). Checkpoint files are unchanged." + }, + "conditions": [ + { + "id": "baseline", + "label": "Baseline, scale 1", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Photosynthesis is the process by which plants use sunlight to create glucose, storing solar energy in chemical form.", + "metrics": { + "generated_tokens": 21, + "prompt_tokens": 29, + "elapsed_seconds": 1.985051171, + "request_wall_seconds": 2.0436750209191814, + "time_to_first_token_seconds": 0.6549666819628328, + "decode_tokens_per_second": 14.402335769662937, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0044633950081139125, + "block12_after_rms": 0.0044633950081139125 + }, + "check": { + "passed": true, + "rule": "sentence", + "expected": [ + "sunlight", + "glucose" + ], + "character_count": 116, + "word_count": 18 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 1 + }, + "measurements": [ + { + "step": 0, + "token_id": 49106, + "input_position": 28, + "layers": [ + { + "after_rms": 0.004260129892907466, + "before_rms": 0.004260129892907466, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.0044633950081139125, + "before_rms": 0.0044633950081139125, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.005860773055104808, + "before_rms": 0.005860773055104808, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.015381996005573538, + "before_rms": 0.015381996005573538, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 237568 + } + ] + }, + { + "step": 1, + "token_id": 25424, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0056475146712665295, + "before_rms": 0.0056475146712665295, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008613622622386482, + "before_rms": 0.008613622622386482, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006688744656356641, + "before_rms": 0.006688744656356641, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.022823769881939016, + "before_rms": 0.022823769881939016, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 332, + "input_position": 30, + "layers": [ + { + "after_rms": 0.0064819400327529395, + "before_rms": 0.0064819400327529395, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0066762037147183205, + "before_rms": 0.0066762037147183205, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008751997501954894, + "before_rms": 0.008751997501954894, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.027229842448171573, + "before_rms": 0.027229842448171573, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 269, + "input_position": 31, + "layers": [ + { + "after_rms": 0.0045512880200197945, + "before_rms": 0.0045512880200197945, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005065420297108188, + "before_rms": 0.005065420297108188, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005762789431004507, + "before_rms": 0.005762789431004507, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017768401646140505, + "before_rms": 0.017768401646140505, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1260, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004939956954683621, + "before_rms": 0.004939956954683621, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0038142763724026384, + "before_rms": 0.0038142763724026384, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005913005531372122, + "before_rms": 0.005913005531372122, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01761890031507447, + "before_rms": 0.01761890031507447, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 491, + "input_position": 33, + "layers": [ + { + "after_rms": 0.00603825473034925, + "before_rms": 0.00603825473034925, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005399106143511229, + "before_rms": 0.005399106143511229, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008736535756150335, + "before_rms": 0.008736535756150335, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.023288508898713978, + "before_rms": 0.023288508898713978, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 687, + "input_position": 34, + "layers": [ + { + "after_rms": 0.004825344575484225, + "before_rms": 0.004825344575484225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0035722557633475504, + "before_rms": 0.0035722557633475504, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004528394446715205, + "before_rms": 0.004528394446715205, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015558510785105506, + "before_rms": 0.015558510785105506, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 7370, + "input_position": 35, + "layers": [ + { + "after_rms": 0.00599941128253067, + "before_rms": 0.00599941128253067, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00554379069270193, + "before_rms": 0.00554379069270193, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00600245072054164, + "before_rms": 0.00600245072054164, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021459060251264164, + "before_rms": 0.021459060251264164, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 896, + "input_position": 36, + "layers": [ + { + "after_rms": 0.0051950504054689125, + "before_rms": 0.0051950504054689125, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006141893723174439, + "before_rms": 0.006141893723174439, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009373710872705807, + "before_rms": 0.009373710872705807, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024138161560965755, + "before_rms": 0.024138161560965755, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 9, + "token_id": 24044, + "input_position": 37, + "layers": [ + { + "after_rms": 0.008031516167890848, + "before_rms": 0.008031516167890848, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005368163828714329, + "before_rms": 0.005368163828714329, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005839425814527791, + "before_rms": 0.005839425814527791, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019011043273201134, + "before_rms": 0.019011043273201134, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 10, + "token_id": 295, + "input_position": 38, + "layers": [ + { + "after_rms": 0.009314847401164192, + "before_rms": 0.009314847401164192, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00741898116192189, + "before_rms": 0.00741898116192189, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00969530415712232, + "before_rms": 0.00969530415712232, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02606507768278446, + "before_rms": 0.02606507768278446, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 11, + "token_id": 1815, + "input_position": 39, + "layers": [ + { + "after_rms": 0.0051360181463417694, + "before_rms": 0.0051360181463417694, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005146656362487788, + "before_rms": 0.005146656362487788, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005882866866859857, + "before_rms": 0.005882866866859857, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017485786719118993, + "before_rms": 0.017485786719118993, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 12, + "token_id": 18175, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004435782376176152, + "before_rms": 0.004435782376176152, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007327068248162281, + "before_rms": 0.007327068248162281, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006088986661020862, + "before_rms": 0.006088986661020862, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020237327919821312, + "before_rms": 0.020237327919821312, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 13, + "token_id": 11, + "input_position": 41, + "layers": [ + { + "after_rms": 0.010509592928354338, + "before_rms": 0.010509592928354338, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0072267658458743745, + "before_rms": 0.0072267658458743745, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0072706489196035584, + "before_rms": 0.0072706489196035584, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.026778168430377582, + "before_rms": 0.026778168430377582, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 14, + "token_id": 20827, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0036465031366290737, + "before_rms": 0.0036465031366290737, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003456715901650377, + "before_rms": 0.003456715901650377, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005546723538161339, + "before_rms": 0.005546723538161339, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.014684442530549374, + "before_rms": 0.014684442530549374, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 15, + "token_id": 8437, + "input_position": 43, + "layers": [ + { + "after_rms": 0.00628374835469256, + "before_rms": 0.00628374835469256, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007316398861469263, + "before_rms": 0.007316398861469263, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008040298007985535, + "before_rms": 0.008040298007985535, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021912364803827578, + "before_rms": 0.021912364803827578, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 16, + "token_id": 2488, + "input_position": 44, + "layers": [ + { + "after_rms": 0.008596546950231054, + "before_rms": 0.008596546950231054, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005564570157863948, + "before_rms": 0.005564570157863948, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007757301988850941, + "before_rms": 0.007757301988850941, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020669327020220094, + "before_rms": 0.020669327020220094, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 17, + "token_id": 293, + "input_position": 45, + "layers": [ + { + "after_rms": 0.011432653860628939, + "before_rms": 0.011432653860628939, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005583320073300356, + "before_rms": 0.005583320073300356, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009875868656773605, + "before_rms": 0.009875868656773605, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024941505866581348, + "before_rms": 0.024941505866581348, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 18, + "token_id": 6999, + "input_position": 46, + "layers": [ + { + "after_rms": 0.0053301379768005384, + "before_rms": 0.0053301379768005384, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005000846356165002, + "before_rms": 0.005000846356165002, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005456690270258401, + "before_rms": 0.005456690270258401, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016748723982626335, + "before_rms": 0.016748723982626335, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 19, + "token_id": 981, + "input_position": 47, + "layers": [ + { + "after_rms": 0.007946814942115127, + "before_rms": 0.007946814942115127, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005158417494632955, + "before_rms": 0.005158417494632955, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007460860282340885, + "before_rms": 0.007460860282340885, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020411307476897476, + "before_rms": 0.020411307476897476, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 20, + "token_id": 13, + "input_position": 48, + "layers": [ + { + "after_rms": 0.005514051374901345, + "before_rms": 0.005514051374901345, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0070394921880856, + "before_rms": 0.0070394921880856, + "layer": 12, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0066148112542871755, + "before_rms": 0.0066148112542871755, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020639906847932543, + "before_rms": 0.020639906847932543, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "half", + "label": "Half FFN, scale 0.5", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Photosynthesis is the process by which plants use sunlight to create glucose, storing solar energy in chemical form.", + "metrics": { + "generated_tokens": 21, + "prompt_tokens": 29, + "elapsed_seconds": 2.160672568, + "request_wall_seconds": 2.2178376889787614, + "time_to_first_token_seconds": 0.6635361380176619, + "decode_tokens_per_second": 12.866885314377939, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0044633950081139125, + "block12_after_rms": 0.0022316975040569563 + }, + "check": { + "passed": true, + "rule": "sentence", + "expected": [ + "sunlight", + "glucose" + ], + "character_count": 116, + "word_count": 18 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0.5 + }, + "measurements": [ + { + "step": 0, + "token_id": 49106, + "input_position": 28, + "layers": [ + { + "after_rms": 0.004260129892907466, + "before_rms": 0.004260129892907466, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.0022316975040569563, + "before_rms": 0.0044633950081139125, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.005735151133850015, + "before_rms": 0.005735151133850015, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.01486159838115774, + "before_rms": 0.01486159838115774, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 237568 + } + ] + }, + { + "step": 1, + "token_id": 25424, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0056475146712665295, + "before_rms": 0.0056475146712665295, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004306811311193241, + "before_rms": 0.008613622622386482, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007436545118137283, + "before_rms": 0.007436545118137283, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020810310578413545, + "before_rms": 0.020810310578413545, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 332, + "input_position": 30, + "layers": [ + { + "after_rms": 0.0064819400327529395, + "before_rms": 0.0064819400327529395, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0033381018573591602, + "before_rms": 0.0066762037147183205, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009007999001296582, + "before_rms": 0.009007999001296582, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.026166335899804105, + "before_rms": 0.026166335899804105, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 269, + "input_position": 31, + "layers": [ + { + "after_rms": 0.0045512880200197945, + "before_rms": 0.0045512880200197945, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002532710148554094, + "before_rms": 0.005065420297108188, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005911283538375086, + "before_rms": 0.005911283538375086, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016813015849254234, + "before_rms": 0.016813015849254234, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1260, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004939956954683621, + "before_rms": 0.004939956954683621, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0019071381862013192, + "before_rms": 0.0038142763724026384, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00595557221350112, + "before_rms": 0.00595557221350112, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017146784199957084, + "before_rms": 0.017146784199957084, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 491, + "input_position": 33, + "layers": [ + { + "after_rms": 0.00603825473034925, + "before_rms": 0.00603825473034925, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0026995530717556144, + "before_rms": 0.005399106143511229, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008767531752803006, + "before_rms": 0.008767531752803006, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.022466743651620685, + "before_rms": 0.022466743651620685, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 687, + "input_position": 34, + "layers": [ + { + "after_rms": 0.004825344575484225, + "before_rms": 0.004825344575484225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0017861278816737752, + "before_rms": 0.0035722557633475504, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.004811283125730314, + "before_rms": 0.004811283125730314, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.014872505126769303, + "before_rms": 0.014872505126769303, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 7370, + "input_position": 35, + "layers": [ + { + "after_rms": 0.00599941128253067, + "before_rms": 0.00599941128253067, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002771895346350965, + "before_rms": 0.00554379069270193, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006169826908450906, + "before_rms": 0.006169826908450906, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02094382532383083, + "before_rms": 0.02094382532383083, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 896, + "input_position": 36, + "layers": [ + { + "after_rms": 0.0051950504054689125, + "before_rms": 0.0051950504054689125, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0030709468615872193, + "before_rms": 0.006141893723174439, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009589415026818995, + "before_rms": 0.009589415026818995, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.023335435813653795, + "before_rms": 0.023335435813653795, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 9, + "token_id": 24044, + "input_position": 37, + "layers": [ + { + "after_rms": 0.008031516167890848, + "before_rms": 0.008031516167890848, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0026840819143571647, + "before_rms": 0.005368163828714329, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005801143981718462, + "before_rms": 0.005801143981718462, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01859743623390119, + "before_rms": 0.01859743623390119, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 10, + "token_id": 295, + "input_position": 38, + "layers": [ + { + "after_rms": 0.009314847401164192, + "before_rms": 0.009314847401164192, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.003709490580960945, + "before_rms": 0.00741898116192189, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00977155465584999, + "before_rms": 0.00977155465584999, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02491673002871025, + "before_rms": 0.02491673002871025, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 11, + "token_id": 1815, + "input_position": 39, + "layers": [ + { + "after_rms": 0.0051360181463417694, + "before_rms": 0.0051360181463417694, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002573328181243894, + "before_rms": 0.005146656362487788, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005799911492815732, + "before_rms": 0.005799911492815732, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016933047785703814, + "before_rms": 0.016933047785703814, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 12, + "token_id": 18175, + "input_position": 40, + "layers": [ + { + "after_rms": 0.004435782376176152, + "before_rms": 0.004435782376176152, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0036635341240811403, + "before_rms": 0.007327068248162281, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00613417633614374, + "before_rms": 0.00613417633614374, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019118337346322598, + "before_rms": 0.019118337346322598, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 13, + "token_id": 11, + "input_position": 41, + "layers": [ + { + "after_rms": 0.010509592928354338, + "before_rms": 0.010509592928354338, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0036133829229371872, + "before_rms": 0.0072267658458743745, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007475141941789141, + "before_rms": 0.007475141941789141, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.025868505159618332, + "before_rms": 0.025868505159618332, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 14, + "token_id": 20827, + "input_position": 42, + "layers": [ + { + "after_rms": 0.0036465031366290737, + "before_rms": 0.0036465031366290737, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0017283579508251886, + "before_rms": 0.003456715901650377, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00533638225204982, + "before_rms": 0.00533638225204982, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.014436190642373524, + "before_rms": 0.014436190642373524, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 15, + "token_id": 8437, + "input_position": 43, + "layers": [ + { + "after_rms": 0.00628374835469256, + "before_rms": 0.00628374835469256, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0036581994307346316, + "before_rms": 0.007316398861469263, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008102150834431271, + "before_rms": 0.008102150834431271, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020984366848385224, + "before_rms": 0.020984366848385224, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 16, + "token_id": 2488, + "input_position": 44, + "layers": [ + { + "after_rms": 0.008596546950231054, + "before_rms": 0.008596546950231054, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002782285078931974, + "before_rms": 0.005564570157863948, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007806100403261486, + "before_rms": 0.007806100403261486, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020226905469513747, + "before_rms": 0.020226905469513747, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 17, + "token_id": 293, + "input_position": 45, + "layers": [ + { + "after_rms": 0.011432653860628939, + "before_rms": 0.011432653860628939, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002791660036650178, + "before_rms": 0.005583320073300356, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009973486185624623, + "before_rms": 0.009973486185624623, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024628251420628085, + "before_rms": 0.024628251420628085, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 18, + "token_id": 6999, + "input_position": 46, + "layers": [ + { + "after_rms": 0.0053301379768005384, + "before_rms": 0.0053301379768005384, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.002500423178082501, + "before_rms": 0.005000846356165002, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0053801173686054875, + "before_rms": 0.0053801173686054875, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016217885602163445, + "before_rms": 0.016217885602163445, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 19, + "token_id": 981, + "input_position": 47, + "layers": [ + { + "after_rms": 0.007946814942115127, + "before_rms": 0.007946814942115127, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0025792087473164773, + "before_rms": 0.005158417494632955, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007857079141286923, + "before_rms": 0.007857079141286923, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020573162868035226, + "before_rms": 0.020573162868035226, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 20, + "token_id": 13, + "input_position": 48, + "layers": [ + { + "after_rms": 0.005514051374901345, + "before_rms": 0.005514051374901345, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0035197460940428, + "before_rms": 0.0070394921880856, + "layer": 12, + "scale": 0.5, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006700234982153566, + "before_rms": 0.006700234982153566, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019504194134602675, + "before_rms": 0.019504194134602675, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + }, + { + "id": "zero", + "label": "Zero FFN, scale 0", + "model": "Recursive Flint", + "quant": "Q8_0", + "output": "Photosynthesis is the process by which plants use sunlight to convert carbon dioxide and water into glucose, releasing oxygen as a byproduct.", + "metrics": { + "generated_tokens": 25, + "prompt_tokens": 29, + "elapsed_seconds": 2.391091285, + "request_wall_seconds": 2.459595420048572, + "time_to_first_token_seconds": 0.7541828820249066, + "decode_tokens_per_second": 14.071746536403907, + "process_peak_rss_kib": 41801304, + "termination": "eos", + "complete": true, + "objective_check_passed": true, + "checkpoint_bytes": 42350359392, + "block12_before_rms": 0.0044633950081139125, + "block12_after_rms": 0.0 + }, + "check": { + "passed": true, + "rule": "sentence", + "expected": [ + "sunlight", + "glucose" + ], + "character_count": 141, + "word_count": 22 + }, + "settings": { + "max_tokens": 64, + "temperature": 0, + "seed": 42, + "reasoning": false, + "intervention_layer": 12, + "intervention_scale": 0 + }, + "measurements": [ + { + "step": 0, + "token_id": 49106, + "input_position": 28, + "layers": [ + { + "after_rms": 0.004260129892907466, + "before_rms": 0.004260129892907466, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.0, + "before_rms": 0.0044633950081139125, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.005661502074284029, + "before_rms": 0.005661502074284029, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 237568 + }, + { + "after_rms": 0.014899536247766184, + "before_rms": 0.014899536247766184, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 237568 + } + ] + }, + { + "step": 1, + "token_id": 25424, + "input_position": 29, + "layers": [ + { + "after_rms": 0.0056475146712665295, + "before_rms": 0.0056475146712665295, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.008613622622386482, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008525602935404572, + "before_rms": 0.008525602935404572, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019733810862655715, + "before_rms": 0.019733810862655715, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 2, + "token_id": 332, + "input_position": 30, + "layers": [ + { + "after_rms": 0.0064819400327529395, + "before_rms": 0.0064819400327529395, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0066762037147183205, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009135572197523601, + "before_rms": 0.009135572197523601, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02557741137246633, + "before_rms": 0.02557741137246633, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 3, + "token_id": 269, + "input_position": 31, + "layers": [ + { + "after_rms": 0.0045512880200197945, + "before_rms": 0.0045512880200197945, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005065420297108188, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006092569880549685, + "before_rms": 0.006092569880549685, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.016503642792897465, + "before_rms": 0.016503642792897465, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 4, + "token_id": 1260, + "input_position": 32, + "layers": [ + { + "after_rms": 0.004939956954683621, + "before_rms": 0.004939956954683621, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0038142763724026384, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006157849949139536, + "before_rms": 0.006157849949139536, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017203518557001942, + "before_rms": 0.017203518557001942, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 5, + "token_id": 491, + "input_position": 33, + "layers": [ + { + "after_rms": 0.00603825473034925, + "before_rms": 0.00603825473034925, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005399106143511229, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00904802642504343, + "before_rms": 0.00904802642504343, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02225766191781135, + "before_rms": 0.02225766191781135, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 6, + "token_id": 687, + "input_position": 34, + "layers": [ + { + "after_rms": 0.004825344575484225, + "before_rms": 0.004825344575484225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.0035722557633475504, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005533683753255847, + "before_rms": 0.005533683753255847, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.014873267868110489, + "before_rms": 0.014873267868110489, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 7, + "token_id": 7370, + "input_position": 35, + "layers": [ + { + "after_rms": 0.00599941128253067, + "before_rms": 0.00599941128253067, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.00554379069270193, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006390727713863449, + "before_rms": 0.006390727713863449, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02091061618332828, + "before_rms": 0.02091061618332828, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 8, + "token_id": 896, + "input_position": 36, + "layers": [ + { + "after_rms": 0.0051950504054689125, + "before_rms": 0.0051950504054689125, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.006141893723174439, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009947159109400337, + "before_rms": 0.009947159109400337, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.023151455733089174, + "before_rms": 0.023151455733089174, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 9, + "token_id": 24044, + "input_position": 37, + "layers": [ + { + "after_rms": 0.008031516167890848, + "before_rms": 0.008031516167890848, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005368163828714329, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006423750612719223, + "before_rms": 0.006423750612719223, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.019295778586505254, + "before_rms": 0.019295778586505254, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 10, + "token_id": 295, + "input_position": 38, + "layers": [ + { + "after_rms": 0.009314847401164192, + "before_rms": 0.009314847401164192, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.00741898116192189, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.009693459099318338, + "before_rms": 0.009693459099318338, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024237413637575162, + "before_rms": 0.024237413637575162, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 11, + "token_id": 6766, + "input_position": 39, + "layers": [ + { + "after_rms": 0.0051360181463417694, + "before_rms": 0.0051360181463417694, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005146656362487788, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005767883251008539, + "before_rms": 0.005767883251008539, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.01708687035677014, + "before_rms": 0.01708687035677014, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 12, + "token_id": 7185, + "input_position": 40, + "layers": [ + { + "after_rms": 0.0073586153911031, + "before_rms": 0.0073586153911031, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.006591188015659504, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0070010157973419635, + "before_rms": 0.0070010157973419635, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.022579139951650742, + "before_rms": 0.022579139951650742, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 13, + "token_id": 23776, + "input_position": 41, + "layers": [ + { + "after_rms": 0.008194359117077644, + "before_rms": 0.008194359117077644, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.007596196041918696, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0075531294632927015, + "before_rms": 0.0075531294632927015, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021887967030610505, + "before_rms": 0.021887967030610505, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 14, + "token_id": 299, + "input_position": 42, + "layers": [ + { + "after_rms": 0.007857027218294529, + "before_rms": 0.007857027218294529, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.008314784468177716, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.008147101435200647, + "before_rms": 0.008147101435200647, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.02264674988179593, + "before_rms": 0.02264674988179593, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 15, + "token_id": 1952, + "input_position": 43, + "layers": [ + { + "after_rms": 0.004688345103889225, + "before_rms": 0.004688345103889225, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.004946909109298921, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006079777076870683, + "before_rms": 0.006079777076870683, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015899963838293046, + "before_rms": 0.015899963838293046, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 16, + "token_id": 831, + "input_position": 44, + "layers": [ + { + "after_rms": 0.007450920463116443, + "before_rms": 0.007450920463116443, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.006978048184271669, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006659637268169784, + "before_rms": 0.006659637268169784, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020155346635423232, + "before_rms": 0.020155346635423232, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 17, + "token_id": 18175, + "input_position": 45, + "layers": [ + { + "after_rms": 0.007443990344351715, + "before_rms": 0.007443990344351715, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005543317793659353, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.00888447540290056, + "before_rms": 0.00888447540290056, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.020591210203462766, + "before_rms": 0.020591210203462766, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 18, + "token_id": 11, + "input_position": 46, + "layers": [ + { + "after_rms": 0.009927458171845966, + "before_rms": 0.009927458171845966, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.006904034655866483, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007498876939890208, + "before_rms": 0.007498876939890208, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024867085464211167, + "before_rms": 0.024867085464211167, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 19, + "token_id": 27762, + "input_position": 47, + "layers": [ + { + "after_rms": 0.003583892367306353, + "before_rms": 0.003583892367306353, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.003674512887912858, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.005366533667651147, + "before_rms": 0.005366533667651147, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.015057416354372646, + "before_rms": 0.015057416354372646, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 20, + "token_id": 11472, + "input_position": 48, + "layers": [ + { + "after_rms": 0.0071015945130851466, + "before_rms": 0.0071015945130851466, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.006051436756781676, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006862596596600148, + "before_rms": 0.006862596596600148, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021336925675778752, + "before_rms": 0.021336925675778752, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 21, + "token_id": 389, + "input_position": 49, + "layers": [ + { + "after_rms": 0.007580029165171899, + "before_rms": 0.007580029165171899, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.006902773819282416, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007204155372984763, + "before_rms": 0.007204155372984763, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.021924978969430717, + "before_rms": 0.021924978969430717, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 22, + "token_id": 260, + "input_position": 50, + "layers": [ + { + "after_rms": 0.00546253050622268, + "before_rms": 0.00546253050622268, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.005206685222875008, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.006133790801518977, + "before_rms": 0.006133790801518977, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.017978821140417742, + "before_rms": 0.017978821140417742, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 23, + "token_id": 93485, + "input_position": 51, + "layers": [ + { + "after_rms": 0.00462712758092533, + "before_rms": 0.00462712758092533, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.004455167000851032, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0069830694392977915, + "before_rms": 0.0069830694392977915, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0157750242951739, + "before_rms": 0.0157750242951739, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + }, + { + "step": 24, + "token_id": 13, + "input_position": 52, + "layers": [ + { + "after_rms": 0.010873441338987835, + "before_rms": 0.010873441338987835, + "layer": 11, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.0, + "before_rms": 0.006011455301735444, + "layer": 12, + "scale": 0.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.007365985817524404, + "before_rms": 0.007365985817524404, + "layer": 13, + "scale": 1.0, + "tensor": "ffn_out", + "tensor_bytes": 8192 + }, + { + "after_rms": 0.024886336584602996, + "before_rms": 0.024886336584602996, + "layer": 13, + "scale": 1.0, + "tensor": "l_out", + "tensor_bytes": 8192 + } + ] + } + ] + } + ], + "comparison": { + "untargeted_first_frame_block11_identical": true, + "downstream_first_frame_block13_changed": true, + "weights_modified": false, + "all_saved_frames_available_in_results": true + } + } + ] + }, + "memory": { + "title": "Remembered, but outdated?", + "summary": "Six short fictional logs test current facts and original facts. Flint and alphaBell Qwen Open answered all six correctly. Predict the answer, then inspect each recorded result.", + "method": "Three log structures, each queried for the current and original access code. Exact four-digit answer scoring. Same decoding settings for both models.", + "limitations": [ + "These prompts are at most 144 tokens. They test short in-context correction and recall, not long-context performance or permanent memory." + ], + "cases": [ + { + "id": "distractors-current", + "title": "A correction among distractors: current fact", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "question": "What is the current museum access code?", + "ground_truth": "8624", + "choices": [ + "4319", + "8624", + "5972" + ], + "kind": "current", + "conditions": [ + { + "id": "flint", + "label": "Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "output": "8624", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=memory-distractors-current-flint&token=1", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 126, + "elapsed_seconds": 1.662128068, + "termination": "eos", + "exact_answer_passed": true + } + }, + { + "id": "qwen", + "label": "alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "output": "8624", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=memory-distractors-current-qwen&token=1", + "metrics": { + "generated_tokens": 4, + "prompt_tokens": 144, + "elapsed_seconds": 10.655367113, + "termination": "eos", + "exact_answer_passed": true + } + } + ], + "checks": [ + "Expected four-digit answer: 8624", + "Recursive Flint: passed", + "alphaBell Qwen Open 27B: passed" + ] + }, + { + "id": "distractors-original", + "title": "A correction among distractors: original fact", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "question": "What is the original museum access code?", + "ground_truth": "4319", + "choices": [ + "4319", + "8624", + "5972" + ], + "kind": "original", + "conditions": [ + { + "id": "flint", + "label": "Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "output": "4319", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=memory-distractors-original-flint&token=1", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 126, + "elapsed_seconds": 1.715333822, + "termination": "eos", + "exact_answer_passed": true + } + }, + { + "id": "qwen", + "label": "alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "output": "4319", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=memory-distractors-original-qwen&token=1", + "metrics": { + "generated_tokens": 4, + "prompt_tokens": 144, + "elapsed_seconds": 10.655514738, + "termination": "eos", + "exact_answer_passed": true + } + } + ], + "checks": [ + "Expected four-digit answer: 4319", + "Recursive Flint: passed", + "alphaBell Qwen Open 27B: passed" + ] + }, + { + "id": "short-current", + "title": "One correction: current fact", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "question": "What is the current museum access code?", + "ground_truth": "8624", + "choices": [ + "4319", + "8624", + "5972" + ], + "kind": "current", + "conditions": [ + { + "id": "flint", + "label": "Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "output": "8624", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=memory-short-current-flint&token=1", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 67, + "elapsed_seconds": 1.14846904, + "termination": "eos", + "exact_answer_passed": true + } + }, + { + "id": "qwen", + "label": "alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "output": "8624", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=memory-short-current-qwen&token=1", + "metrics": { + "generated_tokens": 4, + "prompt_tokens": 85, + "elapsed_seconds": 7.286205131, + "termination": "eos", + "exact_answer_passed": true + } + } + ], + "checks": [ + "Expected four-digit answer: 8624", + "Recursive Flint: passed", + "alphaBell Qwen Open 27B: passed" + ] + }, + { + "id": "short-original", + "title": "One correction: original fact", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "question": "What is the original museum access code?", + "ground_truth": "4319", + "choices": [ + "4319", + "8624", + "5972" + ], + "kind": "original", + "conditions": [ + { + "id": "flint", + "label": "Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "output": "4319", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=memory-short-original-flint&token=1", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 67, + "elapsed_seconds": 1.1675653, + "termination": "eos", + "exact_answer_passed": true + } + }, + { + "id": "qwen", + "label": "alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "output": "4319", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=memory-short-original-qwen&token=1", + "metrics": { + "generated_tokens": 4, + "prompt_tokens": 85, + "elapsed_seconds": 7.234881181, + "termination": "eos", + "exact_answer_passed": true + } + } + ], + "checks": [ + "Expected four-digit answer: 4319", + "Recursive Flint: passed", + "alphaBell Qwen Open 27B: passed" + ] + }, + { + "id": "twice-current", + "title": "Two successive corrections: current fact", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "question": "What is the current museum access code?", + "ground_truth": "5972", + "choices": [ + "4319", + "8624", + "5972" + ], + "kind": "current", + "conditions": [ + { + "id": "flint", + "label": "Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "output": "5972", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=memory-twice-current-flint&token=1", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 105, + "elapsed_seconds": 1.501386008, + "termination": "eos", + "exact_answer_passed": true + } + }, + { + "id": "qwen", + "label": "alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "output": "5972", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=memory-twice-current-qwen&token=1", + "metrics": { + "generated_tokens": 4, + "prompt_tokens": 125, + "elapsed_seconds": 9.111306724, + "termination": "eos", + "exact_answer_passed": true + } + } + ], + "checks": [ + "Expected four-digit answer: 5972", + "Recursive Flint: passed", + "alphaBell Qwen Open 27B: passed" + ] + }, + { + "id": "twice-original", + "title": "Two successive corrections: original fact", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "question": "What is the original museum access code?", + "ground_truth": "4319", + "choices": [ + "4319", + "8624", + "5972" + ], + "kind": "original", + "conditions": [ + { + "id": "flint", + "label": "Recursive Flint", + "model": "Recursive Flint", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "output": "4319", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=flint&capture=memory-twice-original-flint&token=1", + "metrics": { + "generated_tokens": 2, + "prompt_tokens": 105, + "elapsed_seconds": 1.648641691, + "termination": "eos", + "exact_answer_passed": true + } + }, + { + "id": "qwen", + "label": "alphaBell Qwen Open 27B", + "model": "alphaBell Qwen Open 27B", + "quant": "Q8_0", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "output": "4319", + "settings": { + "max_tokens": 16, + "temperature": 0, + "seed": 42, + "reasoning": false + }, + "trace_url": "../index.html?model=qwen&capture=memory-twice-original-qwen&token=1", + "metrics": { + "generated_tokens": 4, + "prompt_tokens": 125, + "elapsed_seconds": 9.107945102, + "termination": "eos", + "exact_answer_passed": true + } + } + ], + "checks": [ + "Expected four-digit answer: 4319", + "Recursive Flint: passed", + "alphaBell Qwen Open 27B: passed" + ] + } + ] + } + } +} diff --git a/experiments/data/summary.json b/experiments/data/summary.json new file mode 100644 index 0000000000000000000000000000000000000000..cb012ab3aff4294ec702b95d1e58d70773e20d9b --- /dev/null +++ b/experiments/data/summary.json @@ -0,0 +1,18 @@ +{ + "capture_runs": 24, + "quant_runs": 24, + "surgery_conditions": 18, + "surgery_additional_runs": 12, + "unique_measured_runs": 60, + "quant_pairs": 12, + "identical_quant_pairs": 11, + "memory_answers_passed": 12, + "memory_answers_total": 12, + "full_capture_frames": 80, + "first_frame_changed_expert_blocks": { + "json-color-flint": 18, + "opposite-flint": 25, + "python-flint": 20 + }, + "results_sha256": "b60b8d8b302e5077c01a738cc667d94ef7768ce8085c8dd0636b1d7068c9a140" +} diff --git a/experiments/data/traces.json b/experiments/data/traces.json new file mode 100644 index 0000000000000000000000000000000000000000..db38b557d8320806c70105ffafdb122c928803e2 --- /dev/null +++ b/experiments/data/traces.json @@ -0,0 +1,314 @@ +[ + { + "id": "memory-distractors-current-flint", + "model": "flint", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "tokens": 2, + "file": "traces/experiments/memory-distractors-current-flint.json.gz", + "sha256": "5afd29ce9f49160b2f35f8f9dcb880848a52722c2e59e7887e06185f13922777", + "gzip_sha256": "9fb651c14a66e00268d4814e874e9073a97aa391fa776c69f8b7146420547d8f", + "bytes": 960941, + "compressed_bytes": 372753, + "created": 1792753004575, + "label": "A correction among distractors: current fact \u00b7 flint" + }, + { + "id": "memory-distractors-current-qwen", + "model": "qwen", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "tokens": 4, + "file": "traces/experiments/memory-distractors-current-qwen.json.gz", + "sha256": "4912dac302ed9cf3075d82423964d7f2cb78fd37cef9b02771cddf7f94121238", + "gzip_sha256": "31b0d9a35744a6a4563d128763ea43535be5120ec6ea8a635bd0e21a009d64bf", + "bytes": 1205054, + "compressed_bytes": 518952, + "created": 1792753006737, + "label": "A correction among distractors: current fact \u00b7 qwen" + }, + { + "id": "memory-distractors-original-flint", + "model": "flint", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "tokens": 2, + "file": "traces/experiments/memory-distractors-original-flint.json.gz", + "sha256": "117d4f848f94ee889eeeeba4334c8e913f08021e3858428622f12d1808d30287", + "gzip_sha256": "0dade5419d2dcce1886350aa7b6f9d006f980c246dce74be9e52c4a96524509a", + "bytes": 960809, + "compressed_bytes": 373102, + "created": 1792753018956, + "label": "A correction among distractors: original fact \u00b7 flint" + }, + { + "id": "memory-distractors-original-qwen", + "model": "qwen", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "tokens": 4, + "file": "traces/experiments/memory-distractors-original-qwen.json.gz", + "sha256": "41adb8cdd9b1be4587d767ee73513a15b8fff5b7d32e27a6dba6c0fee1eb60a5", + "gzip_sha256": "faa06f0a66ddd34bbb520f85179239a500e10b51ef3b79522bfd5ced0ed90b3d", + "bytes": 1203653, + "compressed_bytes": 518535, + "created": 1792753021157, + "label": "A correction among distractors: original fact \u00b7 qwen" + }, + { + "id": "memory-short-current-flint", + "model": "flint", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "tokens": 2, + "file": "traces/experiments/memory-short-current-flint.json.gz", + "sha256": "6e1345ee29f71094f8d51982f2ed905fbd0be51a26b3780152fc78ed004d45c7", + "gzip_sha256": "1c857156e7a8284e79de717741508a6eab5ea099e6bc42c49bb52ee799091b32", + "bytes": 943320, + "compressed_bytes": 372203, + "created": 1792752983606, + "label": "One correction: current fact \u00b7 flint" + }, + { + "id": "memory-short-current-qwen", + "model": "qwen", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "tokens": 4, + "file": "traces/experiments/memory-short-current-qwen.json.gz", + "sha256": "f91be43725b8603628494a4b71bfb76888d1f69165295b98209857d10e4a06e5", + "gzip_sha256": "2177ad2e9e2d62726f2e94e339c9423cf7c5cf12c1010fe89b2f0ea9d8a9360d", + "bytes": 1195637, + "compressed_bytes": 517455, + "created": 1792752985768, + "label": "One correction: current fact \u00b7 qwen" + }, + { + "id": "memory-short-original-flint", + "model": "flint", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "tokens": 2, + "file": "traces/experiments/memory-short-original-flint.json.gz", + "sha256": "34e0fdef73702915345d5a00ceb2b97407c3d3403d33bdeffbe1e4df6260baac", + "gzip_sha256": "e730312e416fa2331342bc02277b13e57ee3bb3941c2f0acdfddf86214191a2e", + "bytes": 943171, + "compressed_bytes": 372679, + "created": 1792752994024, + "label": "One correction: original fact \u00b7 flint" + }, + { + "id": "memory-short-original-qwen", + "model": "qwen", + "prompt": "Read this fictional log. The original museum access code is 4319. The museum access code is now 8624. The old code is no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "tokens": 4, + "file": "traces/experiments/memory-short-original-qwen.json.gz", + "sha256": "5e8b91b89677b0d014f3a705cbca017b559984960928b11c0b28241a7eb792e0", + "gzip_sha256": "afefafa284bf44fdfa36cadd5493ba204afd1a9ba93870525b272f7f452d3829", + "bytes": 1195602, + "compressed_bytes": 517420, + "created": 1792752996323, + "label": "One correction: original fact \u00b7 qwen" + }, + { + "id": "memory-twice-current-flint", + "model": "flint", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "tokens": 2, + "file": "traces/experiments/memory-twice-current-flint.json.gz", + "sha256": "d15d5a0d54ab5283f053b2671fbeb018a4cb58f40fc12fc045773122a5a177e6", + "gzip_sha256": "21b7e8fbdb5eb767cf71c9780b2763cf1f0723f2a5f2854a2ac4403b11c28475", + "bytes": 954308, + "compressed_bytes": 373646, + "created": 1792753033494, + "label": "Two successive corrections: current fact \u00b7 flint" + }, + { + "id": "memory-twice-current-qwen", + "model": "qwen", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the current museum access code? Answer with only the four digits.", + "tokens": 4, + "file": "traces/experiments/memory-twice-current-qwen.json.gz", + "sha256": "20cc6fc733b5b12a5b2257f6ae28b24979a0096465868020e9101f60131a2ef6", + "gzip_sha256": "e6dac7b0201e32adc60202638f0f0c84ed2f257c065513c4649d5ab39610a6af", + "bytes": 1201186, + "compressed_bytes": 518453, + "created": 1792753035778, + "label": "Two successive corrections: current fact \u00b7 qwen" + }, + { + "id": "memory-twice-original-flint", + "model": "flint", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "tokens": 2, + "file": "traces/experiments/memory-twice-original-flint.json.gz", + "sha256": "ff79ad7fc15b469c91b7d959b4c86d5242feb8e83b375d0196700d940c023fe8", + "gzip_sha256": "15f3245da6145e40c7c3b56722bd78ebcd26f0f4363ef64c153a538ad1a7cf8e", + "bytes": 954081, + "compressed_bytes": 373616, + "created": 1792753046085, + "label": "Two successive corrections: original fact \u00b7 flint" + }, + { + "id": "memory-twice-original-qwen", + "model": "qwen", + "prompt": "Read this fictional log. The original museum access code is 4319. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. A crate is sealed. A lamp is on. The museum access code changes to 8624. Later it changes again to 5972. The earlier codes are no longer current. Question: What is the original museum access code? Answer with only the four digits.", + "tokens": 4, + "file": "traces/experiments/memory-twice-original-qwen.json.gz", + "sha256": "5490a9f4775c6de2478a68b85327c007c625ac5384691eb2c28b904661ad1c8b", + "gzip_sha256": "3a0baf879172d8a97353305063312519a98d1c97151400743b2b0e11fab58d74", + "bytes": 1202381, + "compressed_bytes": 518903, + "created": 1792753048335, + "label": "Two successive corrections: original fact \u00b7 qwen" + }, + { + "id": "reroute-json-color-flint-a", + "model": "flint", + "prompt": "Return only JSON with one key named color and value blue.", + "tokens": 5, + "file": "traces/experiments/reroute-json-color-flint-a.json.gz", + "sha256": "ff0da38c0292fe7f8e46a2e8a932ccb983e69877327081f738e212ad8100ea25", + "gzip_sha256": "a1a2c501c40294dcb039a15c115513850d55a69bea61e1fe2fca715acae43ebe", + "bytes": 2346341, + "compressed_bytes": 919171, + "created": 1792752946971, + "label": "One word changes the JSON value \u00b7 a" + }, + { + "id": "reroute-json-color-flint-b", + "model": "flint", + "prompt": "Return only JSON with one key named color and value red.", + "tokens": 5, + "file": "traces/experiments/reroute-json-color-flint-b.json.gz", + "sha256": "f9f13339bdc7df2f3fd42d40a9b87d04cf531c752c2a99f943093d058fea616b", + "gzip_sha256": "a6de8843f680c529d5123fff8bc1c663fdb59d4b9018308bdd41b4beb3e52213", + "bytes": 2346598, + "compressed_bytes": 919420, + "created": 1792752951404, + "label": "One word changes the JSON value \u00b7 b" + }, + { + "id": "reroute-json-color-qwen-a", + "model": "qwen", + "prompt": "Return only JSON with one key named color and value blue.", + "tokens": 5, + "file": "traces/experiments/reroute-json-color-qwen-a.json.gz", + "sha256": "a13f3e97372c12e6242c17777e7f0f4a27f01ec91d6ed03304979bd72c0549e0", + "gzip_sha256": "892c728cea74109639ecf7d125e6677df9ed9f2c68c32a5292204c200ec47bbe", + "bytes": 1490772, + "compressed_bytes": 643912, + "created": 1792752953882, + "label": "One word changes the JSON value \u00b7 a" + }, + { + "id": "reroute-json-color-qwen-b", + "model": "qwen", + "prompt": "Return only JSON with one key named color and value red.", + "tokens": 5, + "file": "traces/experiments/reroute-json-color-qwen-b.json.gz", + "sha256": "c1d3757878ef7c54bcfeb06d0658fca745a39d59fba2aeabf20e709f0fff532c", + "gzip_sha256": "b87a820c6240ff199010edefd907d98cb2340122f670aaaac3c3182c6f52af51", + "bytes": 1491042, + "compressed_bytes": 643871, + "created": 1792752960146, + "label": "One word changes the JSON value \u00b7 b" + }, + { + "id": "reroute-opposite-flint-a", + "model": "flint", + "prompt": "Answer with one word: name the opposite of hot.", + "tokens": 1, + "file": "traces/experiments/reroute-opposite-flint-a.json.gz", + "sha256": "27eefa9070e87ee9c6139055d8b5ff8ab6eea0f7d5be48672fd7bf815fa998f8", + "gzip_sha256": "83f6529578d08dd395eabecd1f2e3d68adc06337feca4cd8270990a897a76f6e", + "bytes": 473055, + "compressed_bytes": 185653, + "created": 1792752934308, + "label": "One word changes the opposite \u00b7 a" + }, + { + "id": "reroute-opposite-flint-b", + "model": "flint", + "prompt": "Answer with one word: name the opposite of cold.", + "tokens": 1, + "file": "traces/experiments/reroute-opposite-flint-b.json.gz", + "sha256": "226b7b20822499069e0f582c56a563988526549967acde9c9367050e26bc4fd5", + "gzip_sha256": "6f378e33a36b498d171dde4e97d0b33f97643eaa3f035ab9c61f7ff7634d0c66", + "bytes": 473107, + "compressed_bytes": 185674, + "created": 1792752936544, + "label": "One word changes the opposite \u00b7 b" + }, + { + "id": "reroute-opposite-qwen-a", + "model": "qwen", + "prompt": "Answer with one word: name the opposite of hot.", + "tokens": 1, + "file": "traces/experiments/reroute-opposite-qwen-a.json.gz", + "sha256": "49939c0748d0ae66d5a51e7a1e4701cbbf663ae360c81d3faeabf5981cf312c9", + "gzip_sha256": "6090859303c7aa620c03e66ab3a5229e4c761f60119db663e0a777253d9aa6e2", + "bytes": 302818, + "compressed_bytes": 131424, + "created": 1792752938769, + "label": "One word changes the opposite \u00b7 a" + }, + { + "id": "reroute-opposite-qwen-b", + "model": "qwen", + "prompt": "Answer with one word: name the opposite of cold.", + "tokens": 1, + "file": "traces/experiments/reroute-opposite-qwen-b.json.gz", + "sha256": "4ddaa3b02ff685bccee1d6a947800842f9396da12b4623b356fef63106b84ff2", + "gzip_sha256": "c8cc0a1236967408295aadf413b5c10d3443857c56d8418b845b1343d9bb7453", + "bytes": 302398, + "compressed_bytes": 131168, + "created": 1792752942914, + "label": "One word changes the opposite \u00b7 b" + }, + { + "id": "reroute-python-flint-a", + "model": "flint", + "prompt": "Write only a Python expression that selects the maximum of x and y.", + "tokens": 5, + "file": "traces/experiments/reroute-python-flint-a.json.gz", + "sha256": "a80bb8703120bb432927555d1c599ea6a7769645a13414fb7d57e23c240e684e", + "gzip_sha256": "3d0bcff6db675e0a5450ac2744eb4031862eeb5844ae9ac0e3592ceef3875d57", + "bytes": 2347333, + "compressed_bytes": 919644, + "created": 1792752966319, + "label": "One word changes the function \u00b7 a" + }, + { + "id": "reroute-python-flint-b", + "model": "flint", + "prompt": "Write only a Python expression that selects the minimum of x and y.", + "tokens": 5, + "file": "traces/experiments/reroute-python-flint-b.json.gz", + "sha256": "80d072551a9135db60dc2ee6902731cd1bcb4843cc629c9c17038f4eeafbc586", + "gzip_sha256": "4a40ab0cdf7341870e8e378633e5377b4a6614e7fcaef307d69af94326d15c6d", + "bytes": 2347533, + "compressed_bytes": 920189, + "created": 1792752968702, + "label": "One word changes the function \u00b7 b" + }, + { + "id": "reroute-python-qwen-a", + "model": "qwen", + "prompt": "Write only a Python expression that selects the maximum of x and y.", + "tokens": 5, + "file": "traces/experiments/reroute-python-qwen-a.json.gz", + "sha256": "3eeb0196c7ad311743476653d65cb26d64561afd87452792f508f5371a59531e", + "gzip_sha256": "c52ca893eaf76cd8db897dd9b0b66f01545cc945c0edc562c08b2bb6aecd3bec", + "bytes": 1490170, + "compressed_bytes": 643193, + "created": 1792752971166, + "label": "One word changes the function \u00b7 a" + }, + { + "id": "reroute-python-qwen-b", + "model": "qwen", + "prompt": "Write only a Python expression that selects the minimum of x and y.", + "tokens": 5, + "file": "traces/experiments/reroute-python-qwen-b.json.gz", + "sha256": "ab952c5215dfc8dae2c2136497eca8c38d470f6169829bec1b8df571a4665a72", + "gzip_sha256": "23025da3491932313747db32ff949ac6cbd208171716ad624c7a5d847b03fa9f", + "bytes": 1489618, + "compressed_bytes": 642994, + "created": 1792752977371, + "label": "One word changes the function \u00b7 b" + } +] diff --git a/experiments/index.html b/experiments/index.html new file mode 100644 index 0000000000000000000000000000000000000000..0fcd504b300716b89b069bec99f2fd76b0a73892 --- /dev/null +++ b/experiments/index.html @@ -0,0 +1,22 @@ + + + + +Five ways to look inside a model | alphaBell Experiments + + + + + +
αalphaBell / experiments
+
+

THE ALPHABELL EXPERIMENT SERIES

Small changes.
Surprising models.

Change one word. Hide the precision. Correct a fact. Five hands-on experiments with real model outputs and measurements.

Make a prediction.
Then open the evidence.

Recorded model runs. Explore at your own pace.
+ +
Opening measured experiments…
+ +
+ + + + + diff --git a/experiments/lab.css b/experiments/lab.css new file mode 100644 index 0000000000000000000000000000000000000000..fb897bf7e215f0b6f61cfb6328edbcb0fbff3637 --- /dev/null +++ b/experiments/lab.css @@ -0,0 +1,3 @@ +:root{--bg:#f6f7f3;--paper:#fff;--ink:#24372f;--muted:#66756c;--line:#e0e7dd;--green:#176b59;--lime:#d5f59b;--mono:ui-monospace,SFMono-Regular,Consolas,monospace;--sans:system-ui,-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif}*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--ink);font:14px/1.6 var(--sans)}button,input,select{font:inherit}button,a,input,select,summary{-webkit-tap-highlight-color:transparent}a{color:var(--green);text-underline-offset:4px}button{color:inherit;cursor:pointer;border:1px solid var(--line);background:var(--paper);border-radius:7px;padding:10px 15px;min-height:44px}button:hover{border-color:#8dad9a;background:#f2f7ed}button:disabled{opacity:.45;cursor:default}button:focus-visible,a:focus-visible,input:focus-visible,select:focus-visible,summary:focus-visible{outline:3px solid var(--green);outline-offset:4px}[hidden]{display:none!important}header{display:flex;justify-content:space-between;align-items:center;gap:20px;padding:17px 36px;border-bottom:1px solid var(--line);background:var(--paper)}.brand{display:flex;align-items:center;gap:11px;color:var(--ink);text-decoration:none;font-size:15px}.brand b{font-weight:500}.brand-icon{display:grid;place-items:center;background:var(--green);color:var(--lime);width:35px;height:35px;border-radius:9px;font-size:25px}.brand-divider{margin:0 5px;color:#a1afa3}header nav{display:flex;gap:24px;font-size:12px}.skip{position:fixed;top:-80px;left:12px;z-index:20;background:white;padding:10px}.skip:focus{top:10px}main{max-width:1320px;margin:auto;padding:0 36px 56px}.hero{display:grid;grid-template-columns:minmax(0,1fr) 300px;align-items:center;gap:60px;padding:62px 0 42px}.eyebrow{font:10px/1.5 var(--mono);letter-spacing:1.4px;color:var(--green);margin:0 0 14px}h1{font-size:clamp(38px,5vw,66px);font-weight:500;letter-spacing:-2.7px;line-height:1.07;margin:0 0 22px}h1 span{color:var(--green)}.intro{max-width:610px;font-size:16px;color:var(--muted);margin:0;line-height:1.7}.hero-note{border-left:1px solid var(--line);padding:26px 0 26px 35px}.hero-note p{font-size:20px;line-height:1.4;margin:18px 0 13px}.hero-note>span:last-child{font-size:11px;color:var(--muted)}.signal{display:flex;align-items:center;gap:6px;height:37px}.signal i{display:block;width:8px;background:var(--green);border-radius:2px;height:35%}.signal i:nth-child(2){height:80%;background:#95bd75}.signal i:nth-child(3){height:55%}.signal i:nth-child(4){height:100%;background:#badf8d}.signal i:nth-child(5){height:70%}.signal i:nth-child(6){height:40%;background:#95bd75}.signal i:nth-child(7){height:65%}.experiment-grid{display:grid;grid-template-columns:repeat(5,minmax(0,1fr));gap:10px;margin-bottom:34px}.experiment-card{position:relative;display:flex;flex-direction:column;min-width:0;padding:18px 17px 16px;text-decoration:none;color:var(--ink);border:1px solid var(--line);border-radius:9px;background:var(--paper);transition:border-color .15s,transform .15s}.experiment-card:hover{border-color:#88aa93;transform:translateY(-2px)}.experiment-card[aria-current=page]{background:#e8f1df;border-color:#a5c28c}.card-number{font:10px var(--mono);color:var(--green);margin-bottom:22px}.experiment-card strong{font-size:15px;font-weight:500;line-height:1.35;min-height:40px}.experiment-card small{font-size:11px;color:var(--muted);line-height:1.5;margin-top:10px}.card-arrow{position:absolute;right:16px;top:14px;color:var(--green)}#experiment{scroll-margin-top:20px;outline:none}.experiment-head{display:flex;justify-content:space-between;gap:28px;align-items:flex-end;padding:30px 0 24px;border-top:1px solid var(--line)}h2{font-size:30px;font-weight:500;letter-spacing:-1px;line-height:1.2;margin:0 0 11px}.experiment-head p:not(.eyebrow){color:var(--muted);margin:0;max-width:700px}.case-picker{display:flex;flex-direction:column;gap:7px;min-width:195px;font-size:11px;color:var(--muted)}select{width:100%;border:1px solid var(--line);border-radius:6px;background:var(--paper);color:var(--ink);padding:10px 32px 10px 11px;min-height:44px;max-width:350px}.case-layout{background:var(--paper);border:1px solid var(--line);border-radius:11px;overflow:hidden}.case-top{display:flex;justify-content:space-between;gap:25px;padding:23px 26px;border-bottom:1px solid var(--line);background:#fcfdf9}.case-top h3{font-size:18px;font-weight:500;line-height:1.4;margin:0 0 8px}.case-top p{margin:0;color:var(--muted);font-size:12px}.tag{display:inline-flex;align-items:center;border:1px solid #d7e2cd;border-radius:5px;padding:4px 8px;background:#f1f6e9;font:10px var(--mono);white-space:nowrap;align-self:flex-start;color:#4b6e41}.prompt{background:#f5f7f1;border:1px solid var(--line);padding:15px 17px;border-radius:7px;white-space:pre-wrap;overflow-wrap:anywhere;font:12px/1.8 var(--mono);margin:12px 0 0;color:#4e6557}.case-body{padding:25px}.columns{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:20px}.columns.three{grid-template-columns:repeat(3,minmax(0,1fr))}.condition{min-width:0;border:1px solid var(--line);border-radius:8px;overflow:hidden;background:white}.condition-heading{padding:15px 17px;border-bottom:1px solid var(--line);display:flex;justify-content:space-between;gap:12px;align-items:center}.condition-heading h4{font-size:13px;font-weight:600;margin:0}.condition-content{padding:17px}.output{white-space:pre-wrap;overflow-wrap:anywhere;font-size:13px;line-height:1.8;margin:0;max-height:370px;overflow:auto;scrollbar-width:thin}.condition .prompt{margin:0 0 16px;font-size:11px}.metrics{display:grid;grid-template-columns:repeat(auto-fit,minmax(115px,1fr));gap:14px 17px;margin:19px 0 0;padding-top:17px;border-top:1px solid var(--line)}.metrics div{min-width:0}.metrics dt{font-size:10px;color:var(--muted);overflow-wrap:anywhere}.metrics dd{margin:5px 0 0;font:12px var(--mono);color:var(--green);overflow-wrap:anywhere}.annotation{font-size:12px;color:var(--muted);margin:18px 0;line-height:1.8}.annotation strong{color:var(--ink);font-weight:500}.notice{padding:15px 17px;border-radius:7px;background:#edf4e6;border:1px solid #dbe8cf;font-size:12px;line-height:1.8;margin-bottom:20px}.notice strong{font-weight:600}.controls{display:flex;align-items:center;flex-wrap:wrap;gap:10px;margin-top:20px}.controls button.primary,.primary{background:var(--green);color:#fff;border-color:var(--green)}.controls button.primary:hover,.primary:hover{background:#105947}.controls .quiet,button.quiet{background:transparent;color:var(--muted);font-size:12px}.share{font-size:11px;white-space:nowrap}.result-note{padding:16px 18px;border:1px solid #c8dfb7;background:#edf6e4;border-radius:7px;margin-top:19px;font-size:13px}.result-note p{margin:7px 0 0;font-size:12px;color:var(--muted)}.question{font-size:19px;font-weight:500;line-height:1.4;margin:0 0 17px}.choice-row{display:flex;gap:10px;flex-wrap:wrap}.choice-row button{flex:1;min-width:145px;text-align:center}.choice-row button[aria-pressed=true]{background:#e2f1cf;border-color:#8bb96c;color:#214c35}.guess-panel{padding:22px;background:#f5f8ef;border:1px solid var(--line);border-radius:8px;margin-top:22px}.local-tally{font:10px/1.8 var(--mono);color:var(--muted);margin:13px 0 0}.token-controls{display:flex;align-items:center;gap:12px;flex-wrap:wrap;background:#f5f8ef;border:1px solid var(--line);border-radius:7px;padding:14px 16px;margin:0 0 20px}.token-controls label{font:11px var(--mono)}.token-controls input{flex:1;min-width:100px;accent-color:var(--green)}.token-controls button{padding:4px 10px;min-height:34px}.token-readout{font:11px var(--mono);color:var(--green);white-space:nowrap}.expert-grid{display:grid;grid-template-columns:repeat(32,minmax(0,1fr));gap:3px;margin:12px 0 10px}.expert-dot{aspect-ratio:1;background:#eaf0e4;border-radius:2px;border:1px solid #dfe8d6}.expert-dot.active{background:var(--green);border-color:var(--green)}.routing-token{font:12px var(--mono);white-space:pre-wrap;overflow-wrap:anywhere;min-height:32px;padding:7px 10px;border-radius:4px;background:#f1f6e9;display:block}.routing-label{font:10px var(--mono);color:var(--muted);margin:13px 0 0}.expert-list{font:11px/1.8 var(--mono);color:var(--green);overflow-wrap:anywhere;min-height:40px}.routing-key{font-size:10px;color:var(--muted);margin:0}.condition-link{display:inline-block;font-size:11px;margin-top:14px}.reveal-blank{border:1px dashed #c8d6bf;border-radius:7px;padding:18px;margin-top:20px;font-size:12px;color:var(--muted)}details{border-top:1px solid var(--line);padding:17px 0;margin-top:20px}summary{font-size:12px;font-weight:500;cursor:pointer;width:fit-content;padding:3px 0}details p,details li{font-size:12px;line-height:1.8;color:var(--muted)}details pre{max-height:340px;overflow:auto;padding:16px;background:#f2f5ee;border-radius:5px;font:11px/1.8 var(--mono);white-space:pre-wrap;overflow-wrap:anywhere}details ul,details ol{padding-left:22px}.evidence{display:grid;grid-template-columns:minmax(0,1fr) auto;gap:20px;margin-top:44px;padding:30px 0 0;border-top:1px solid var(--line)}.evidence h2{font-size:23px}.evidence p:not(.eyebrow){font-size:12px;color:var(--muted);margin:0;max-width:640px}.evidence-actions{display:flex;gap:8px;align-items:center;flex-wrap:wrap;justify-content:flex-end}.evidence-actions button{font-size:11px}.evidence details{grid-column:1/-1;margin-top:0}.loading,.error{padding:55px 25px;border:1px solid var(--line);border-radius:10px;text-align:center;background:var(--paper);color:var(--muted)}.error h2{font-size:22px;color:var(--ink)}.error p{font-size:13px;max-width:550px;margin:15px auto}.error button{margin-top:6px}footer{display:flex;justify-content:space-between;gap:20px;padding:24px 36px;border-top:1px solid var(--line);font-size:10px;color:var(--muted)}footer a{text-decoration:none}footer span{text-align:center}.toast{position:fixed;left:50%;bottom:25px;transform:translateX(-50%);padding:12px 20px;background:var(--ink);color:white;border-radius:7px;max-width:90%;font-size:12px;z-index:10}.sr-only{position:absolute;width:1px;height:1px;padding:0;overflow:hidden;clip:rect(0,0,0,0);white-space:nowrap;border:0}@media(max-width:1000px){.experiment-grid{grid-template-columns:repeat(3,minmax(0,1fr))}.experiment-card:last-child{grid-column:span 2}.hero{gap:30px;grid-template-columns:minmax(0,1fr) 235px}.hero-note{padding-left:25px}.columns.three{grid-template-columns:1fr}.evidence{grid-template-columns:1fr}.evidence-actions{justify-content:flex-start}.experiment-head{align-items:flex-start}.case-picker{min-width:180px}}@media(max-width:680px){header{padding:14px 18px;align-items:flex-start;flex-wrap:wrap;gap:12px}.brand{font-size:13px}.brand-icon{width:31px;height:31px;font-size:23px}header nav{gap:18px;font-size:10px;margin-left:42px;margin-top:-2px}main{padding:0 18px 35px}.hero{grid-template-columns:1fr;padding:36px 0 28px;gap:22px}h1{font-size:43px;letter-spacing:-2px}.intro{font-size:14px}.hero-note{display:flex;align-items:center;gap:16px;padding:13px 0;border-left:0;border-top:1px solid var(--line);border-bottom:1px solid var(--line)}.hero-note p{font-size:13px;margin:0;min-width:125px}.hero-note>span:last-child{font-size:10px;line-height:1.5}.signal{gap:3px;width:44px;flex-shrink:0}.signal i{width:4px}.experiment-grid{grid-template-columns:repeat(2,minmax(0,1fr));gap:8px;margin-bottom:22px}.experiment-card{padding:14px 13px}.experiment-card:last-child{grid-column:span 2;display:grid;grid-template-columns:32px minmax(0,1fr);gap:6px 10px;align-items:center}.experiment-card:last-child strong{min-height:0}.experiment-card:last-child small{grid-column:2;margin-top:0}.experiment-card:last-child .card-number{margin:0;grid-row:span 2}.card-number{margin-bottom:15px}.experiment-card strong{font-size:13px;min-height:35px}.experiment-card small{font-size:10px}.experiment-head{flex-direction:column;gap:18px;padding:25px 0 19px}.experiment-head h2{font-size:28px}.experiment-head p:not(.eyebrow){font-size:12px}.case-picker{width:100%;min-width:0}.case-picker select{max-width:none}.case-top{padding:19px 17px;gap:12px}.case-top h3{font-size:16px}.case-top .tag{font-size:8px;white-space:normal;max-width:95px;text-align:center}.case-body{padding:17px}.columns{grid-template-columns:1fr;gap:15px}.condition-heading,.condition-content{padding:14px}.output{font-size:12px;max-height:320px}.metrics{gap:12px}.guess-panel{padding:17px}.question{font-size:17px}.choice-row button{min-width:100px;font-size:12px}.token-controls{gap:10px;padding:12px}.token-controls label{width:100%}.token-controls input{min-width:70px}.expert-grid{gap:2px}.prompt{font-size:11px;padding:12px}.evidence{margin-top:28px}.evidence h2{font-size:21px}.evidence-actions{gap:7px}.evidence-actions button{padding:9px 11px}footer{padding:20px 18px;flex-wrap:wrap;gap:12px}footer span{order:3;flex-basis:100%;text-align:left}.controls{gap:8px}.controls button{font-size:11px;padding:9px 12px}.share{font-size:10px}.hero-note>span:last-child{max-width:120px}}@media(prefers-reduced-motion:reduce){*{scroll-behavior:auto!important;transition:none!important}} + +#experiment{scroll-margin-top:20px} diff --git a/experiments/lab.js b/experiments/lab.js new file mode 100644 index 0000000000000000000000000000000000000000..9db036966ec9825019d994da2d0b3f02203cc55d --- /dev/null +++ b/experiments/lab.js @@ -0,0 +1,325 @@ +(function (global) { + 'use strict'; + const DEFINITIONS = { + reroute: { number: '01', title: 'One word changed.', short: 'Follow the routing', summary: 'Compare two prompts and the measured expert selections they produced.' }, + quant: { number: '02', title: 'Spot the 4-bit model.', short: 'A blind precision test', summary: 'Read two answers before seeing their precision, timings and checks.' }, + surgery: { number: '03', title: 'What else broke?', short: 'A controlled intervention', summary: 'Inspect a behavior intervention and its effects across the same test cases.' }, + memory: { number: '04', title: 'Remembered, but outdated.', short: 'The correction detective', summary: 'Predict the right answer, then see whether the model used a correction or repeated an old fact.' }, + sound: { number: '05', title: 'A model, heard.', short: 'Listen to real routing', summary: 'Turn measured model activity into a musical score you can inspect.' } + }; + function assert(condition, message) { if (!condition) throw new Error(message); } + function validateResults(data) { + assert(data && data.schema_version === 1, 'This results file uses an unsupported format.'); + assert(data.experiments && typeof data.experiments === 'object', 'Experiment results are missing.'); + for (const key of ['reroute', 'quant', 'surgery', 'memory']) { + const exp = data.experiments[key]; + assert(exp && Array.isArray(exp.cases) && exp.cases.length > 0, `The ${key} experiment has no cases.`); + const seen = new Set(); + for (const item of exp.cases) { + assert(item && typeof item.id === 'string' && item.id.length > 0 && !seen.has(item.id), `Invalid or repeated case ID in ${key}.`); + seen.add(item.id); + assert(Array.isArray(item.conditions) && item.conditions.length >= 1 && item.conditions.length <= 12, `Conditions are missing in ${key}/${item.id}.`); + const conditionIds = new Set(); + for (const condition of item.conditions) { + assert(condition && typeof condition.id === 'string' && !conditionIds.has(condition.id), `Invalid condition ID in ${key}/${item.id}.`); + conditionIds.add(condition.id); + assert(typeof condition.output === 'string', `A captured answer is missing in ${key}/${item.id}.`); + if (condition.routing && condition.routing.tokens !== undefined) { + assert(Array.isArray(condition.routing.tokens), `Routing tokens must be an array in ${key}/${item.id}.`); + for (const frame of condition.routing.tokens) { + assert(frame && typeof frame.token === 'string' && Array.isArray(frame.experts), 'A routing frame is invalid.'); + assert(frame.experts.every(id => Number.isInteger(id) && id >= 0 && id < (condition.routing.expert_count || 256)), 'An expert index is outside this model.'); + if (frame.weights !== undefined) assert(Array.isArray(frame.weights) && frame.weights.length === frame.experts.length && frame.weights.every(w => Number.isFinite(w) && w >= 0), 'Routing weights are invalid.'); + } + } + } + if (key === 'quant') assert(item.conditions.length === 2, 'The blind comparison needs exactly two answers.'); + if (key === 'memory') assert(Array.isArray(item.choices) && item.choices.length >= 2 && item.choices.every(c => typeof c === 'string'), 'The memory case needs answer choices.'); + } + } + return data; + } + function stablePairOrder(id, conditions) { + let hash = 2166136261; + for (const char of String(id)) hash = Math.imul(hash ^ char.charCodeAt(0), 16777619) >>> 0; + return hash % 2 ? conditions.slice().reverse() : conditions.slice(); + } + function isLowBit(condition) { return /(?:^|[^0-9])q?4(?:[^0-9]|$)/i.test(String(condition && condition.quant || '')); } + function formatMetric(value) { + if (typeof value === 'number') return Number.isInteger(value) ? value.toLocaleString('en-US') : value.toLocaleString('en-US', { maximumFractionDigits: 5 }); + if (typeof value === 'boolean') return value ? 'Yes' : 'No'; + return typeof value === 'object' && value !== null ? JSON.stringify(value) : String(value ?? ''); + } + function readableKey(key) { return key.replace(/_/g, ' ').replace(/\bms\b/g, '(ms)').replace(/\bseconds\b/g, '(s)'); } + function promptFor(item, condition) { + if (typeof condition.prompt === 'string') return condition.prompt; + const linked = Array.isArray(item.prompts) && item.prompts.find(p => p.id === condition.prompt_id); + return linked ? linked.text : typeof item.prompt === 'string' ? item.prompt : ''; + } + function fixturesFrom(data) { + const experiments = {}; + for (const [key, exp] of Object.entries(data.experiments)) { + experiments[key] = { + method: exp.method, + cases: exp.cases.map(item => ({ id: item.id, title: item.title, prompt: item.prompt, prompts: item.prompts, question: item.question, ground_truth: item.ground_truth, choices: item.choices, intervention: item.intervention, conditions: item.conditions.map(c => ({ id: c.id, model: c.model, quant: c.quant, prompt: promptFor(item, c), settings: c.settings, intervention: c.intervention })) })) + }; + } + return { schema_version: 1, generated_at: data.generated_at, provenance: data.provenance, methodology: data.methodology, experiments }; + } + function routingFrame(condition, step) { + const frames = condition.routing && condition.routing.tokens; + return Array.isArray(frames) && step < frames.length ? frames[step] : null; + } + const api = { validateResults, stablePairOrder, isLowBit, formatMetric, promptFor, fixturesFrom, routingFrame }; + if (typeof module !== 'undefined' && module.exports) module.exports = api; + if (typeof document === 'undefined') return; + const $ = id => document.getElementById(id); + let data, active = 'reroute', currentCase, tokenStep = 0, toastTimer; + const reveals = new Set(); + const stateKey = 'alphabell-experiments-v1'; + let guesses = { quant: {}, memory: {} }; + try { const saved = JSON.parse(localStorage.getItem(stateKey) || 'null'); if (saved && saved.quant && saved.memory) guesses = saved; } catch (_) { /* Private browser storage may be unavailable. */ } + function node(tag, className, content) { + const element = document.createElement(tag); + if (className) element.className = className; + if (content !== undefined) element.textContent = String(content); + return element; + } + function append(parent, ...children) { for (const child of children) if (child) parent.append(child); return parent; } + function button(text, action, className) { const el = node('button', className, text); el.type = 'button'; el.addEventListener('click', action); return el; } + function safeURL(url) { + if (typeof url !== 'string') return null; + try { const parsed = new URL(url, location.href); return ['http:', 'https:'].includes(parsed.protocol) ? parsed.href : null; } catch (_) { return null; } + } + function link(text, url, className) { + const href = safeURL(url); + if (!href) return null; + const el = node('a', className, text); el.href = href; + if (new URL(href).origin !== location.origin) { el.target = '_blank'; el.rel = 'noreferrer'; } + return el; + } + function toast(message) { $('toast').textContent = message; $('toast').hidden = false; clearTimeout(toastTimer); toastTimer = setTimeout(() => { $('toast').hidden = true; }, 3000); } + function save() { try { localStorage.setItem(stateKey, JSON.stringify(guesses)); } catch (_) { toast('This browser cannot save guesses. Your current answer is still recorded for this visit.'); } } + function download(object, name) { + const url = URL.createObjectURL(new Blob([JSON.stringify(object, null, 2) + '\n'], { type: 'application/json' })); + const anchor = document.createElement('a'); anchor.href = url; anchor.download = name; anchor.click(); setTimeout(() => URL.revokeObjectURL(url), 1000); + } + function displayValue(parent, value) { + if (value === undefined || value === null || value === '') return; + if (typeof value === 'string') { parent.append(node('p', '', value)); return; } + if (Array.isArray(value)) { const list = node('ul'); value.forEach(item => append(list, node('li', '', typeof item === 'object' ? JSON.stringify(item) : item))); parent.append(list); return; } + parent.append(node('pre', '', JSON.stringify(value, null, 2))); + } + function metrics(object) { + const list = node('dl', 'metrics'); + for (const [key, value] of Object.entries(object || {})) { + if (value === undefined || value === null) continue; + list.append(append(node('div'), node('dt', '', readableKey(key)), node('dd', '', formatMetric(value)))); + } + return list.children.length ? list : null; + } + function setupNav() { + $('experimentNav').replaceChildren(); + for (const [key, def] of Object.entries(DEFINITIONS)) { + const anchor = node('a', 'experiment-card'); anchor.href = key === 'sound' ? 'sound.html' : '?experiment=' + key; + anchor.dataset.experiment = key; + append(anchor, node('span', 'card-number', def.number), node('strong', '', def.title), node('small', '', def.short), node('span', 'card-arrow', '↗')); + if (key !== 'sound') anchor.addEventListener('click', event => { event.preventDefault(); selectExperiment(key, null, true); }); + $('experimentNav').append(anchor); + } + } + function selectExperiment(key, id, push) { + active = ['reroute', 'quant', 'surgery', 'memory'].includes(key) ? key : 'reroute'; + if (!data) { + const url = new URL(location.href); url.searchParams.set('experiment', active); url.searchParams.delete('case'); history.pushState(null, '', url); + return; + } + const cases = data.experiments[active].cases; + currentCase = cases.find(item => item.id === id) || cases[0]; + tokenStep = 0; + if (push) { + const url = new URL(location.href); url.searchParams.set('experiment', active); url.searchParams.set('case', currentCase.id); history.pushState(null, '', url); + } + document.querySelectorAll('[data-experiment]').forEach(a => { if (a.dataset.experiment === active) a.setAttribute('aria-current', 'page'); else a.removeAttribute('aria-current'); }); + render(); + if (push) $('experiment').scrollIntoView({block:'start',behavior:'smooth'}); + } + function conditionCard(item, condition, options = {}) { + const card = node('section', 'condition'); + const heading = node('div', 'condition-heading'); + append(heading, node('h4', '', options.label || condition.label || condition.id)); + if (!options.blind) heading.append(node('span', 'tag', [condition.model, condition.quant].filter(Boolean).join(' · '))); + const content = node('div', 'condition-content'); + const prompt = promptFor(item, condition); + if (options.showPrompt && prompt) content.append(node('div', 'prompt', prompt)); + content.append(node('div', 'output', condition.output || '(The model produced no answer.)')); + if (!options.blind) { + append(content, metrics(condition.metrics)); + if (condition.checks) { const detail = node('details'); append(detail, node('summary', '', 'Answer checks')); displayValue(detail, condition.checks); content.append(detail); } + append(content, link('Open measured trace ↗', condition.trace_url, 'condition-link')); + } + append(card, heading, content); return card; + } + function shareButton() { + return button('Share this case ↗', async () => { + const url = new URL(location.href); url.searchParams.set('experiment', active); url.searchParams.set('case', currentCase.id); + try { await navigator.clipboard.writeText(url.href); toast('Case link copied.'); } catch (_) { toast('Copy the address in your browser to share this case.'); } + }, 'quiet share'); + } + function render() { + const exp = data.experiments[active], def = DEFINITIONS[active], item = currentCase; + document.title = def.title + ' | alphaBell Experiments'; + const fragment = document.createDocumentFragment(), head = node('div', 'experiment-head'); + const title = node('h2', '', exp.title || def.title); title.id = 'experimentTitle'; + append(head, append(node('div'), node('p', 'eyebrow', 'EXPERIMENT ' + def.number), title, node('p', '', exp.summary || def.summary))); + const picker = node('label', 'case-picker', 'Choose a recorded case'); const select = node('select'); select.id = 'caseSelect'; + exp.cases.forEach((c, index) => { const option = node('option', '', String(index + 1).padStart(2, '0') + ' · ' + (c.title || c.id)); option.value = c.id; option.selected = c.id === item.id; select.append(option); }); + select.addEventListener('change', () => selectExperiment(active, select.value, true)); append(picker, select); head.append(picker); fragment.append(head); + const layout = node('article', 'case-layout'); + const top = node('div', 'case-top'), context = append(node('div'), node('h3', '', item.title || item.id)); + if (item.description) context.append(node('p', '', item.description)); + if (item.prompt && active !== 'reroute') context.append(node('div', 'prompt', item.prompt)); + append(top, context, node('span', 'tag', 'Recorded experiment')); + const body = node('div', 'case-body'); + if (active === 'reroute') renderReroute(body, item); + if (active === 'quant') renderQuant(body, item); + if (active === 'surgery') renderSurgery(body, item); + if (active === 'memory') renderMemory(body, item); + const controls = node('div', 'controls'); + append(controls, button('Download this case ↓', () => download({ schema_version: data.schema_version, experiment: active, provenance: data.provenance, method: exp.method, case: item }, 'alphabell-' + active + '-' + item.id.replace(/[^a-z0-9_-]/gi, '_') + '.json'), 'quiet'), shareButton()); + body.append(controls); + const method = node('details'); append(method, node('summary', '', 'How to read this experiment')); displayValue(method, exp.method); displayValue(method, exp.limitations); displayValue(method, exp.reproduce); if (item.provenance) displayValue(method, item.provenance); body.append(method); + append(layout, top, body); fragment.append(layout); $('experiment').replaceChildren(fragment); + } + function renderReroute(body, item) { + if (item.conditions.every(c => !c.routing)) body.append(node('p', 'annotation', 'Qwen Open uses recurrent heads and dense feed-forward blocks. Compare measured head activity in the full traces; expert grids apply to Flint.')); + body.append(node('div', 'notice', 'A routing change is a measured change in expert selection. It does not identify what an expert means, or prove that an expert caused a particular answer.')); + const frames = item.conditions.map(c => c.routing && c.routing.tokens || []), maxFrames = Math.max(...frames.map(f => f.length)); + const columns = node('div', item.conditions.length > 2 ? 'columns three' : 'columns'); + const panels = []; + if (maxFrames > 0) { + const controls = node('div', 'token-controls'), label = node('label', '', 'Generated token step'); label.htmlFor = 'routingStep'; + const slider = node('input'); slider.type = 'range'; slider.id = 'routingStep'; slider.min = '0'; slider.max = String(maxFrames - 1); slider.value = String(tokenStep); + const readout = node('span', 'token-readout'); readout.setAttribute('aria-live', 'polite'); + const update = step => { tokenStep = Math.max(0, Math.min(maxFrames - 1, step)); slider.value = String(tokenStep); readout.textContent = (tokenStep + 1) + ' / ' + maxFrames; panels.forEach(panel => paintRouting(panel, tokenStep)); }; + const previous = button('‹', () => update(tokenStep - 1)); previous.setAttribute('aria-label', 'Previous token'); + const next = button('›', () => update(tokenStep + 1)); next.setAttribute('aria-label', 'Next token'); + slider.addEventListener('input', () => update(Number(slider.value))); append(controls, label, previous, slider, next, readout); body.append(controls); + body.append(node('p', 'annotation', 'The slider pairs tokens by generated position. Different answers may use different token boundaries and stop at different lengths.')); + readout.textContent = (tokenStep + 1) + ' / ' + maxFrames; + } + for (const condition of item.conditions) { + const card = conditionCard(item, condition, { showPrompt: true }); + if (condition.routing && Array.isArray(condition.routing.tokens) && condition.routing.tokens.length) { + const panel = node('div', 'routing-panel'); panel.condition = condition; + const label = 'Selected experts' + (Number.isInteger(condition.routing.layer) ? ', block ' + condition.routing.layer : ''); + append(panel, node('p', 'routing-label', label), node('code', 'routing-token'), node('div', 'expert-grid'), node('p', 'expert-list'), node('p', 'routing-key', 'Each square is an expert ID. Dark squares were selected in this captured token.')); + const grid = panel.querySelector('.expert-grid'); + for (let id = 0; id < (condition.routing.expert_count || 256); id++) { const dot = node('span', 'expert-dot'); dot.title = 'Expert ' + id; dot.setAttribute('aria-hidden', 'true'); grid.append(dot); } + card.querySelector('.condition-content').prepend(panel); panels.push(panel); paintRouting(panel, tokenStep); + } + columns.append(card); + } + body.append(columns); + if (item.comparison) { const details = node('details'); append(details, node('summary', '', 'Measured comparison')); displayValue(details, item.comparison); body.append(details); } + } + function paintRouting(panel, step) { + const frame = routingFrame(panel.condition, step), token = panel.querySelector('.routing-token'), list = panel.querySelector('.expert-list'); + token.textContent = frame ? frame.token : 'This answer has ended.'; + const experts = new Set(frame ? frame.experts : []); + Array.from(panel.querySelector('.expert-grid').children).forEach((dot, id) => dot.classList.toggle('active', experts.has(id))); + list.textContent = frame ? frame.experts.map((id, index) => '#' + id + (frame.weights ? ' (' + formatMetric(frame.weights[index]) + ')' : '')).join(' · ') : 'No captured frame at this position.'; + } + function renderQuant(body, item) { + const ordered = stablePairOrder(item.id, item.conditions), key = 'quant:' + item.id, revealed = reveals.has(key); + body.append(node('p', 'annotation', 'Both answers use the same case prompt. Model names, precision and measurements stay hidden until you reveal the evidence. Answer order is deterministic for each case.')); + const columns = node('div', 'columns'); ordered.forEach((condition, index) => columns.append(conditionCard(item, condition, { blind: !revealed, label: revealed ? 'Answer ' + String.fromCharCode(65 + index) + ' · ' + (condition.label || condition.id) : 'Answer ' + String.fromCharCode(65 + index) }))); body.append(columns); + const panel = node('section', 'guess-panel'); append(panel, node('h4', 'question', 'Which answer came from the 4-bit model?')); + const choices = node('div', 'choice-row'); + ordered.forEach((condition, index) => { + const pick = button('Answer ' + String.fromCharCode(65 + index) + ' is 4-bit', () => { guesses.quant[item.id] = condition.id; save(); reveals.add(key); render(); }); + pick.disabled = revealed; pick.setAttribute('aria-pressed', String(guesses.quant[item.id] === condition.id)); choices.append(pick); + }); panel.append(choices); + if (!revealed) { const controls = node('div', 'controls'); controls.append(button('Reveal without guessing', () => { reveals.add(key); render(); }, 'quiet')); panel.append(controls); } + else { + const low = ordered.find(isLowBit), guess = guesses.quant[item.id], chosen = ordered.find(c => c.id === guess); + const message = !low ? 'Precision labels are shown above.' : guess ? (isLowBit(chosen) ? 'You spotted it.' : 'The other answer was 4-bit.') : 'The precision is revealed.'; + const result = node('div', 'result-note', message); + if (low) result.append(node('p', '', 'Answer ' + String.fromCharCode(65 + ordered.indexOf(low)) + ' used ' + low.quant + '. A single answer is not a quality benchmark. Check the full outputs and measured results.')); + append(panel, result); if (item.checks) displayValue(panel, item.checks); + } + let total = 0, correct = 0; + for (const test of data.experiments.quant.cases) { const guess = guesses.quant[test.id]; if (guess) { total++; const cond = test.conditions.find(c => c.id === guess); if (cond && isLowBit(cond)) correct++; } } + panel.append(node('p', 'local-tally', 'LOCAL ONLY · ' + correct + ' / ' + total + ' correct guesses saved in this browser. No votes are uploaded.')); + body.append(panel); + } + function renderSurgery(body, item) { + const intervention = item.intervention || {}; + const notice = node('div', 'notice'); append(notice, node('strong', '', intervention.kind ? readableKey(intervention.kind) : 'Recorded intervention')); + if (intervention.description) notice.append(node('div', '', intervention.description)); + if (/prompt/i.test(intervention.kind || '')) notice.append(node('div', '', 'This intervention changes the prompt. It does not modify model weights.')); + if (/activation/i.test(intervention.kind || '')) notice.append(node('div', '', 'This intervention changes activations during execution. It does not rewrite the model weights.')); + body.append(notice); + const columns = node('div', item.conditions.length > 2 ? 'columns three' : 'columns'); item.conditions.forEach(c => columns.append(conditionCard(item, c, { showPrompt: true }))); body.append(columns); + if (item.checks) { const results = node('div', 'result-note'); results.append(node('strong', '', 'Behavior and regression checks')); displayValue(results, item.checks); body.append(results); } + if (item.comparison) { const results = node('details'); results.append(node('summary', '', 'Intervention comparison')); displayValue(results, item.comparison); body.append(results); } + body.append(node('p', 'annotation', 'A visible change is a result for this case and intervention. Check the other cases before generalizing about the model. Negative and unchanged results are useful evidence too.')); + } + function truthText(item) { + return typeof item.ground_truth === 'string' ? item.ground_truth : item.ground_truth && (item.ground_truth.answer || item.ground_truth.value) || JSON.stringify(item.ground_truth); + } + function renderMemory(body, item) { + const key = 'memory:' + item.id, revealed = reveals.has(key), selected = guesses.memory[item.id]; + const panel = node('section', 'guess-panel'); append(panel, node('h4', 'question', item.question || 'What should the model answer after the correction?')); + const choices = node('div', 'choice-row'); + item.choices.forEach(choice => { + const pick = button(choice, () => { guesses.memory[item.id] = choice; save(); render(); }); pick.disabled = revealed; pick.setAttribute('aria-pressed', String(selected === choice)); choices.append(pick); + }); panel.append(choices); + if (!revealed) { + const reveal = button('Reveal model answers', () => { reveals.add(key); render(); }, 'primary'); reveal.disabled = !selected; + const controls = node('div', 'controls'); controls.append(reveal); panel.append(controls); + panel.append(node('p', 'local-tally', 'Choose the answer the model should give. Your choice stays in this browser.')); + body.append(panel, node('div', 'reveal-blank', 'The captured answers and scoring stay hidden until you make a prediction.')); + } else { + const truth = truthText(item), correct = String(selected).trim().toLowerCase() === String(truth).trim().toLowerCase(); + const result = node('div', 'result-note', (correct ? 'Your answer matches the ground truth. ' : 'Compare your prediction with the ground truth. ') + 'Expected answer: ' + truth); + result.append(node('p', '', 'Your prediction: ' + selected + '. The model answers below are separate measured results.')); + panel.append(result); body.append(panel); + const columns = node('div', item.conditions.length > 2 ? 'columns three' : 'columns'); item.conditions.forEach(c => columns.append(conditionCard(item, c, { showPrompt: true }))); body.append(columns); + if (item.checks) { const check = node('div', 'result-note'); check.append(node('strong', '', 'Correction and recall checks')); displayValue(check, item.checks); body.append(check); } + } + body.append(node('p', 'annotation', 'Using an in-context correction and remembering a fact are different tests. The prompts show which information each condition received. These results do not measure a permanent change to model memory.')); + } + function renderMethods() { + const target = $('methodBody'); target.replaceChildren(); + if (data.generated_at) target.append(node('p', '', 'Results prepared: ' + new Intl.DateTimeFormat('en-US', {dateStyle:'medium',timeStyle:'short',timeZone:'America/Los_Angeles'}).format(new Date(data.generated_at)) + ' Pacific time.')); + displayValue(target, data.methodology); + const provenance = data.provenance || {}; + if (Array.isArray(provenance.model_sources)) { + target.append(node('h3', '', 'Source models')); + const list = node('ul'); provenance.model_sources.forEach(source => { const li = node('li'); append(li, link(source.name || source.url, source.url)); if (source.revision) li.append(document.createTextNode(' · revision ' + source.revision)); list.append(li); }); target.append(list); + } + const detail = { ...provenance }; delete detail.model_sources; displayValue(target, detail); + } + async function load() { + $('experiment').replaceChildren(node('div', 'loading', 'Opening measured experiments…')); + try { + const response = await fetch('data/results.json', { cache: 'no-cache' }); + if (!response.ok) throw new Error('The results could not be downloaded (HTTP ' + response.status + ').'); + data = validateResults(await response.json()); + renderMethods(); $('evidence').hidden = false; + const params = new URLSearchParams(location.search); selectExperiment(params.get('experiment'), params.get('case'), false); if (params.has('experiment')) requestAnimationFrame(() => $('experiment').scrollIntoView({block:'start'})); + } catch (error) { + $('evidence').hidden = true; + const panel = node('div', 'error'); panel.setAttribute('role', 'alert'); + append(panel, node('h2', '', 'The measured results could not be opened.'), node('p', '', error.message || 'Please try loading the experiment again.'), button('Try again', load)); + $('experiment').replaceChildren(panel); + } + } + setupNav(); + $('downloadResults').addEventListener('click', () => { if (data) download(data, 'alphabell-experiment-results.json'); }); + $('downloadFixtures').addEventListener('click', () => { if (data) download(fixturesFrom(data), 'alphabell-experiment-prompts.json'); }); + $('resetGuesses').addEventListener('click', () => { guesses = { quant: {}, memory: {} }; reveals.clear(); save(); if (data) render(); toast('Local guesses cleared.'); }); + window.addEventListener('popstate', () => { if (data) { const params = new URLSearchParams(location.search); selectExperiment(params.get('experiment'), params.get('case'), false); } }); + load(); +})(typeof window !== 'undefined' ? window : globalThis); diff --git a/experiments/reproduce/LLAMA-LICENSE b/experiments/reproduce/LLAMA-LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..e7dca554bcb802f98408383a864404e3aa4eacca --- /dev/null +++ b/experiments/reproduce/LLAMA-LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2023-2026 The ggml authors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/experiments/reproduce/LLAMA-MIT-LICENSE b/experiments/reproduce/LLAMA-MIT-LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..e7dca554bcb802f98408383a864404e3aa4eacca --- /dev/null +++ b/experiments/reproduce/LLAMA-MIT-LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2023-2026 The ggml authors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/experiments/reproduce/README.md b/experiments/reproduce/README.md new file mode 100644 index 0000000000000000000000000000000000000000..ac806514f4b85daf11f3498a3c63731c17f2e3e2 --- /dev/null +++ b/experiments/reproduce/README.md @@ -0,0 +1,94 @@ +# Matched quantization and FFN surgery + +The scripts replay six fixed prompts on Flint and alphaBell Qwen Open with matched Q4_1 and Q8_0 checkpoints. A second experiment scales Flint's computed feed-forward output in block 12 to 1, 0.5 or 0 before the residual addition. Block numbering starts at zero. The intervention applies to every prompt and decode position; it changes runtime activations and leaves checkpoint files unchanged. + +The suite uses CPU execution, eight threads, nice 10, a 2048-token context, Flash Attention disabled, temperature zero, seed 42, reasoning disabled and a 64-token output limit. Each checkpoint receives a warm-up. Every request clears model memory before processing its prompt. + +## Runtime source + +The measured runtime identifies its assembled source as `74686f084c53506a7afa471d857db02e5c912543`. Rebuild the same source using the public upstream base and included patch. The resulting Git tree must be `83e9d4c276f84753cce063502a2a9e2325f15ce3`; the build script checks this exact tree and rejects unstaged changes. No upstream commit containing the assembled runtime is required. + +Prerequisites tested on Ubuntu 24.04: GCC 13.3.0, CMake 3.28.3, Git 2.43.0 and Python 3.12.3. The JSON header is vendored in llama.cpp. The Python runner uses only the standard library. + +Run from this directory. The commands create a dedicated runtime checkout: + +```bash +git clone --no-checkout https://github.com/ggml-org/llama.cpp.git runtime-source +git -C runtime-source checkout --detach 0a1ef309cdbb9f2024f58a38fb4c42352c5db569 +git -C runtime-source apply --index ../llama-flint-runtime.patch +cmake -S runtime-source -B runtime-source/build \ + -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=OFF +cmake --build runtime-source/build --target llama -j 4 +bash quant-surgery-build.sh runtime-source runtime-source/build/bin quant-surgery +``` + +The included patch adds Flint support and hexadecimal string escapes. llama.cpp retains its MIT license. The measured build had CUDA support available but explicitly selected an empty GPU device list and zero GPU layers. The CPU-only build above avoids allocating a GPU. + +## Checkpoints and execution + +The exact matched checkpoints are available in [alphaBell Flint Matched GGUF](https://huggingface.co/alphabell/Flint-Matched-GGUF) and [alphaBell Qwen Open GGUF](https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF). + +Set `FLINT_Q8_GGUF`, `FLINT_Q4_GGUF`, `QWEN_Q8_GGUF` and `QWEN_Q4_GGUF` to the matching checkpoint files. The release records each file's byte count and SHA-256. Each pair was converted from one BF16 source, then quantized with `llama-quantize --pure` into Q4_1 and Q8_0. Qwen excludes vision and the multi-token prediction head. Its source is the alphaBell post-trained BF16 model, rather than the unmodified Qwen parent or the published FP8 file. + +```bash +python3 quant-surgery.py \ + --engine ./quant-surgery \ + --flint-q8 "$FLINT_Q8_GGUF" --flint-q4 "$FLINT_Q4_GGUF" \ + --qwen-q8 "$QWEN_Q8_GGUF" --qwen-q4 "$QWEN_Q4_GGUF" \ + --flint-config ../../config.json \ + --qwen-config ../../models/qwen.json \ + --output results/quant-surgery.json +``` + +The runner starts one temporary CPU worker at a time, closes each worker after its checkpoint's cases, and saves results after each case. It writes model specifications and diagnostic logs beside the output. No service restart or running model endpoint is involved. + +## Measurements and checks + +Each result includes the exact output, completion reason, input/output token counts, native elapsed time, client request time, first-token latency, decode throughput and process peak resident memory. Peak memory uses Linux `ru_maxrss`, in KiB, and includes loading and warm-up. Latency includes this instrumented CPU engine's overhead. These short cases are an exploratory comparison, rather than a general performance benchmark. + +For each output token, the surgery records FFN output RMS in blocks 11, 12 and 13, plus block 13 residual RMS. The validation checks that block 12 RMS follows the requested scale in every recorded frame. On the first frame, block 11 must match the baseline exactly and block 13's residual must change under either intervention. Later generated tokens can differ, so later frames need not share identical inputs. + +The fixed checks cover exact arithmetic, strict JSON, factual recall, fact correction, a Python expression and a one-sentence explanation. Python expressions are checked against an explicit allowlist; generated code is never executed. All six cases and both quantizations are retained. Identical answers remain visible in the blind comparison. A score here describes these six prompts only. + +Flint's checkpoint license is Apache 2.0. The alphaBell Qwen Open derivative has alphaBell's research and noncommercial license, with its Qwen and Alpaca notices. Follow each checkpoint's model card and license when using or redistributing weights. + +## Full topology captures + +`capture-study.cpp` is the standalone Observatory trace recorder used by the one-word routing, fact-correction and trace-music projects. It records Flint attention/head RMS, top-six routing, shared/routed FFN RMS and residual RMS. For alphaBell Qwen Open it records full attention and dense output, plus recurrent head RMS, write gates, retention and state RMS. Tensor byte strides and reshaped routing weights are preserved. + +Build it against the same pinned runtime source and libraries: + +```bash +bash capture-study-build.sh runtime-source runtime-source/build/bin capture-study +``` + +The runner reads the exact 24 public routing and correction fixtures from `../data/results.json`, using 16 CPU threads. Routing captures allow 48 output tokens; correction captures allow 16. All use temperature zero, seed 42, reasoning disabled, a 2048-token context and Flash Attention disabled. It verifies the Q8_0 checkpoint hashes before generation and clears model memory between prompts. + +```bash +python3 capture-study.py \ + --engine ./capture-study \ + --flint-model "$FLINT_Q8_GGUF" --qwen-model "$QWEN_Q8_GGUF" \ + --output-dir captures \ + --compare-root ../.. +``` + +`--compare-root` checks every new token ID, byte sequence, input position and layer measurement against the recorded public traces. It excludes timing and creation metadata, which describe the new run. Use repeated `--only` arguments to select particular fixtures: + +```bash +python3 capture-study.py \ + --engine ./capture-study \ + --flint-model "$FLINT_Q8_GGUF" \ + --only reroute-json-color-flint-a \ + --output-dir blue-capture \ + --compare-root ../.. +``` + +The output contains compressed trace JSON, a SHA-256 manifest and a verification report. Native specifications and diagnostic logs stay in a separate `worker-files` subdirectory. Public trace files contain model source URLs and public fixture IDs, with no host or filesystem paths. To import one into Observatory: + +```bash +gzip -dc blue-capture/reroute-json-color-flint-a.json.gz > blue-capture.json +``` + +The original recorded Flint Q8 container and the matched release container have different descriptive names in their headers. Their tensor descriptors, every tensor payload byte, tokenizer arrays and remaining metadata match. `flint-capture-equivalence.json` records that comparison and both container hashes. + +On 23 October 2026, the rebuilt capture producer replayed all 24 fixtures with 80 frames. Every recorded layer measurement, output token ID, token byte sequence and input position matched the published captures exactly. Both regenerated model formats also passed Observatory import validation. See `capture-study-verification.json`. diff --git a/experiments/reproduce/capture-study-build.sh b/experiments/reproduce/capture-study-build.sh new file mode 100644 index 0000000000000000000000000000000000000000..cff9a6d3082796099a959d99cf7db51385674ea1 --- /dev/null +++ b/experiments/reproduce/capture-study-build.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +set -euo pipefail +if [ "$#" -ne 3 ]; then + echo 'usage: capture-study-build.sh LLAMA_SOURCE LLAMA_LIBRARY_DIR OUTPUT' >&2 + exit 2 +fi +experiment_dir=$(cd "$(dirname "$0")" && pwd) +expected_tree=83e9d4c276f84753cce063502a2a9e2325f15ce3 +actual_tree=$(git -C "$1" write-tree) +if [ "$actual_tree" != "$expected_tree" ] || ! git -C "$1" diff --quiet; then + echo 'Runtime source differs from the measured source tree. Follow README.md.' >&2 + exit 1 +fi +nice -n 10 c++ -std=c++17 -O2 \ + -I"$1/include" -I"$1/ggml/include" -I"$1/vendor" \ + "$experiment_dir/capture-study.cpp" -L"$2" -Wl,-rpath,"$2" \ + -l:libllama.so.0 -lggml -lggml-base -pthread -o "$3" diff --git a/experiments/reproduce/capture-study-verification.json b/experiments/reproduce/capture-study-verification.json new file mode 100644 index 0000000000000000000000000000000000000000..73dc7b7f0191b8a2cb9c9a83f8b40bf8068cf9e5 --- /dev/null +++ b/experiments/reproduce/capture-study-verification.json @@ -0,0 +1,168 @@ +{ + "completed_at": "2026-10-23T11:36:10.484924+00:00", + "captures": 24, + "frames": 80, + "comparison_count": 24, + "comparisons": [ + { + "capture": "reroute-json-color-flint-a", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-json-color-flint-b", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-opposite-flint-a", + "frames": 1, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-opposite-flint-b", + "frames": 1, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-python-flint-a", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-python-flint-b", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-distractors-current-flint", + "frames": 2, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-distractors-original-flint", + "frames": 2, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-short-current-flint", + "frames": 2, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-short-original-flint", + "frames": 2, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-twice-current-flint", + "frames": 2, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-twice-original-flint", + "frames": 2, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-json-color-qwen-a", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-json-color-qwen-b", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-opposite-qwen-a", + "frames": 1, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-opposite-qwen-b", + "frames": 1, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-python-qwen-a", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "reroute-python-qwen-b", + "frames": 5, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-distractors-current-qwen", + "frames": 4, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-distractors-original-qwen", + "frames": 4, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-short-current-qwen", + "frames": 4, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-short-original-qwen", + "frames": 4, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-twice-current-qwen", + "frames": 4, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + }, + { + "capture": "memory-twice-original-qwen", + "frames": 4, + "token_ids_bytes_positions_equal": true, + "all_layer_measurements_exact": true + } + ], + "all_selected_captures_complete": true, + "capture_source_sha256": "645a408083eb77a67269e39aae8248994ee3c59ee7842630c537c0e84a7c4cf0", + "engine_source_tree": "83e9d4c276f84753cce063502a2a9e2325f15ce3", + "checkpoint_source": "Matched Q8_0 release files, verified against checkpoints.json", + "observatory_import_validation": [ + { + "model": "flint", + "frames": 5, + "accepted": true + }, + { + "model": "qwen", + "frames": 5, + "accepted": true + } + ] +} diff --git a/experiments/reproduce/capture-study.cpp b/experiments/reproduce/capture-study.cpp new file mode 100644 index 0000000000000000000000000000000000000000..0520850476342b433bd3a4046f5ab469dcf8150b --- /dev/null +++ b/experiments/reproduce/capture-study.cpp @@ -0,0 +1,134 @@ +#include "llama.h" +#include "ggml.h" +#include "ggml-backend.h" +#include "nlohmann/json.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using json=nlohmann::json; +using Clock=std::chrono::steady_clock; +static std::string STOP,MODEL_ID,ACTIVE_ID; +static void emit(json j){j["model"]=MODEL_ID;if(!ACTIVE_ID.empty())j["id"]=ACTIVE_ID;std::cout<(Clock::now()-t).count();} +static void quiet(enum ggml_log_level level,const char*text,void*){if(level>=GGML_LOG_LEVEL_WARN)std::cerr<():config.at("num_attention_heads").get();} + int dimension(int l)const{return linear(l)?config.at("linear_value_head_dim").get():config.at("head_dim").get();} +}; +static bool stopped(void*){ + if(STOP.empty()||access(STOP.c_str(),F_OK)!=0)return false; + std::ifstream file(STOP);std::string id;std::getline(file,id);return id==ACTIVE_ID; +} +static float at(const ggml_tensor*t,const std::vector&b,int x,int y=0,int z=0,int w=0){ + size_t off=x*t->nb[0]+y*t->nb[1]+z*t->nb[2]+w*t->nb[3]; + if(off+ggml_type_size(t->type)>b.size())throw std::runtime_error(std::string(t->name)+" tensor stride outside capture"); + if(t->type==GGML_TYPE_F32){float v;memcpy(&v,b.data()+off,4);return v;} + if(t->type==GGML_TYPE_F16){ggml_fp16_t v;memcpy(&v,b.data()+off,2);return ggml_fp16_to_fp32(v);} + if(t->type==GGML_TYPE_I32){int32_t v;memcpy(&v,b.data()+off,4);return v;} + throw std::runtime_error("Unsupported activation dtype"); +} +static bool capture(ggml_tensor*t,bool ask,void*user){ + auto&c=*static_cast(user);if(!c.enabled)return !ask; + std::string name=t->name;auto dash=name.rfind('-');if(dash==std::string::npos)return !ask; + std::string kind=name.substr(0,dash); + bool head=MODEL_ID=="flint"?kind=="kqv_out":kind=="final_output"||kind=="attn_gated"; + bool wanted=head||kind=="kq_soft_max"||kind=="ffn_moe_topk"||kind=="ffn_moe_weights_norm"||kind=="ffn_shexp"||kind=="ffn_moe_out"||kind=="ffn_out"||kind=="l_out"||kind=="beta_sigmoid"||kind=="gate"||kind=="new_state"; + if(ask)return wanted;if(!wanted)return true; + int layer=std::stoi(name.substr(dash+1));if(layer<0||layer>=c.config.at("num_hidden_layers").get())return true; + auto&l=c.layers[layer];std::vectorbuf(ggml_nbytes(t));ggml_backend_tensor_get(t,buf.data(),0,buf.size());int last=t->ne[1]-1; + if(head){ + json heads=json::array();int dim=c.dimension(layer),count=c.heads(layer); + if(t->ne[0]!=count*dim)throw std::runtime_error("Head output shape does not match model configuration"); + for(int h=0;h(t->ne[0],c.position+1); + for(int h=0;h(c.heads(layer),t->ne[2]);h++){ + std::vector>ranked;double entropy=0,mass=0; + for(int k=0;k0)entropy-=w*std::log(w);ranked.emplace_back(w,k);} + int count=std::min(8,ranked.size());std::partial_sort(ranked.begin(),ranked.begin()+count,ranked.end(),std::greater<>()); + json top=json::array();for(int i=0;i0)top.push_back({ranked[i].second,ranked[i].first}); + heads.push_back({{"top",top},{"entropy",entropy},{"sink",std::max(0.0,1-mass)}}); + }l["attention"]=heads; + }else if(kind=="beta_sigmoid"||kind=="gate"){ + json values=json::array();for(int h=0;hne[0]==1?at(t,buf,0,h,t->ne[2]-1):at(t,buf,h,last); + values.push_back(kind=="beta_sigmoid"?v:std::exp(v)); + } + l[kind=="beta_sigmoid"?"write_gate":"retention"]=values; + }else if(kind=="new_state"){ + json values=json::array();for(int h=0;hne[1];y++)for(int x=0;xne[0];x++){double v=at(t,buf,x,y,h);sq+=v*v;}values.push_back(std::sqrt(sq/(t->ne[0]*t->ne[1])));}l["state_rms"]=values; + }else if(kind=="ffn_moe_topk"){ + json ids=json::array();for(int i=0;i();i++)ids.push_back(int(at(t,buf,i,last)));l["experts"]=ids; + }else if(kind=="ffn_moe_weights_norm"){ + json weights=json::array();for(int i=0;i();i++)weights.push_back(t->ne[0]==1?at(t,buf,0,i,t->ne[2]-1):at(t,buf,i,last));l["weights"]=weights; + }else{ + double sq=0;for(int i=0;ine[0];i++){double v=at(t,buf,i,last);sq+=v*v;} + l[kind=="l_out"?"residual":kind=="ffn_shexp"?"shared":kind=="ffn_out"?"ffn":"routed"]=std::sqrt(sq/t->ne[0]); + if(kind=="l_out"){auto live=l;live.erase("attention");emit({{"type","layer"},{"step",c.step},{"layer",layer},{"data",live}});} + } + return !stopped(nullptr); +} +static std::string piece(const llama_vocab*v,llama_token t){ + std::vectorb(512);int n=llama_token_to_piece(v,t,b.data(),b.size(),0,true);if(n<0){b.resize(-n);n=llama_token_to_piece(v,t,b.data(),b.size(),0,true);}return std::string(b.data(),std::max(0,n)); +} +int main(int argc,char**argv){ + if(argc<3){std::cerr<<"usage: capture-study MODEL.gguf SPEC.json\n";return 2;} + Capture c;json spec; + try{std::ifstream file(argv[2]);file>>spec;c.config=spec.at("architecture");MODEL_ID=spec.at("id");STOP=spec.at("stop_path");}catch(const std::exception&e){emit({{"type","fatal"},{"error",std::string("Invalid model specification: ")+e.what()}});return 1;} + llama_log_set(quiet,nullptr);llama_backend_init();auto mp=llama_model_default_params();mp.n_gpu_layers=0;ggml_backend_dev_t devices[]={nullptr};mp.devices=devices; + auto model=llama_model_load_from_file(argv[1],mp);if(!model){emit({{"type","fatal"},{"error","Could not load model checkpoint"}});return 1;} + if(llama_model_n_layer(model)!=c.config.at("num_hidden_layers").get()||llama_model_n_embd(model)!=c.config.at("hidden_size").get()){emit({{"type","fatal"},{"error","Checkpoint dimensions do not match the model specification"}});return 1;} + auto vocab=llama_model_get_vocab(model); + auto cp=llama_context_default_params();cp.n_ctx=2048;cp.n_batch=256;cp.n_ubatch=256;cp.n_threads=16;cp.n_threads_batch=16;cp.n_rs_seq=0; + cp.flash_attn_type=LLAMA_FLASH_ATTN_TYPE_DISABLED;cp.cb_eval=capture;cp.cb_eval_user_data=&c;cp.abort_callback=stopped; + auto ctx=llama_init_from_model(model,cp);if(!ctx){emit({{"type","fatal"},{"error","Could not initialize tracing context"}});return 1;} + emit({{"type","ready"},{"engine","llama.cpp 74686f08"},{"quantization",spec.at("quantization")},{"device","CPU"},{"threads",16},{"context",2048}}); + std::string line; + while(std::getline(std::cin,line)){ + llama_sampler*sampler=nullptr; + try{ + auto req=json::parse(line);ACTIVE_ID=req.value("id","manual");unlink(STOP.c_str());llama_memory_clear(llama_get_memory(ctx),true); + std::string prompt=req.at("prompt");for(size_t pos=0;(pos=prompt.find("<|",pos))!=std::string::npos;pos+=5)prompt.replace(pos,2,"<|"); + bool reasoning=req.value("reasoning",false),flint=MODEL_ID=="flint"; + std::string system="You are a helpful assistant. Answer clearly and concisely."; + std::string text=flint?"<|system|>"+system+"<|user|>"+prompt+"<|assistant|>"+(reasoning?"":""):"<|im_start|>system\n"+system+"<|im_end|>\n<|im_start|>user\n"+prompt+"<|im_end|>\n<|im_start|>assistant\n\n"+(reasoning?"":"\n\n\n"); + int n=-llama_tokenize(vocab,text.data(),text.size(),nullptr,0,flint,true); + if(n<=0)throw std::runtime_error("Could not tokenize the prompt"); + if(n>256)throw std::runtime_error("Prompt exceeds the tracing limit of 256 tokens. Shorten it and run again."); + std::vectortokens(n);llama_tokenize(vocab,text.data(),text.size(),tokens.data(),n,flint,true); + json pieces=json::array();for(auto t:tokens)pieces.push_back(piece(vocab,t)); + int limit=std::clamp(req.value("max_tokens",64),1,256);sampler=llama_sampler_chain_init(llama_sampler_chain_default_params());double temperature=req.value("temperature",0.0); + if(temperature<=0)llama_sampler_chain_add(sampler,llama_sampler_init_greedy());else{llama_sampler_chain_add(sampler,llama_sampler_init_top_k(40));llama_sampler_chain_add(sampler,llama_sampler_init_top_p(.95,1));llama_sampler_chain_add(sampler,llama_sampler_init_temp(temperature));llama_sampler_chain_add(sampler,llama_sampler_init_dist(req.value("seed",42)));} + emit({{"type","start"},{"prompt_tokens",n},{"tokens",pieces},{"max_tokens",limit},{"temperature",temperature},{"seed",req.value("seed",42)},{"reasoning",reasoning}}); + auto started=Clock::now();int pos=0,count=n,generated=0;std::string reason="limit"; + for(int step=0;step();i++)c.layers.push_back({{"index",i}});c.enabled=true; + auto before=Clock::now();auto batch=llama_batch_get_one(tokens.data(),count);int rc=llama_decode(ctx,batch);llama_synchronize(ctx); + if(rc){if(stopped(nullptr)){reason="stopped";break;}throw std::runtime_error("Inference failed with code "+std::to_string(rc));}pos+=count; + for(int l=0;l();l++){ + const auto&v=c.layers[l];if(!v.contains("heads")||v["heads"].size()!=size_t(c.heads(l))||!v.contains("residual"))throw std::runtime_error("Incomplete head capture at layer "+std::to_string(l)); + if(c.linear(l)&&(!v.contains("state_rms")||!v.contains("write_gate")||!v.contains("retention")))throw std::runtime_error("Incomplete recurrent capture at layer "+std::to_string(l)); + if(!c.linear(l)&&(!v.contains("attention")||v["attention"].size()!=size_t(c.heads(l))))throw std::runtime_error("Incomplete attention capture at layer "+std::to_string(l)); + } + auto id=llama_sampler_sample(sampler,ctx,-1);llama_sampler_accept(sampler,id);if(llama_vocab_is_eog(vocab,id)){reason="eos";break;} + auto p=piece(vocab,id);generated++; + emit({{"type","token"},{"step",step},{"token_id",id},{"text",p},{"token_bytes",std::vector(p.begin(),p.end())},{"input_position",c.position},{"input_text",pieces.back()},{"layers",c.layers},{"elapsed",seconds(started)},{"duration",seconds(before)},{"phase",step==0?"prefill":"decode"}}); + pieces.push_back(p);tokens={id};count=1;if(stopped(nullptr)){reason="stopped";break;} + } + c.enabled=false;llama_sampler_free(sampler);sampler=nullptr;emit({{"type","done"},{"reason",reason},{"generated",generated},{"elapsed",seconds(started)}}); + }catch(const std::exception&e){c.enabled=false;if(sampler)llama_sampler_free(sampler);emit({{"type","error"},{"error",e.what()}});} + ACTIVE_ID.clear(); + } + llama_free(ctx);llama_model_free(model);llama_backend_free(); +} diff --git a/experiments/reproduce/capture-study.py b/experiments/reproduce/capture-study.py new file mode 100644 index 0000000000000000000000000000000000000000..1850a8875937d75de92e41efa385d61d536b5514 --- /dev/null +++ b/experiments/reproduce/capture-study.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Produce full routing, attention and recurrent-state traces for public fixtures.""" +import argparse, datetime, gzip, hashlib, json, pathlib, queue, re, subprocess, threading, time +from urllib.parse import parse_qs, urlparse + +REPRO = pathlib.Path(__file__).resolve().parent +PROJECT = REPRO.parent.parent +MODELS = { + 'flint':{'id':'flint','title':'Recursive Flint','short_name':'Flint','family':'MoE transformer','quantization':'Q8_0','source_url':'https://huggingface.co/alphabell/Flint-Matched-GGUF','learned_sink':True,'stats':[['49','','layers'],['32','','query heads / layer'],['6','256','routed experts / token'],['2.5B','39.8B','active parameters']]}, + 'qwen':{'id':'qwen','title':'alphaBell Qwen Open 27B','short_name':'Qwen Open','family':'Hybrid recurrent transformer','quantization':'Q8_0','source_url':'https://huggingface.co/alphabell/alphaBell-Qwen3.8-27B-Open-GGUF','learned_sink':False,'stats':[['64','','layers'],['48','24','linear / full heads'],['17408','','dense FFN width'],['5120','','hidden width']],'checkpoint_origin':'alphaBell fine-tuned Qwen3.8-27B, converted from the lab BF16 checkpoint to GGUF; the public FP8 release is a sibling format.'} +} + +def load_cases(path): + document=json.loads(path.read_text());cases=[] + for section in ('reroute','memory'): + for case in document['experiments'][section]['cases']: + for condition in case['conditions']: + query=parse_qs(urlparse(condition['trace_url']).query);model=query['model'][0];capture=query['capture'][0] + if model not in MODELS or not re.fullmatch('[a-z0-9-]+',capture):raise ValueError('Invalid public fixture identity') + settings=condition['settings'] + expected_limit=48 if section=='reroute' else 16 + if settings!={'max_tokens':expected_limit,'temperature':0,'seed':42,'reasoning':False}:raise ValueError('Unexpected fixture decoding settings') + cases.append({'id':capture,'model':model,'prompt':condition.get('prompt',case.get('prompt')),'settings':settings,'expected_output':condition['output'],'title':case['title'],'section':section}) + if len(cases)!=24 or len({c['id'] for c in cases})!=24:raise ValueError('Expected 24 distinct public fixtures') + return cases + +class Worker: + def __init__(self,engine,checkpoint,spec,log): + self.q=queue.Queue();self.log=log.open('w') + self.p=subprocess.Popen(['nice','-n','10',str(engine.resolve()),str(checkpoint.resolve()),str(spec.resolve())],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=self.log,text=True,bufsize=1) + def reader(): + try: + for line in self.p.stdout: + try:self.q.put(json.loads(line)) + except ValueError:self.q.put({'type':'error','error':'Invalid native JSON'}) + finally:self.q.put({'type':'eof'}) + self.reader=threading.Thread(target=reader,daemon=True);self.reader.start() + self.ready=self.q.get(timeout=300) + if self.ready.get('type')!='ready' or self.ready.get('threads')!=16 or self.ready.get('device')!='CPU':self.close();raise RuntimeError('Capture worker failed to start with the required CPU settings') + def capture(self,case,architecture,provenance): + self.p.stdin.write(json.dumps({'id':case['id'],'prompt':case['prompt'],**case['settings']})+'\n');self.p.stdin.flush() + frames=[];start=None;sequence=0;created=int(time.time()*1000) + while True: + event=self.q.get(timeout=180);event['seq']=sequence;sequence+=1 + if event['type'] in ('error','fatal','eof'):raise RuntimeError(event.get('error','Capture worker ended unexpectedly')) + if event.get('model')!=case['model'] or event.get('id')!=case['id']:raise RuntimeError('Mismatched native event identity') + if event['type']=='start':start=event + elif event['type']=='token':frames.append(event) + elif event['type']=='done':break + if not start or not frames or event.get('reason')!='eos':raise RuntimeError('Fixture did not produce a complete EOS trace') + output=b''.join(bytes(f['token_bytes']) for f in frames).decode('utf-8',errors='replace') + if output!=case['expected_output']:raise RuntimeError('Fixture output differs from the recorded public result') + return {'schema':'flint.trace.v1' if case['model']=='flint' else 'model.trace.v1','model':{**MODELS[case['model']],'architecture':architecture},'id':case['id'],'prompt':case['prompt'],'settings':case['settings'],'created':created,'engine':self.ready,'architecture':architecture,'start':start,'frames':frames,'result':event,'provenance':provenance} + def close(self): + if self.p.poll() is None: + try:self.p.stdin.close();self.p.wait(timeout=30) + except (BrokenPipeError,subprocess.TimeoutExpired):self.p.terminate();self.p.wait(timeout=30) + self.log.close() + +def compare_trace(actual,reference): + if actual['prompt']!=reference['prompt'] or actual['settings']!=reference['settings']:raise ValueError('Fixture comparison settings differ') + if len(actual['frames'])!=len(reference['frames']):raise ValueError('Fixture frame count differs') + for current,prior in zip(actual['frames'],reference['frames']): + for key in ('layers','step','token_id','token_bytes','input_position'): + if current[key]!=prior[key]:raise ValueError('Recorded trace differs at token '+str(current['step'])+', field '+key) + return {'capture':actual['id'],'frames':len(actual['frames']),'token_ids_bytes_positions_equal':True,'all_layer_measurements_exact':True} + +def main(): + parser=argparse.ArgumentParser();parser.add_argument('--engine',type=pathlib.Path,required=True);parser.add_argument('--flint-model',type=pathlib.Path);parser.add_argument('--qwen-model',type=pathlib.Path);parser.add_argument('--flint-config',type=pathlib.Path,default=PROJECT/'config.json');parser.add_argument('--qwen-config',type=pathlib.Path,default=PROJECT/'models/qwen.json');parser.add_argument('--fixtures',type=pathlib.Path,default=PROJECT/'experiments/data/results.json');parser.add_argument('--output-dir',type=pathlib.Path,required=True);parser.add_argument('--only',action='append',help='Exact public capture ID; repeat to select several fixtures');parser.add_argument('--compare-root',type=pathlib.Path,help='Public Observatory checkout containing the recorded trace manifest and gzip files');args=parser.parse_args() + cases=load_cases(args.fixtures) + if args.only: + wanted=set(args.only);cases=[c for c in cases if c['id'] in wanted] + if len(cases)!=len(wanted):raise ValueError('Unknown fixture selection') + args.output_dir.mkdir(exist_ok=True,parents=True);work=args.output_dir/'worker-files';work.mkdir(exist_ok=True) + expected=json.loads((REPRO/'checkpoints.json').read_text());manifest=[];comparisons=[] + reference_index={} + if args.compare_root:reference_index={e['id']:e for e in json.loads((args.compare_root/'experiments/data/traces.json').read_text())} + for model in ('flint','qwen'): + selected=[c for c in cases if c['model']==model] + if not selected:continue + checkpoint=args.flint_model if model=='flint' else args.qwen_model + if checkpoint is None:raise ValueError('Supply the selected model checkpoint') + required=expected[model]['Q8_0'] + if checkpoint.stat().st_size!=required['bytes']:raise ValueError('Checkpoint byte count differs from the release manifest') + with checkpoint.open('rb') as f:checkpoint_sha=hashlib.file_digest(f,'sha256').hexdigest() + if checkpoint_sha!=required['sha256']:raise ValueError('Checkpoint SHA-256 differs from the release manifest') + architecture=json.loads((args.flint_config if model=='flint' else args.qwen_config).read_text());spec=work/(model+'-spec.json') + spec.write_text(json.dumps({'id':model,'quantization':'Q8_0','architecture':architecture,'stop_path':str((work/('stop-'+model)).resolve())})) + provenance={'producer':'alphaBell standalone capture study','engine_commit':'74686f084c53506a7afa471d857db02e5c912543','engine_source_tree':'83e9d4c276f84753cce063502a2a9e2325f15ce3','capture_source_sha256':hashlib.sha256((REPRO/'capture-study.cpp').read_bytes()).hexdigest(),'binary_sha256':hashlib.sha256(args.engine.read_bytes()).hexdigest(),'checkpoint_sha256':checkpoint_sha,'checkpoint_bytes':checkpoint.stat().st_size,'fixture_sha256':hashlib.sha256(args.fixtures.read_bytes()).hexdigest(),'device':'CPU','threads':16,'context':2048,'flash_attention':False,'weights_modified':False} + worker=Worker(args.engine,checkpoint,spec,work/(model+'.stderr.log')) + try: + for case in selected: + trace=worker.capture(case,architecture,provenance) + if args.compare_root: + entry=reference_index[case['id']];reference=json.loads(gzip.decompress((args.compare_root/entry['file']).read_bytes()));comparisons.append(compare_trace(trace,reference)) + raw=json.dumps(trace,ensure_ascii=False,separators=(',',':')).encode();compressed=gzip.compress(raw,mtime=0);name=case['id']+'.json.gz';(args.output_dir/name).write_bytes(compressed) + manifest.append({'id':case['id'],'model':model,'prompt':case['prompt'],'tokens':len(trace['frames']),'file':name,'sha256':hashlib.sha256(raw).hexdigest(),'gzip_sha256':hashlib.sha256(compressed).hexdigest(),'bytes':len(raw),'compressed_bytes':len(compressed),'created':trace['created'],'label':case['title']}) + (args.output_dir/'index.json').write_text(json.dumps(manifest,indent=2)+'\n');print(case['id'],len(trace['frames']),'complete',flush=True) + finally:worker.close() + report={'completed_at':datetime.datetime.now(datetime.timezone.utc).isoformat(),'captures':len(manifest),'frames':sum(c['tokens'] for c in manifest),'comparison_count':len(comparisons),'comparisons':comparisons,'all_selected_captures_complete':True} + (args.output_dir/'verification.json').write_text(json.dumps(report,indent=2)+'\n');print(json.dumps(report),flush=True) +if __name__=='__main__':main() diff --git a/experiments/reproduce/checkpoints.json b/experiments/reproduce/checkpoints.json new file mode 100644 index 0000000000000000000000000000000000000000..1eba3a7a2b64f7157b4891b1dcdd6ad544b7e751 --- /dev/null +++ b/experiments/reproduce/checkpoints.json @@ -0,0 +1,30 @@ +{ + "qwen": { + "Q8_0": { + "file": "qwen-q8_0.gguf", + "bytes": 28595763712, + "sha256": "b2a3172342ff220a8f9a06fc2fd252d1e84615ce8dfc00b2a66941610aa8f43c", + "gguf_name": "Qwen3.8 27B Open" + }, + "Q4_1": { + "file": "qwen-q4_1.gguf", + "bytes": 16829921792, + "sha256": "7c0814054c69120ff0e5b3ac3f625f98a9f86185fc39f5d71eae02cb54eb5825", + "gguf_name": "Qwen3.8 27B Open" + } + }, + "flint": { + "Q8_0": { + "file": "flint-q8_0.gguf", + "bytes": 42350359392, + "sha256": "f97ff2fc77564a05919357685ba8e1892bec279cf733fdb6e3ef0ea8c8e45543", + "gguf_name": "Flint Bf16" + }, + "Q4_1": { + "file": "flint-q4_1.gguf", + "bytes": 24955171680, + "sha256": "1d8ea8d2fd133baa54a7412d56d39d20499a9efc080b45486f23853e3f072143", + "gguf_name": "Flint Bf16" + } + } +} diff --git a/experiments/reproduce/flint-capture-equivalence.json b/experiments/reproduce/flint-capture-equivalence.json new file mode 100644 index 0000000000000000000000000000000000000000..d815a690ac7e9a2dbaccfc1d474816cb11354379 --- /dev/null +++ b/experiments/reproduce/flint-capture-equivalence.json @@ -0,0 +1,33 @@ +{ + "schema": "alphabell.gguf-equivalence.v1", + "original_container": { + "label": "Recorded Observatory Flint Q8_0 capture checkpoint", + "bytes": 42350359392, + "sha256": "147f02266f809e7bb4b28fdeef9606520d9693b601f8bc0d817caf11a2ed1e5b" + }, + "matched_container": { + "file": "flint-q8_0.gguf", + "bytes": 42350359392, + "sha256": "f97ff2fc77564a05919357685ba8e1892bec279cf733fdb6e3ef0ea8c8e45543" + }, + "method": "Read-only byte comparison of all tensor descriptors, padding and tensor payload; decoded comparison of every GGUF metadata value including tokenizer arrays.", + "tensor_descriptors_and_padding_equal": true, + "tensor_descriptor_region_bytes_compared": 49153, + "tensor_payload_equal": true, + "tensor_payload_bytes_compared": 42344889472, + "tensor_payload_offset": 5469920, + "tokenizer_values_equal": true, + "only_metadata_differences": [ + { + "key": "general.name", + "original": "Recursive Flint", + "matched": "Flint Bf16" + }, + { + "key": "general.size_label", + "original": "40B", + "matched": "256x2.1B" + } + ], + "passed": true +} diff --git a/experiments/reproduce/llama-flint-runtime.patch b/experiments/reproduce/llama-flint-runtime.patch new file mode 100644 index 0000000000000000000000000000000000000000..66e81f612920a94854ee63ba4a1cd33547f24f23 --- /dev/null +++ b/experiments/reproduce/llama-flint-runtime.patch @@ -0,0 +1,704 @@ +diff --git a/common/jinja/lexer.cpp b/common/jinja/lexer.cpp +index 598982c2fe4a30bc362c5140f2f904cca89f041e..bf2dfacdadd28815e05f527f3066ab8a62cbe71a 100644 +--- a/common/jinja/lexer.cpp ++++ b/common/jinja/lexer.cpp +@@ -1,7 +1,9 @@ + #include "lexer.h" + #include "runtime.h" ++#include "unicode.h" + + #include ++#include + #include + #include + #include +@@ -75,6 +77,24 @@ lexer_result lexer::tokenize(const std::string & source) { + } + // add escaped char + char escaped_char = src[pos++]; ++ if (escaped_char == 'x') { ++ if (pos + 1 >= src.size()) { ++ throw lexer_exception("incomplete hexadecimal escape", source, pos); ++ } ++ ++ unsigned int value = 0; ++ const char * first = &src[pos]; ++ const char * last = first + 2; ++ ++ auto [ptr, ec] = std::from_chars(first, last, value, 16); ++ if (ec != std::errc{} || ptr != last) { ++ throw lexer_exception("invalid hexadecimal escape", source, pos); ++ } ++ ++ str += common_unicode_cpt_to_utf8(value); ++ pos += 2; ++ continue; ++ } + if (escape_chars.find(escaped_char) == escape_chars.end()) { + throw lexer_exception(std::string("unknown escape character \\") + escaped_char, source, pos); + } +diff --git a/conversion/__init__.py b/conversion/__init__.py +index bddfe3c1f531a656cf0c1be79615f77f2fc7247b..1476a9901a1417aef1bec82bd3d62afcb65e3df4 100644 +--- a/conversion/__init__.py ++++ b/conversion/__init__.py +@@ -19,6 +19,7 @@ __all__ = [ + TEXT_MODEL_MAP: dict[str, str] = { + "AfmoeForCausalLM": "afmoe", + "LagunaForCausalLM": "laguna", ++ "FlintForCausalLM": "flint", + "ApertusForCausalLM": "llama", + "ArceeForCausalLM": "llama", + "ArcticForCausalLM": "arctic", +diff --git a/conversion/base.py b/conversion/base.py +index dec1d43222ea63d0b8cba923d5798dea8e404736..7d787181c4761870c861786742ae383eed953b88 100644 +--- a/conversion/base.py ++++ b/conversion/base.py +@@ -1701,7 +1701,7 @@ class TextModel(ModelBase): + self.gguf_writer.add_expert_group_used_count(n_group_used) + logger.info(f"gguf: expert groups used count = {n_group_used}") + +- if (score_func := self.find_hparam(["score_function", "scoring_func", "score_func", "moe_router_activation", "moe_router_activation_func", "expert_selection_fn", "router_score_func"], optional=True)) is not None: ++ if (score_func := self.find_hparam(["score_function", "scoring_func", "score_func", "moe_router_activation", "moe_router_activation_func", "expert_selection_fn", "router_score_func", "router_score_function"], optional=True)) is not None: + if score_func == "sigmoid": + self.gguf_writer.add_expert_gating_func(gguf.ExpertGatingFuncType.SIGMOID) + elif score_func == "softmax": +@@ -1939,6 +1939,9 @@ class TextModel(ModelBase): + if chkhsh == "52df12b4c8d4176e7481aab4b6e8454d1fd0a210a04a574f6d4e067d10e23c3e": + # ref: https://huggingface.co/CohereLabs/North-Mini-Code-1.0 + res = "cohere2moe" ++ if chkhsh == "d5420e601e979078c670bc503d9ac411c311fa76b76a76267c31e0d6ba816637": ++ # ref: https://huggingface.co/RecursiveSI/flint-preview ++ res = "flint" + if chkhsh == "e636dc30a262dcc0d8c323492e32ae2b70728f4df7dfe9737d9f920a282b8aea": + # ref: https://huggingface.co/Qwen/Qwen1.5-7B + res = "qwen2" +diff --git a/conversion/flint.py b/conversion/flint.py +new file mode 100644 +index 0000000000000000000000000000000000000000..1fd4776f294e76d6782005bd588314f733a216ad +--- /dev/null ++++ b/conversion/flint.py +@@ -0,0 +1,38 @@ ++from __future__ import annotations ++ ++from .base import ModelBase, gguf ++from .qwen import Qwen2MoeModel ++ ++ ++@ModelBase.register("FlintForCausalLM") ++@ModelBase.example("RecursiveSI/flint-preview") ++class FlintModel(Qwen2MoeModel): ++ model_arch = gguf.MODEL_ARCH.FLINT ++ ++ def set_gguf_parameters(self): ++ super().set_gguf_parameters() ++ dense = self.hparams["mlp_only_layers"] ++ if dense != list(range(len(dense))): ++ raise ValueError("Flint dense layers must precede the MoE layers") ++ self.gguf_writer.add_leading_dense_block_count(len(dense)) ++ self.gguf_writer.add_expert_shared_count( ++ self.hparams["shared_expert_intermediate_size"] // self.hparams["moe_intermediate_size"] ++ ) ++ self.gguf_writer.add_expert_weights_norm(True) ++ self.gguf_writer.add_expert_weights_scale(1.0) ++ self.gguf_writer.add_sliding_window(self.hparams["sliding_window"]) ++ self.gguf_writer.add_sliding_window_pattern([ ++ kind == "sliding_attention" for kind in self.hparams["layer_types"] ++ ]) ++ self.gguf_writer.add_rope_pattern([ ++ i not in self.hparams["no_rope_layer_indices"] for i in range(self.block_count) ++ ]) ++ ++ def modify_tensors(self, data_torch, name, bid): ++ if name.endswith(".softmax_offset"): ++ name += ".weight" ++ if name.endswith(".gate.expert_bias"): ++ if not self.hparams["router_expert_bias"]: ++ return ++ name += ".bias" ++ yield from super().modify_tensors(data_torch, name, bid) +diff --git a/convert_hf_to_gguf_update.py b/convert_hf_to_gguf_update.py +index d3aa3c3807f01d1343a08b6be1ba6188df01a314..683e061db68af946be5b43d706a292a7f6e96a48 100755 +--- a/convert_hf_to_gguf_update.py ++++ b/convert_hf_to_gguf_update.py +@@ -101,6 +101,7 @@ models = [ + {"name": "command-r", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/CohereForAI/c4ai-command-r-v01", }, + {"name": "tiny_aya", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/CohereLabs/tiny-aya-base", }, + {"name": "cohere2moe", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/CohereLabs/North-Mini-Code-1.0", }, ++ {"name": "flint", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/RecursiveSI/flint-preview", }, + {"name": "qwen2", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/Qwen/Qwen1.5-7B", }, + {"name": "olmo", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/allenai/OLMo-1.7-7B-hf", }, + {"name": "dbrx", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/databricks/dbrx-base", }, +diff --git a/gguf-py/gguf/constants.py b/gguf-py/gguf/constants.py +index 2840348cba242b7b286de98807b7504dfccc1d56..70d141c1e9ea16dc8c3160e1ef1771e73b7295c5 100644 +--- a/gguf-py/gguf/constants.py ++++ b/gguf-py/gguf/constants.py +@@ -575,6 +575,7 @@ class MODEL_ARCH(IntEnum): + QWEN2VL = auto() + QWEN3 = auto() + QWEN3MOE = auto() ++ FLINT = auto() + QWEN3NEXT = auto() + HRM_TEXT = auto() + BEAM = auto() +@@ -1391,6 +1392,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = { + MODEL_ARCH.QWEN2MOE: "qwen2moe", + MODEL_ARCH.QWEN2VL: "qwen2vl", + MODEL_ARCH.QWEN3: "qwen3", ++ MODEL_ARCH.FLINT: "flint", + MODEL_ARCH.QWEN3MOE: "qwen3moe", + MODEL_ARCH.QWEN3NEXT: "qwen3next", + MODEL_ARCH.HRM_TEXT: "hrm_text", +@@ -2953,6 +2955,33 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = { + MODEL_TENSOR.FFN_DOWN_EXP, + MODEL_TENSOR.FFN_UP_EXP, + ], ++ ++ MODEL_ARCH.FLINT: [ ++ MODEL_TENSOR.TOKEN_EMBD, ++ MODEL_TENSOR.OUTPUT_NORM, ++ MODEL_TENSOR.OUTPUT, ++ MODEL_TENSOR.ATTN_NORM, ++ MODEL_TENSOR.ATTN_QKV, ++ MODEL_TENSOR.ATTN_Q, ++ MODEL_TENSOR.ATTN_Q_NORM, ++ MODEL_TENSOR.ATTN_K, ++ MODEL_TENSOR.ATTN_K_NORM, ++ MODEL_TENSOR.ATTN_V, ++ MODEL_TENSOR.ATTN_OUT, ++ MODEL_TENSOR.FFN_NORM, ++ MODEL_TENSOR.FFN_GATE_INP, ++ MODEL_TENSOR.FFN_GATE_EXP, ++ MODEL_TENSOR.FFN_DOWN_EXP, ++ MODEL_TENSOR.FFN_UP_EXP, ++ MODEL_TENSOR.ATTN_SINKS, ++ MODEL_TENSOR.FFN_GATE, ++ MODEL_TENSOR.FFN_UP, ++ MODEL_TENSOR.FFN_DOWN, ++ MODEL_TENSOR.FFN_GATE_SHEXP, ++ MODEL_TENSOR.FFN_UP_SHEXP, ++ MODEL_TENSOR.FFN_DOWN_SHEXP, ++ MODEL_TENSOR.FFN_EXP_PROBS_B, ++ ], + MODEL_ARCH.QWEN3NEXT: [ + MODEL_TENSOR.TOKEN_EMBD, + MODEL_TENSOR.OUTPUT_NORM, +diff --git a/gguf-py/gguf/tensor_mapping.py b/gguf-py/gguf/tensor_mapping.py +index 3586d65285ca5f76446bb624cf384c006c76b386..79c9add78a1383b24a6f56212d2b995c19e69b9e 100644 +--- a/gguf-py/gguf/tensor_mapping.py ++++ b/gguf-py/gguf/tensor_mapping.py +@@ -429,6 +429,7 @@ class TensorNameMap: + ), + + MODEL_TENSOR.ATTN_SINKS: ( ++ "model.layers.{bid}.self_attn.softmax_offset", # flint + "model.layers.{bid}.self_attn.sinks", # openai-moe + "model.layers.{bid}.self_attn.attention_sink_bias", # mimov2 + "model.layers.{bid}.self_attn.learnable_sink_param", # hy-v4 +diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp +index 4534be22f37b5e088d50a359902a72723dfcb749..27236619ad03c2fdff6bc7b4a69d7b6b68ca7671 100644 +--- a/src/llama-arch.cpp ++++ b/src/llama-arch.cpp +@@ -34,6 +34,7 @@ static const std::map LLM_ARCH_NAMES = { + { LLM_ARCH_QWEN2MOE, "qwen2moe" }, + { LLM_ARCH_QWEN2VL, "qwen2vl" }, + { LLM_ARCH_QWEN3, "qwen3" }, ++ { LLM_ARCH_FLINT, "flint" }, + { LLM_ARCH_QWEN3MOE, "qwen3moe" }, + { LLM_ARCH_QWEN3NEXT, "qwen3next" }, + { LLM_ARCH_QWEN3VL, "qwen3vl" }, +diff --git a/src/llama-arch.h b/src/llama-arch.h +index 5a8ca425001cb82d2cc281ab8f4e05c5e079901a..54bb7003b3e852e7d65d1cac252aa6ee7b8a5e4b 100644 +--- a/src/llama-arch.h ++++ b/src/llama-arch.h +@@ -40,6 +40,7 @@ enum llm_arch { + LLM_ARCH_QWEN2VL, + LLM_ARCH_QWEN3, + LLM_ARCH_QWEN3MOE, ++ LLM_ARCH_FLINT, + LLM_ARCH_QWEN3NEXT, + LLM_ARCH_QWEN3VL, + LLM_ARCH_QWEN3VLMOE, +diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp +index 59caffe87fce5d1f5c9deee138a8f59cd32dbad1..1c84d8246c23d586a8fcd640eacfc18f9f553f1d 100644 +--- a/src/llama-graph.cpp ++++ b/src/llama-graph.cpp +@@ -1684,7 +1684,10 @@ llm_graph_amax_src1 llm_graph_context::build_amax_src1_f16( + constexpr float k_ln_2 = 0.6931471805599453f; + + ggml_tensor * e; +- e = ggml_scale(ctx0, amax, 1.0f/32768.0f); ++ // note: this can technically be: ++ // e = ggml_scale(ctx0, amax, 1.0f/32768.0f); ++ // but we leave x32 headroom for the F16 sums of 32 values in Q8 activation blocks ++ e = ggml_scale(ctx0, amax, 1.0f/1024.0f); + e = ggml_log (ctx0, e); + e = ggml_scale(ctx0, e, k_log2_e); + e = ggml_ceil (ctx0, e); +@@ -2505,7 +2508,7 @@ ggml_tensor * llm_graph_context::build_moe_ffn( + // ref: https://github.com/ggml-org/llama.cpp/pull/30656#issuecomment-6269597922 + llm_graph_amax_src1 amax_src1 = { nullptr, nullptr }; + +- if (arch == LLM_ARCH_MISTRAL4 || arch == LLM_ARCH_MELLUM) { ++ if (arch == LLM_ARCH_MISTRAL4 || arch == LLM_ARCH_MELLUM || arch == LLM_ARCH_FLINT) { + amax_src1 = build_amax_src1_f16(cur); + cur = amax_src1.res; + } +diff --git a/src/llama-graph.h b/src/llama-graph.h +index abe5133c97a22b380c46b3488111c5615dee7d43..58cbbae5bf83db45f7aff180dae024ced75d1978 100644 +--- a/src/llama-graph.h ++++ b/src/llama-graph.h +@@ -1121,7 +1121,7 @@ struct llm_graph_context { + ggml_tensor * slots = nullptr) const; + + // rescale `cur` (the src1 of a matmul) by a power of two, computed from its maximum absolute +- // when the values already fit in F16, the factors are exactly 1.0f ++ // keep both individual values and Q8 activation block sums within F16 range + llm_graph_amax_src1 build_amax_src1_f16( + ggml_tensor * cur) const; + +diff --git a/src/llama-kv-cache.cpp b/src/llama-kv-cache.cpp +index b7dfadef4638f6221608f48aff0a1f83df281063..6f0e3f7f796f8705317d219a2555d727d6c3a1c6 100644 +--- a/src/llama-kv-cache.cpp ++++ b/src/llama-kv-cache.cpp +@@ -882,7 +882,10 @@ bool llama_kv_cache::update(llama_context * lctx, bool do_shift, const stream_co + LLAMA_LOG_DEBUG("%s: applying K-shift\n", __func__); + + // apply K-shift if needed +- if (hparams.rope_type != LLAMA_ROPE_TYPE_NONE) { ++ if (hparams.rope_type != LLAMA_ROPE_TYPE_NONE && ++ std::any_of(layers.begin(), layers.end(), [&](const auto & layer) { ++ return hparams.has_rope(layer.il); ++ })) { + ggml_backend_sched_reset(sched); + + auto * res = lctx->get_gf_res_reserve(); +diff --git a/src/llama-model.cpp b/src/llama-model.cpp +index 6712f0f22f2a742f7c93d0f5bfebed25d53337ce..0dd49db7ef1c91f27bb12e661710e68db106e162 100644 +--- a/src/llama-model.cpp ++++ b/src/llama-model.cpp +@@ -112,6 +112,8 @@ static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params + return new llama_model_qwen2moe(params); + case LLM_ARCH_QWEN3: + return new llama_model_qwen3(params); ++ case LLM_ARCH_FLINT: ++ return new llama_model_flint(params); + case LLM_ARCH_QWEN3MOE: + return new llama_model_qwen3moe(params); + case LLM_ARCH_QWEN3VL: +@@ -3569,6 +3571,7 @@ llama_rope_type llama_model_rope_type(const llama_model * model) { + case LLM_ARCH_QWEN2MOE: + case LLM_ARCH_QWEN3: + case LLM_ARCH_VLLM_DECISION2: ++ case LLM_ARCH_FLINT: + case LLM_ARCH_QWEN3MOE: + case LLM_ARCH_LLADA_MOE: + case LLM_ARCH_RND1: +diff --git a/src/llama-vocab.cpp b/src/llama-vocab.cpp +index a3d5dedef33246a9d53fbf9738985fb82b37b396..05c0f96a29ccc136c952f1edc358bf2c2306382b 100644 +--- a/src/llama-vocab.cpp ++++ b/src/llama-vocab.cpp +@@ -281,6 +281,13 @@ struct llm_tokenizer_bpe : llm_tokenizer { + llm_tokenizer_bpe(const llama_vocab & vocab) { + GGML_ASSERT(vocab.get_type() == LLAMA_VOCAB_TYPE_BPE); + switch (vocab.get_pre_type()) { ++ case LLAMA_VOCAB_PRE_TYPE_FLINT: ++ regex_exprs = { ++ "\\p{N}{1,3}", ++ "[一-龥぀-ゟ゠-ヿ]+", ++ "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])?|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])?| ?[\\p{P}\\p{S}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+", ++ }; ++ break; + case LLAMA_VOCAB_PRE_TYPE_LLAMA3: + regex_exprs = { + // original regex from tokenizer.json +@@ -2417,6 +2424,9 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) { + LLAMA_LOG_WARN("%s: ************************************ \n", __func__); + LLAMA_LOG_WARN("%s: \n", __func__); + pre_type = LLAMA_VOCAB_PRE_TYPE_DEFAULT; ++ } else if (tokenizer_pre == "flint") { ++ pre_type = LLAMA_VOCAB_PRE_TYPE_FLINT; ++ clean_spaces = false; + } else if (tokenizer_pre == "default") { + pre_type = LLAMA_VOCAB_PRE_TYPE_DEFAULT; + } else if (tokenizer_pre == "minicpm5") { +@@ -3147,7 +3157,6 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) { + // this is currently determined based on the token text, which is obviously not ideal + // ref: https://github.com/ggml-org/llama.cpp/issues/9606 + special_eog_ids.clear(); +- + if (special_fim_pad_id != LLAMA_TOKEN_NULL && special_eog_ids.count(special_fim_pad_id) == 0) { + special_eog_ids.insert(special_fim_pad_id); + } +@@ -3181,6 +3190,7 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) { + || t.first == "[EOT]" // Kimi-K2 + || t.first == "[EOS]" // Kimi-K2 + || t.first == "<|end_of_text|>" ++ || (pre_type == LLAMA_VOCAB_PRE_TYPE_FLINT && (t.first == "<|user|>" || t.first == "<|observation|>")) + || t.first == "" // smoldocling + || t.first == "" // gemma4 + || t.first == "" // gemma4 +diff --git a/src/llama-vocab.h b/src/llama-vocab.h +index 415acd22bf40448094ea216f9e6487bb18c72134..a87c0de42b33297f143f27de6807cef303144962 100644 +--- a/src/llama-vocab.h ++++ b/src/llama-vocab.h +@@ -70,6 +70,7 @@ enum llama_vocab_pre_type { + LLAMA_VOCAB_PRE_TYPE_UFAKZEKA = 59, + LLAMA_VOCAB_PRE_TYPE_MMBERT = 60, + LLAMA_VOCAB_PRE_TYPE_K2_HORIZON = 61, ++ LLAMA_VOCAB_PRE_TYPE_FLINT = 62, + }; + + struct LLM_KV; +diff --git a/src/models/flint.cpp b/src/models/flint.cpp +new file mode 100644 +index 0000000000000000000000000000000000000000..e7e6f6b792df45b079434de06e1b372e829de13a +--- /dev/null ++++ b/src/models/flint.cpp +@@ -0,0 +1,194 @@ ++#include "models.h" ++ ++void llama_model_flint::load_arch_hparams(llama_model_loader & ml) { ++ ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps); ++ ml.get_key(LLM_KV_LEADING_DENSE_BLOCK_COUNT, hparams.n_layer_dense_lead); ++ ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all); ++ ml.get_key(LLM_KV_EXPERT_SHARED_COUNT, hparams.n_expert_shared); ++ ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE, hparams.expert_weights_scale, false); ++ ml.get_key(LLM_KV_EXPERT_WEIGHTS_NORM, hparams.expert_weights_norm); ++ ml.get_key(LLM_KV_EXPERT_GATING_FUNC, hparams.expert_gating_func); ++ ++ ml.get_key(LLM_KV_EXPERT_SHARED_FEED_FORWARD_LENGTH, hparams.n_ff_shexp); ++ ml.get_arr(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, hparams.is_swa_impl); ++ ml.get_arr(LLM_KV_ATTENTION_ROPE_PATTERN, hparams.rope_pattern); ++ ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa); ++ hparams.swa_type = LLAMA_SWA_TYPE_STANDARD; ++ type = LLM_TYPE_UNKNOWN; ++} ++ ++void llama_model_flint::load_arch_tensors(llama_model_loader &) { ++ LLAMA_LOAD_LOCALS; ++ ++ const int64_t n_ff_exp = hparams.n_ff_exp(); ++ ++ tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0); ++ ++ output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0); ++ output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab}, 0); ++ ++ for (int i = 0; i < n_layer; ++i) { ++ auto & layer = layers[i]; ++ ++ layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0); ++ ++ create_tensor_qkv(layer, i, n_embd, n_embd_head_k * n_head, n_embd_k_gqa, n_embd_v_gqa, 0); ++ layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), {n_embd_head_k * n_head, n_embd}, 0); ++ ++ layer.attn_k_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_NORM, "weight", i), {n_embd_head_k}, 0); ++ layer.attn_q_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_NORM, "weight", i), {n_embd_head_k}, 0); ++ ++ layer.attn_sinks = create_tensor(tn(LLM_TENSOR_ATTN_SINKS, "weight", i), {n_head}, 0); ++ ++ layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0); ++ ++ if (i < (int) hparams.n_layer_dense_lead) { ++ layer.ffn_gate = create_tensor(tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd, n_ff}, 0); ++ layer.ffn_down = create_tensor(tn(LLM_TENSOR_FFN_DOWN, "weight", i), { n_ff, n_embd}, 0); ++ layer.ffn_up = create_tensor(tn(LLM_TENSOR_FFN_UP, "weight", i), {n_embd, n_ff}, 0); ++ } else { ++ layer.ffn_gate_inp = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP, "weight", i), {n_embd, n_expert}, 0); ++ layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias", i), {n_expert}, TENSOR_NOT_REQUIRED); ++ ++ if (n_expert == 0) { ++ throw std::runtime_error("n_expert must be > 0"); ++ } ++ if (n_expert_used == 0) { ++ throw std::runtime_error("n_expert_used must be > 0"); ++ } ++ ++ // MoE branch ++ layer.ffn_gate_exps = create_tensor(tn(LLM_TENSOR_FFN_GATE_EXPS, "weight", i), { n_embd, n_ff_exp, n_expert}, 0); ++ layer.ffn_down_exps = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "weight", i), {n_ff_exp, n_embd, n_expert}, 0); ++ layer.ffn_up_exps = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS, "weight", i), { n_embd, n_ff_exp, n_expert}, 0); ++ ++ // Shared expert branch ++ layer.ffn_gate_shexp = create_tensor(tn(LLM_TENSOR_FFN_GATE_SHEXP, "weight", i), {n_embd, hparams.n_ff_shexp}, 0); ++ layer.ffn_down_shexp = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "weight", i), { hparams.n_ff_shexp, n_embd}, 0); ++ layer.ffn_up_shexp = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP, "weight", i), {n_embd, hparams.n_ff_shexp}, 0); ++ } ++ } ++} ++ ++std::unique_ptr llama_model_flint::build_arch_graph(const llm_graph_params & params) const { ++ return std::make_unique(*this, params); ++} ++ ++llama_model_flint::graph::graph(const llama_model & model, const llm_graph_params & params) : ++ llm_graph_context(params) { ++ const int64_t n_embd_head = hparams.n_embd_head_v(); ++ ++ GGML_ASSERT(n_embd_head == hparams.n_embd_head_k()); ++ GGML_ASSERT(n_embd_head == n_rot); ++ ++ ggml_tensor * cur; ++ ggml_tensor * inpL; ++ ++ inpL = build_inp_embd(model.tok_embd); ++ ++ // inp_pos - contains the positions ++ ggml_tensor * inp_pos = build_inp_pos(); ++ ++ auto * inp_attn = build_attn_inp_kv_iswa(); ++ ++ ggml_tensor * inp_out_ids = build_inp_out_ids(); ++ ++ for (int il = 0; il < n_layer; ++il) { ++ res->t_layer_inp[il] = inpL; ++ ggml_tensor * inpSA = inpL; ++ ++ // norm ++ cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il); ++ cb(cur, "attn_norm", il); ++ ++ // self_attention ++ { ++ // compute Q and K and RoPE them ++ auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur, ++ n_embd_head, n_head, n_head_kv, il); ++ ++ Qcur = build_norm(Qcur, model.layers[il].attn_q_norm, nullptr, LLM_NORM_RMS, il); ++ Kcur = build_norm(Kcur, model.layers[il].attn_k_norm, nullptr, LLM_NORM_RMS, il); ++ if (hparams.has_rope(il)) { ++ Qcur = ggml_rope_ext(ctx0, Qcur, inp_pos, nullptr, n_rot, rope_type, n_ctx_orig, ++ freq_base, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow); ++ Kcur = ggml_rope_ext(ctx0, Kcur, inp_pos, nullptr, n_rot, rope_type, n_ctx_orig, ++ freq_base, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow); ++ } ++ cb(Qcur, "Qcur", il); ++ cb(Kcur, "Kcur", il); ++ cb(Vcur, "Vcur", il); ++ ++ cur = build_attn(inp_attn, ++ model.layers[il].wo, model.layers[il].wo_b, model.layers[il].wo_s, ++ Qcur, Kcur, Vcur, nullptr, model.layers[il].attn_sinks, nullptr, 1.0f / sqrtf(float(n_embd_head)), il); ++ } ++ if (il == n_layer - 1 && inp_out_ids) { ++ cur = ggml_get_rows(ctx0, cur, inp_out_ids); ++ inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids); ++ } ++ ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA); ++ cb(ffn_inp, "ffn_inp", il); ++ ++ // MoE branch ++ cur = build_norm(ffn_inp, model.layers[il].ffn_norm, NULL, LLM_NORM_RMS, il); ++ cb(cur, "ffn_norm", il); ++ ++ if ((uint32_t) il < hparams.n_layer_dense_lead) { ++ cur = build_ffn(cur, ++ model.layers[il].ffn_up, NULL, model.layers[il].ffn_up_s, ++ model.layers[il].ffn_gate, NULL, model.layers[il].ffn_gate_s, ++ model.layers[il].ffn_down, NULL, model.layers[il].ffn_down_s, ++ NULL, LLM_FFN_SILU, LLM_FFN_PAR, il); ++ cb(cur, "ffn_out", il); ++ } else { ++ ggml_tensor * moe_out = build_moe_ffn(cur, ++ model.layers[il].ffn_gate_inp, ++ model.layers[il].ffn_up_exps, ++ model.layers[il].ffn_gate_exps, ++ model.layers[il].ffn_down_exps, ++ model.layers[il].ffn_exp_probs_b, ++ n_expert, n_expert_used, ++ LLM_FFN_SILU, hparams.expert_weights_norm, ++ hparams.expert_weights_scale, ++ (llama_expert_gating_func_type) hparams.expert_gating_func, ++ il, nullptr, nullptr, ++ model.layers[il].ffn_up_exps_s, ++ model.layers[il].ffn_gate_exps_s, ++ model.layers[il].ffn_down_exps_s); ++ cb(moe_out, "ffn_moe_out", il); ++ ++ { ++ ggml_tensor * ffn_shexp = ++ build_ffn(cur, ++ model.layers[il].ffn_up_shexp, NULL, model.layers[il].ffn_up_shexp_s, ++ model.layers[il].ffn_gate_shexp, NULL, model.layers[il].ffn_gate_shexp_s, ++ model.layers[il].ffn_down_shexp, NULL, model.layers[il].ffn_down_shexp_s, ++ NULL, LLM_FFN_SILU, LLM_FFN_PAR, il); ++ cb(ffn_shexp, "ffn_shexp", il); ++ ++ cur = ggml_add(ctx0, moe_out, ffn_shexp); ++ cb(cur, "ffn_out", il); ++ } ++ } ++ cur = ggml_add(ctx0, cur, ffn_inp); ++ ++ cur = build_cvec(cur, il); ++ cb(cur, "l_out", il); ++ ++ // input for next layer ++ inpL = cur; ++ } ++ cur = inpL; ++ ++ cur = build_norm(cur, model.output_norm, NULL, LLM_NORM_RMS, -1); ++ ++ cb(cur, "result_norm", -1); ++ res->t_embd = cur; ++ ++ // lm_head ++ cur = build_lora_mm(model.output, cur, model.output_s); ++ ++ cb(cur, "result_output", -1); ++ res->t_logits = cur; ++} +diff --git a/src/models/models.h b/src/models/models.h +index 9aa450deabff3ab532eb3719f8196166bfdb9a72..7a3f271e85a60b49de1e4a27c33f2f8a41d2ea9b 100644 +--- a/src/models/models.h ++++ b/src/models/models.h +@@ -678,6 +678,19 @@ struct llama_model_qwen3moe : public llama_model_base { + }; + + ++struct llama_model_flint : public llama_model_base { ++ llama_model_flint(const struct llama_model_params & params) : llama_model_base(params) {} ++ void load_arch_hparams(llama_model_loader & ml) override; ++ void load_arch_tensors(llama_model_loader & ml) override; ++ ++ struct graph : public llm_graph_context { ++ graph(const llama_model & model, const llm_graph_params & params); ++ }; ++ ++ std::unique_ptr build_arch_graph(const llm_graph_params & params) const override; ++}; ++ ++ + struct llama_model_qwen3vl : public llama_model_base { + llama_model_qwen3vl(const struct llama_model_params & params) : llama_model_base(params) {} + void load_arch_hparams(llama_model_loader & ml) override; +diff --git a/tests/fusion/MTL.csv b/tests/fusion/MTL.csv +index e6cd43a82b9bef3d435bed4bc33937931bf104e8..56044b350e457f5c2c0fe5d3d7d77a46cc4e220e 100644 +--- a/tests/fusion/MTL.csv ++++ b/tests/fusion/MTL.csv +@@ -71,6 +71,9 @@ falcon ,0 ,any ,ADD+ADD , 2 + falcon ,0 ,any ,NORM+MUL+ADD , 5 + falcon-h1 ,0 ,any ,ADD+ADD , 2 + falcon-h1 ,0 ,any ,RMS_NORM+MUL , 9 ++flint ,0 ,any ,ADD+ADD , 1 ++flint ,0 ,any ,MUL+ADD , 2 ++flint ,0 ,any ,RMS_NORM+MUL , 9 + gemma ,0 ,any ,RMS_NORM+MUL , 5 + gemma2 ,0 ,any ,RMS_NORM+MUL , 5 + gemma2 ,0 ,any ,RMS_NORM+MUL+ADD , 4 +diff --git a/tests/test-jinja.cpp b/tests/test-jinja.cpp +index ddeaa1fe9a799bd27acf5002ef9b37174049a643..327ab365c44ecaa6d3dcec689a6539b88dc53564 100644 +--- a/tests/test-jinja.cpp ++++ b/tests/test-jinja.cpp +@@ -25,6 +25,7 @@ static void test_expressions(testing & t); + static void test_set_statement(testing & t); + static void test_filters(testing & t); + static void test_literals(testing & t); ++static void test_hexadecimal_escapes(testing & t); + static void test_comments(testing & t); + static void test_macros(testing & t); + static void test_namespace(testing & t); +@@ -65,6 +66,7 @@ int main(int argc, char *argv[]) { + t.test("set statement", test_set_statement); + t.test("filters", test_filters); + t.test("literals", test_literals); ++ t.test("hexadecimal escapes", test_hexadecimal_escapes); + t.test("comments", test_comments); + t.test("macros", test_macros); + t.test("namespace", test_namespace); +@@ -1171,6 +1173,54 @@ static void test_literals(testing & t) { + ); + } + ++static void test_hexadecimal_escapes(testing & t) { ++ test_template(t, "remove NUL from message content", ++ R"({{ message.content | replace("\x00", "") }})", ++ {{"message", {{"content", std::string("\0a\0b\0", 5)}}}}, ++ "ab" ++ ); ++ ++ test_template(t, "non-ASCII hex escape produces UTF-8", ++ R"({{ '\xE9' }})", ++ json::object(), ++ "\xc3\xa9" ++ ); ++ ++ test_template(t, "consume exactly two hex digits", ++ R"({{ '\x414' }})", ++ json::object(), ++ "A4" ++ ); ++ ++ test_template(t, "escaped backslashes remain literal", ++ R"({{ '\\xE9' }})", ++ json::object(), ++ R"(\xE9)" ++ ); ++ ++ if (!g_python_mode) { ++ const char * invalid_templates[] = { ++ R"({{ '\x' }})", ++ R"({{ '\x0' }})", ++ R"({{ '\xG1' }})", ++ R"({{ '\x1G' }})", ++ R"({{ '\x)", ++ R"({{ '\x0)", ++ }; ++ for (const auto * tmpl : invalid_templates) { ++ t.test("reject " + json(tmpl).dump(), [&](testing & t) { ++ bool rejected = false; ++ try { ++ jinja::lexer().tokenize(tmpl); ++ } catch (const jinja::lexer_exception &) { ++ rejected = true; ++ } ++ t.assert_true("malformed hex escape must raise a lexer error", rejected); ++ }); ++ } ++ } ++} ++ + static void test_comments(testing & t) { + test_template(t, "inline comment", + "before{# comment #}after", +diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp +index 21d27e17d930eee2e2555f01d80fbd7a81cd4927..e1cd5867ee0e57383b7775d49b7b8df8acd69af3 100644 +--- a/tests/test-llama-archs.cpp ++++ b/tests/test-llama-archs.cpp +@@ -358,13 +358,16 @@ static gguf_context_ptr get_gguf_ctx(const llm_arch arch, const bool moe) { + ms.add_kv(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, uint32_t(5)); + } else if (arch == LLM_ARCH_COHERE2MOE || arch == LLM_ARCH_MIMO2 || arch == LLM_ARCH_STEP35 || arch == LLM_ARCH_SPARK2_5 || + arch == LLM_ARCH_MUSE_GLIMMER || arch == LLM_ARCH_MUSE_SPARK || arch == LLM_ARCH_GRANITE_SWA || arch == LLM_ARCH_DOTS3NOTE || +- arch == LLM_ARCH_KOLIBRI1 || arch == LLM_ARCH_MAPLE) { ++ arch == LLM_ARCH_KOLIBRI1 || arch == LLM_ARCH_MAPLE || arch == LLM_ARCH_FLINT) { + std::vector pattern; + pattern.reserve(n_layer); + for (uint32_t il = 0; il < n_layer; il++) { + pattern.push_back(il % 2); + } + ms.add_kv(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, pattern); ++ if (arch == LLM_ARCH_FLINT) { ++ ms.add_kv(LLM_KV_ATTENTION_ROPE_PATTERN, pattern); ++ } + } else { + ms.add_kv(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, uint32_t(2)); + } +@@ -522,6 +525,9 @@ static gguf_context_ptr get_gguf_ctx(const llm_arch arch, const bool moe) { + ms.add_kv(LLM_KV_EXPERT_GATING_FUNC, gating_func); // sigmoid : softmax : sqrtsoftplus + ms.add_kv(LLM_KV_EXPERT_GROUP_SCALE, 1.0f); + ms.add_kv(LLM_KV_EXPERTS_PER_GROUP, uint32_t(1)); ++ if (arch == LLM_ARCH_FLINT) { ++ ms.add_kv(LLM_KV_EXPERT_WEIGHTS_NORM, true); ++ } + if (arch == LLM_ARCH_K2_HORIZON) { + ms.add_kv(LLM_KV_ATTENTION_VALUE_EXPERT_COUNT, uint32_t(2)); + ms.add_kv(LLM_KV_ATTENTION_VALUE_EXPERT_USED_COUNT, uint32_t(2)); +@@ -751,6 +757,7 @@ static bool check_causal_attn_toggle( + + static bool moe_mandatory(const llm_arch arch) { + switch (arch) { ++ case LLM_ARCH_FLINT: + case LLM_ARCH_LLAMA4: + case LLM_ARCH_MAION_CODER: + case LLM_ARCH_COHERE2MOE: diff --git a/experiments/reproduce/quant-surgery-build.sh b/experiments/reproduce/quant-surgery-build.sh new file mode 100644 index 0000000000000000000000000000000000000000..52dccda13248c83ee35046b08c5f47ff8cd71653 --- /dev/null +++ b/experiments/reproduce/quant-surgery-build.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +set -euo pipefail +if [ "$#" -ne 3 ]; then + echo 'usage: quant-surgery-build.sh LLAMA_SOURCE LLAMA_LIBRARY_DIR OUTPUT' >&2 + exit 2 +fi +experiment_dir=$(cd "$(dirname "$0")" && pwd) +expected_tree=83e9d4c276f84753cce063502a2a9e2325f15ce3 +actual_tree=$(git -C "$1" write-tree) +if [ "$actual_tree" != "$expected_tree" ] || ! git -C "$1" diff --quiet; then + echo 'Runtime source differs from the measured source tree. Follow README.md.' >&2 + exit 1 +fi +nice -n 10 c++ -std=c++17 -O2 \ + -I"$1/include" -I"$1/ggml/include" -I"$1/vendor" \ + "$experiment_dir/quant-surgery.cpp" -L"$2" -Wl,-rpath,"$2" \ + -l:libllama.so.0 -lggml -lggml-base -pthread -o "$3" diff --git a/experiments/reproduce/quant-surgery.cpp b/experiments/reproduce/quant-surgery.cpp new file mode 100644 index 0000000000000000000000000000000000000000..f163c63023c7ed8a6ac973011be5c5574931956d --- /dev/null +++ b/experiments/reproduce/quant-surgery.cpp @@ -0,0 +1,112 @@ +#include "llama.h" +#include "ggml.h" +#include "ggml-backend.h" +#include "nlohmann/json.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using json=nlohmann::json; +using Clock=std::chrono::steady_clock; +static std::string STOP,MODEL_ID,ACTIVE_ID; +static void emit(json j){j["model"]=MODEL_ID;if(!ACTIVE_ID.empty())j["id"]=ACTIVE_ID;std::cout<(Clock::now()-t).count();} +static void quiet(enum ggml_log_level level,const char*text,void*){if(level>=GGML_LOG_LEVEL_WARN)std::cerr<():config.at("num_attention_heads").get();} + int dimension(int l)const{return linear(l)?config.at("linear_value_head_dim").get():config.at("head_dim").get();} +}; +static bool stopped(void*){ + if(STOP.empty()||access(STOP.c_str(),F_OK)!=0)return false; + std::ifstream file(STOP);std::string id;std::getline(file,id);return id==ACTIVE_ID; +} +static float at(const ggml_tensor*t,const std::vector&b,int x,int y=0,int z=0,int w=0){ + size_t off=x*t->nb[0]+y*t->nb[1]+z*t->nb[2]+w*t->nb[3]; + if(off+ggml_type_size(t->type)>b.size())throw std::runtime_error(std::string(t->name)+" tensor stride outside capture"); + if(t->type==GGML_TYPE_F32){float v;memcpy(&v,b.data()+off,4);return v;} + if(t->type==GGML_TYPE_F16){ggml_fp16_t v;memcpy(&v,b.data()+off,2);return ggml_fp16_to_fp32(v);} + if(t->type==GGML_TYPE_I32){int32_t v;memcpy(&v,b.data()+off,4);return v;} + throw std::runtime_error("Unsupported activation dtype"); +} +static double intervention_scale=1.0; +static int intervention_layer=12; +static bool capture(ggml_tensor*t,bool ask,void*user){ + auto&c=*static_cast(user);if(!c.enabled)return !ask; + std::string name=t->name;auto dash=name.rfind('-');if(dash==std::string::npos)return !ask; + std::string kind=name.substr(0,dash);int layer=std::stoi(name.substr(dash+1)); + bool wanted=(kind=="ffn_out"&&(layer>=11&&layer<=13))||(kind=="l_out"&&layer==13); + if(ask)return wanted;if(!wanted)return true; + if(t->type!=GGML_TYPE_F32&&t->type!=GGML_TYPE_F16)throw std::runtime_error("Unexpected intervention activation dtype"); + if(!ggml_is_contiguous(t))throw std::runtime_error("Intervention requires a contiguous FFN output"); + std::vectorbuf(ggml_nbytes(t));ggml_backend_tensor_get(t,buf.data(),0,buf.size()); + auto rms=[&](){double sq=0;int last=t->ne[1]-1;for(int x=0;xne[0];x++){double v=at(t,buf,x,last);sq+=v*v;}return std::sqrt(sq/t->ne[0]);}; + double before=rms();bool changed=MODEL_ID=="flint"&&kind=="ffn_out"&&layer==intervention_layer&&intervention_scale!=1; + if(changed){ + for(int64_t i=0;itype==GGML_TYPE_F32){float v;memcpy(&v,buf.data()+4*i,4);v*=intervention_scale;memcpy(buf.data()+4*i,&v,4);} + else{ggml_fp16_t v;memcpy(&v,buf.data()+2*i,2);v=ggml_fp32_to_fp16(ggml_fp16_to_fp32(v)*intervention_scale);memcpy(buf.data()+2*i,&v,2);} + } + ggml_backend_tensor_set(t,buf.data(),0,buf.size()); + } + double after=rms();c.layers.push_back({{"layer",layer},{"tensor",kind},{"before_rms",before},{"after_rms",after},{"scale",changed?intervention_scale:1.0},{"tensor_bytes",ggml_nbytes(t)}}); + return !stopped(nullptr); +} +static std::string piece(const llama_vocab*v,llama_token t){ + std::vectorb(512);int n=llama_token_to_piece(v,t,b.data(),b.size(),0,true);if(n<0){b.resize(-n);n=llama_token_to_piece(v,t,b.data(),b.size(),0,true);}return std::string(b.data(),std::max(0,n)); +} +int main(int argc,char**argv){ + if(argc<3){std::cerr<<"usage: quant-surgery MODEL.gguf SPEC.json\n";return 2;} + Capture c;json spec; + try{std::ifstream file(argv[2]);file>>spec;c.config=spec.at("architecture");MODEL_ID=spec.at("id");STOP=spec.at("stop_path");}catch(const std::exception&e){emit({{"type","fatal"},{"error",std::string("Invalid model specification: ")+e.what()}});return 1;} + llama_log_set(quiet,nullptr);llama_backend_init();auto mp=llama_model_default_params();mp.n_gpu_layers=0;ggml_backend_dev_t devices[]={nullptr};mp.devices=devices; + auto model=llama_model_load_from_file(argv[1],mp);if(!model){emit({{"type","fatal"},{"error","Could not load model checkpoint"}});return 1;} + if(llama_model_n_layer(model)!=c.config.at("num_hidden_layers").get()||llama_model_n_embd(model)!=c.config.at("hidden_size").get()){emit({{"type","fatal"},{"error","Checkpoint dimensions do not match the model specification"}});return 1;} + auto vocab=llama_model_get_vocab(model); + auto cp=llama_context_default_params();cp.n_ctx=2048;cp.n_batch=256;cp.n_ubatch=256;cp.n_threads=8;cp.n_threads_batch=8;cp.n_rs_seq=0; + cp.flash_attn_type=LLAMA_FLASH_ATTN_TYPE_DISABLED;cp.cb_eval=capture;cp.cb_eval_user_data=&c;cp.abort_callback=stopped; + auto ctx=llama_init_from_model(model,cp);if(!ctx){emit({{"type","fatal"},{"error","Could not initialize tracing context"}});return 1;} + emit({{"type","ready"},{"engine","llama.cpp 74686f08"},{"quantization",spec.at("quantization")},{"device","CPU"},{"threads",8},{"context",2048}}); + std::string line; + while(std::getline(std::cin,line)){ + llama_sampler*sampler=nullptr; + try{ + auto req=json::parse(line);intervention_scale=req.value("intervention_scale",1.0);intervention_layer=req.value("intervention_layer",12);if(intervention_scale<0||intervention_scale>1)throw std::runtime_error("Scale must be between zero and one");if(MODEL_ID!="flint"&&intervention_scale!=1)throw std::runtime_error("Intervention supports Flint only");ACTIVE_ID=req.value("id","manual");unlink(STOP.c_str());llama_memory_clear(llama_get_memory(ctx),true); + std::string prompt=req.at("prompt");for(size_t pos=0;(pos=prompt.find("<|",pos))!=std::string::npos;pos+=5)prompt.replace(pos,2,"<|"); + bool reasoning=req.value("reasoning",false),flint=MODEL_ID=="flint"; + std::string system="You are a helpful assistant. Answer clearly and concisely."; + std::string text=flint?"<|system|>"+system+"<|user|>"+prompt+"<|assistant|>"+(reasoning?"":""):"<|im_start|>system\n"+system+"<|im_end|>\n<|im_start|>user\n"+prompt+"<|im_end|>\n<|im_start|>assistant\n\n"+(reasoning?"":"\n\n\n"); + int n=-llama_tokenize(vocab,text.data(),text.size(),nullptr,0,flint,true); + if(n<=0)throw std::runtime_error("Could not tokenize the prompt"); + if(n>256)throw std::runtime_error("Prompt exceeds the tracing limit of 256 tokens. Shorten it and run again."); + std::vectortokens(n);llama_tokenize(vocab,text.data(),text.size(),tokens.data(),n,flint,true); + json pieces=json::array();for(auto t:tokens)pieces.push_back(piece(vocab,t)); + int limit=std::clamp(req.value("max_tokens",64),1,256);sampler=llama_sampler_chain_init(llama_sampler_chain_default_params());double temperature=req.value("temperature",0.0); + if(temperature<=0)llama_sampler_chain_add(sampler,llama_sampler_init_greedy());else{llama_sampler_chain_add(sampler,llama_sampler_init_top_k(40));llama_sampler_chain_add(sampler,llama_sampler_init_top_p(.95,1));llama_sampler_chain_add(sampler,llama_sampler_init_temp(temperature));llama_sampler_chain_add(sampler,llama_sampler_init_dist(req.value("seed",42)));} + emit({{"type","start"},{"prompt_tokens",n},{"tokens",pieces},{"max_tokens",limit},{"temperature",temperature},{"seed",req.value("seed",42)},{"reasoning",reasoning}}); + auto started=Clock::now();int pos=0,count=n,generated=0;std::string reason="limit"; + for(int step=0;step(p.begin(),p.end())},{"input_position",c.position},{"input_text",pieces.back()},{"layers",c.layers},{"elapsed",seconds(started)},{"duration",seconds(before)},{"phase",step==0?"prefill":"decode"}}); + pieces.push_back(p);tokens={id};count=1;if(stopped(nullptr)){reason="stopped";break;} + } + c.enabled=false;llama_sampler_free(sampler);sampler=nullptr;struct rusage usage;getrusage(RUSAGE_SELF,&usage);emit({{"max_rss_kib",usage.ru_maxrss},{"intervention_scale",intervention_scale},{"intervention_layer",intervention_layer},{"type","done"},{"reason",reason},{"generated",generated},{"elapsed",seconds(started)}}); + }catch(const std::exception&e){c.enabled=false;if(sampler)llama_sampler_free(sampler);emit({{"type","error"},{"error",e.what()}});} + ACTIVE_ID.clear(); + } + llama_free(ctx);llama_model_free(model);llama_backend_free(); +} diff --git a/experiments/reproduce/quant-surgery.py b/experiments/reproduce/quant-surgery.py new file mode 100644 index 0000000000000000000000000000000000000000..93eb6b9d9f6d620c6f9d53baf28f3b495ca4ea49 --- /dev/null +++ b/experiments/reproduce/quant-surgery.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Run matched CPU quantization and reversible activation-scaling experiments.""" +import argparse, datetime, hashlib, json, math, pathlib, queue, re, subprocess, threading, time + +CASES = [ + {"id":"arithmetic","title":"Exact arithmetic","prompt":"Calculate (17 * 23) - 19. Reply with the integer only.","check":"integer","expected":"372"}, + {"id":"json","title":"Strict JSON format","prompt":"Return exactly this JSON object and no other text: {\"status\":\"ok\",\"count\":3}","check":"json","expected":{"status":"ok","count":3}}, + {"id":"fact","title":"Short factual recall","prompt":"What is the chemical symbol for gold? Reply with the symbol only.","check":"exact","expected":"Au"}, + {"id":"correction","title":"Use the corrected fact","prompt":"The access color was blue. Correction: the access color is now amber. What is the current access color? Reply with one word only.","check":"exact","expected":"amber"}, + {"id":"code","title":"Python expression repair","prompt":"Complete the function with one Python expression only, no markdown: def is_even(n): return ___ . The function must return True exactly when integer n is even.","check":"expression","expected":"n % 2 == 0"}, + {"id":"brief","title":"One sentence explanation","prompt":"Explain photosynthesis in exactly one sentence containing the words sunlight and glucose.","check":"sentence","expected":["sunlight","glucose"]}, +] + +def evaluate(case, output): + text=output.strip() + if case['check']=='integer': passed=text==case['expected'] + elif case['check']=='exact': passed=text.casefold()==case['expected'].casefold() + elif case['check']=='json': + try: passed=json.loads(text)==case['expected'] + except (ValueError,TypeError): passed=False + elif case['check']=='sentence': passed=all(w in text.lower() for w in case['expected']) and len(re.findall(r'[.!?]+(?:\s|$)',text))==1 and '\n' not in text + elif case['check']=='expression': + # Compare a small, explicit set of valid expressions. Do not execute model code. + compact=re.sub(r'\s+','',text).lower() + passed=compact in {'n%2==0','(n%2)==0','not(n%2)','notn%2','(n&1)==0','n&1==0','n%2!=1'} + else: raise ValueError(case['check']) + return {"passed":passed,"rule":case['check'],"expected":case['expected'],"character_count":len(text),"word_count":len(text.split())} + +class Engine: + def __init__(self,binary,checkpoint,spec,log): + self.log=open(log,'w'); self.q=queue.Queue();started=time.perf_counter() + self.p=subprocess.Popen(['nice','-n','10',str(binary.resolve()),str(checkpoint.resolve()),str(spec.resolve())],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=self.log,text=True,bufsize=1) + def read(): + try: + for line in self.p.stdout: + try:self.q.put(json.loads(line)) + except ValueError:self.q.put({'type':'error','error':'Non-JSON native output'}) + finally:self.q.put({'type':'eof'}) + self.thread=threading.Thread(target=read,daemon=True);self.thread.start() + self.ready=self.q.get(timeout=300) + if self.ready.get('type')!='ready':self.close();raise RuntimeError(self.ready) + self.load_seconds=time.perf_counter()-started + def run(self,case,condition): + request={"id":case['id']+'-'+condition['id'],"prompt":case['prompt'],"max_tokens":64,"temperature":0,"seed":42,"reasoning":False,"intervention_layer":12,"intervention_scale":condition.get('scale',1.0)} + self.p.stdin.write(json.dumps(request)+'\n');self.p.stdin.flush();started=time.perf_counter();frames=[];first=None;start=None + while True: + event=self.q.get(timeout=180) + if event['type']=='start':start=event + if event['type']=='token': + if first is None:first=time.perf_counter()-started + frames.append(event) + if event['type'] in ('error','fatal','eof'):raise RuntimeError(event) + if event['type']=='done':break + wall=time.perf_counter()-started + output=b''.join(bytes(f['token_bytes']) for f in frames).decode('utf-8',errors='replace') + if event['reason']!='eos':print('TRUNCATED',request['id'],flush=True) + metrics={"generated_tokens":len(frames),"prompt_tokens":start['prompt_tokens'],"elapsed_seconds":event['elapsed'],"request_wall_seconds":wall,"time_to_first_token_seconds":first,"decode_tokens_per_second":(len(frames)-1)/(event['elapsed']-frames[0]['elapsed']) if len(frames)>1 else None,"process_peak_rss_kib":event['max_rss_kib'],"termination":event['reason'],"complete":event['reason']=='eos'} + result={**{k:v for k,v in condition.items() if k!='scale'},"output":output,"metrics":metrics,"check":evaluate(case,output),"settings":{k:request[k] for k in ('max_tokens','temperature','seed','reasoning','intervention_layer','intervention_scale')},"measurements":[{"step":f['step'],"token_id":f['token_id'],"input_position":f['input_position'],"layers":f['layers']} for f in frames]} + return result + def close(self): + if self.p.poll() is None: + try:self.p.stdin.close();self.p.wait(timeout=30) + except (BrokenPipeError,subprocess.TimeoutExpired):self.p.terminate();self.p.wait(timeout=30) + self.log.close() + +def validate_results(results): + aggregate={"quantization_pairs":0,"surgery_cases":0,"intervention_frames":0,"upstream_first_frame_matches":0,"downstream_first_frame_changes":0,"complete_generations":0} + for case in results['quantization_cases']: + if {c['quant'] for c in case['conditions']}!={'Q4_1','Q8_0'}:raise ValueError('Incomplete quantization pair') + aggregate['quantization_pairs']+=1 + for condition in case['conditions']: + if not condition['metrics']['complete']:raise ValueError('Truncated generation') + aggregate['complete_generations']+=1 + for case in results['surgery_cases']: + base=case['conditions'][0];base_first={(v['layer'],v['tensor']):v for v in base['measurements'][0]['layers']} + for condition in case['conditions']: + scale=condition['settings']['intervention_scale'] + for frame in condition['measurements']: + v=next(x for x in frame['layers'] if x['layer']==12 and x['tensor']=='ffn_out') + if not math.isclose(v['after_rms'],v['before_rms']*scale,rel_tol=2e-6,abs_tol=1e-12):raise ValueError('Intervention scaling mismatch') + aggregate['intervention_frames']+=1 + first={(v['layer'],v['tensor']):v for v in condition['measurements'][0]['layers']} + if first[11,'ffn_out']['after_rms']!=base_first[11,'ffn_out']['after_rms']:raise ValueError('Upstream control changed') + aggregate['upstream_first_frame_matches']+=1 + if scale!=1: + if first[13,'l_out']['after_rms']==base_first[13,'l_out']['after_rms']:raise ValueError('Downstream intervention not observed') + aggregate['downstream_first_frame_changes']+=1 + if not condition['metrics']['complete']:raise ValueError('Truncated surgery generation') + aggregate['complete_generations']+=1 + aggregate['surgery_cases']+=1 + aggregate['passed']=True + return aggregate + +def main(): + ap=argparse.ArgumentParser();ap.add_argument('--engine',type=pathlib.Path,required=True);ap.add_argument('--flint-q8',type=pathlib.Path,required=True);ap.add_argument('--flint-q4',type=pathlib.Path,required=True);ap.add_argument('--qwen-q8',type=pathlib.Path,required=True);ap.add_argument('--qwen-q4',type=pathlib.Path,required=True);ap.add_argument('--flint-config',type=pathlib.Path,required=True);ap.add_argument('--qwen-config',type=pathlib.Path,required=True);ap.add_argument('--output',type=pathlib.Path,required=True);args=ap.parse_args() + out=args.output.resolve();out.parent.mkdir(parents=True,exist_ok=True) + spec_dir=out.parent/'specs';spec_dir.mkdir(exist_ok=True) + expected_checkpoints=json.loads(pathlib.Path(__file__).with_name('checkpoints.json').read_text()) + provenance={"engine":"llama.cpp 74686f084c53506a7afa471d857db02e5c912543","device":"CPU","threads":8,"nice":10,"context":2048,"flash_attention":False,"kv_cache":"default F16 attention KV; model-native recurrent state","sampling":{"temperature":0,"seed":42,"max_tokens":64,"reasoning":False},"intervention":{"tensor":"ffn_out-12","layer_indexing":"zero-based","applies_to":"all prompt and generated positions on each forward pass","operation":"scale computed FFN output before residual addition","scales":[1,0.5,0],"weights_modified":False},"memory_metric":"ru_maxrss process high-water mark, includes model loading and warm-up; KiB on Linux","latency_metric":"native monotonic request elapsed plus client wall clock; first-token latency includes prompt prefill","created_at":datetime.datetime.now(datetime.timezone.utc).isoformat(),"source_sha256":hashlib.sha256(pathlib.Path(__file__).with_suffix('.cpp').read_bytes()).hexdigest(),"binary_sha256":hashlib.sha256(args.engine.read_bytes()).hexdigest(),"hardware":{"cpu":next(v["data"] for v in json.loads(subprocess.check_output(["lscpu","-J"],text=True))["lscpu"] if v["field"]=="Model name:")},"checkpoints":[],"warmups":[]} + results={"schema":"alphabell.quant-surgery.v1","provenance":provenance,"quantization_cases":[],"surgery_cases":[]} + records={c['id']:{"id":c['id'],"title":c['title'],"prompt":c['prompt'],"checks":{"rule":c['check'],"expected":c['expected']},"conditions":[]} for c in CASES} + def save(): + temp=out.with_suffix('.tmp');temp.write_text(json.dumps(results,ensure_ascii=False,indent=2));temp.replace(out) + for model,quant,path in [('flint','Q8_0',args.flint_q8),('flint','Q4_1',args.flint_q4),('qwen','Q8_0',args.qwen_q8),('qwen','Q4_1',args.qwen_q4)]: + expected=expected_checkpoints[model][quant] + if path.stat().st_size!=expected['bytes']:raise ValueError('Checkpoint byte count differs from release manifest') + with path.open('rb') as checkpoint_file: checkpoint_sha=hashlib.file_digest(checkpoint_file,'sha256').hexdigest() + if checkpoint_sha!=expected['sha256']:raise ValueError('Checkpoint SHA-256 differs from release manifest') + config=json.loads((args.flint_config if model=='flint' else args.qwen_config).read_text()) + spec=spec_dir/f'{model}-{quant}.json';spec.write_text(json.dumps({'id':model,'quantization':quant,'architecture':config,'stop_path':str(spec_dir/f'stop-{model}-{quant}')})) + provenance['checkpoints'].append({'model':model,'quantization':quant,'file_bytes':path.stat().st_size,'sha256':checkpoint_sha,'gguf_name':expected['gguf_name'],'architecture_sha256':hashlib.sha256(json.dumps(config,sort_keys=True).encode()).hexdigest()}) + engine=Engine(args.engine,path,spec,out.parent/f'{model}-{quant}.stderr.log') + try: + warmcase={'id':'warmup','prompt':'What is 2 plus 2? Reply with the integer only.','check':'integer','expected':'4'} + warm=engine.run(warmcase,{'id':'warmup','model':model,'quant':quant,'label':'Warm-up','scale':1});provenance['warmups'].append({'model':model,'quant':quant,'load_seconds':engine.load_seconds,'metrics':warm['metrics']}) + for case in CASES: + condition={'id':f'{model}-{quant.lower()}','label':quant,'model':model,'quant':quant,'scale':1} + result=engine.run(case,condition);records[case['id']]['conditions'].append(result) + if model=='flint' and quant=='Q8_0': + surgery={"id":case['id'],"title":case['title'],"prompt":case['prompt'],"checks":records[case['id']]['checks'],"intervention":provenance['intervention'],"conditions":[{**result,'id':'baseline','label':'Baseline, scale 1'}]} + for scale,label in [(0.5,'Half FFN, scale 0.5'),(0,'Zero FFN, scale 0')]: + altered=engine.run(case,{'id':'half' if scale else 'zero','label':label,'model':model,'quant':quant,'scale':scale});surgery['conditions'].append(altered) + results['surgery_cases'].append(surgery) + results['quantization_cases']=[{**records[c['id']],"id":modelname+'-'+c['id'],"model":modelname,"conditions":[x for x in records[c['id']]['conditions'] if x['model']==modelname]} for modelname in ('flint','qwen') for c in CASES if any(x['model']==modelname for x in records[c['id']]['conditions'])] + save();print(model,quant,case['id'],len(result['output']),result['check']['passed'],round(result['metrics']['elapsed_seconds'],2),flush=True) + finally:engine.close() + results['validation']=validate_results(results);provenance['completed_at']=datetime.datetime.now(datetime.timezone.utc).isoformat();save() + print('COMPLETE',out,flush=True) +if __name__=='__main__':main() diff --git a/experiments/sound.css b/experiments/sound.css new file mode 100644 index 0000000000000000000000000000000000000000..cb35a09b945888bcbcedfb72f47d033993928c52 --- /dev/null +++ b/experiments/sound.css @@ -0,0 +1,4 @@ +:root{color-scheme:light;--bg:#f4f7ed;--paper:#fffef8;--ink:#203c30;--muted:#60745f;--line:#d5dfcb;--green:#285c43}*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--ink);font:15px/1.6 system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif}a{color:var(--green);text-underline-offset:4px}header,main,footer{width:min(1180px,calc(100% - 48px));margin:auto}header{height:88px;display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid var(--line);gap:20px}header>a:last-child{font-size:13px}.brand{text-decoration:none;font-family:Georgia,serif;font-size:38px;display:flex;align-items:center;gap:10px}.brand span{font-family:system-ui,sans-serif;font-size:20px;font-weight:600;letter-spacing:-.6px}main{display:grid;grid-template-columns:.85fr 1.15fr;gap:52px;padding:66px 0}.eyebrow{font:11px/1.6 ui-monospace,monospace;letter-spacing:1.8px;color:var(--muted);margin:0 0 22px}h1{font:normal clamp(44px,5.8vw,76px)/1.04 Georgia,serif;letter-spacing:-2.5px;margin:0 0 28px}h2{font:normal 34px/1.12 Georgia,serif;letter-spacing:-.7px;margin:0 0 20px}.lede{font-size:19px;line-height:1.65}.honest{font-size:13px;border-left:2px solid #9fb58f;padding-left:16px;color:var(--muted);max-width:350px}.studio{min-width:0;border:1px solid var(--line);border-radius:16px;background:var(--paper);padding:24px;box-shadow:0 10px 32px #31552c08}.controls{display:grid;grid-template-columns:1fr 1fr;gap:15px}label{font-size:11px;text-transform:uppercase;letter-spacing:.7px;color:var(--muted);font-weight:600;display:block}select{display:block;width:100%;min-width:0;margin-top:6px;border:1px solid var(--line);border-radius:7px;padding:9px 8px;background:#f7f9f1;color:var(--ink);font:13px system-ui,sans-serif;min-height:40px}input[type=range]{accent-color:var(--green);width:100%;min-height:24px;cursor:pointer;margin:8px 0 0}output{float:right}.status{font-size:12px;line-height:1.5;color:var(--muted);margin:20px 0 12px;min-height:36px}.status.error{color:#943d27}.retry{margin-bottom:12px}.visual{background:#f0f4e8;border:1px solid var(--line);border-radius:9px;padding:14px 12px}.canvas-head,.canvas-foot{display:flex;justify-content:space-between;gap:10px;font:10px/1.4 ui-monospace,monospace;color:var(--muted)}#soundCanvas{width:100%;height:250px;display:block;margin:12px 0}.canvas-foot{font-size:9px}.canvas-foot>span:last-child{display:flex;align-items:center;gap:5px;white-space:nowrap}.canvas-foot i{width:40px;height:7px;background:linear-gradient(90deg,#dce6d0,#246644);display:inline-block}.transport{display:flex;flex-wrap:wrap;gap:8px;margin:18px 0}button{cursor:pointer;min-height:42px;border:1px solid var(--line);border-radius:7px;padding:10px 13px;background:#fafcf5;color:var(--ink);font:600 12px system-ui,sans-serif}button.primary{background:var(--green);color:#fff;border-color:var(--green)}button:disabled{opacity:.45;cursor:default}button:focus-visible,select:focus-visible,input:focus-visible,a:focus-visible{outline:3px solid #78a366;outline-offset:3px}.scrubber-label{display:flex;justify-content:space-between}.token-now{display:flex;align-items:center;gap:15px;padding:15px 0;border-bottom:1px solid var(--line);min-width:0}.token-now span{font:9px ui-monospace,monospace;color:var(--muted);flex-shrink:0}.token-now strong{white-space:pre-wrap;overflow-wrap:anywhere;font:22px/1.4 Georgia,serif;min-width:0}.prompt{font-size:12px;color:var(--muted)}.prompt strong{font-size:10px;text-transform:uppercase;letter-spacing:.6px}.readings{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:7px}.reading{background:#f0f4e8;padding:8px;border-radius:5px;font:10px/1.7 ui-monospace,monospace;overflow-wrap:anywhere}.reading strong{display:block;font-size:11px;color:var(--green)}.notes{grid-column:1/-1;display:grid;grid-template-columns:.85fr 1.15fr;gap:52px;border-top:1px solid var(--line);padding-top:46px;margin-top:12px}.notes p{color:var(--muted);font-size:14px}.mapping h3{font-size:14px;margin:0 0 7px}.mapping p{margin:0 0 24px}.mapping p.fine{font-size:12px}footer{border-top:1px solid var(--line);display:flex;flex-wrap:wrap;gap:20px;padding:24px 0 36px;color:var(--muted);font-size:11px}footer span{margin-right:auto}[hidden]{display:none!important}@media(max-width:780px){header,main,footer{width:calc(100% - 32px)}header{height:72px}main{grid-template-columns:1fr;gap:30px;padding-top:34px}h1{font-size:54px}.intro{max-width:580px}.lede{font-size:17px}.honest{max-width:none}.notes{grid-template-columns:1fr;gap:18px}.studio{padding:18px}.notes h2{max-width:350px}}@media(max-width:380px){header,main,footer{width:calc(100% - 24px)}header>a:last-child{font-size:11px}.brand{font-size:32px}.brand span{font-size:18px}h1{font-size:46px}.studio{padding:13px}.controls{gap:10px}select{font-size:11px;padding-left:5px}.transport{gap:6px}button{font-size:11px;padding:9px}#soundCanvas{height:230px}.canvas-head,.canvas-foot{font-size:8px}.readings{gap:5px}.reading{padding:6px;font-size:9px}.reading strong{font-size:10px}} + +.recorded-clips{max-width:1200px;margin:0 auto;padding:0 32px 44px;font-size:13px;line-height:2}.recorded-clips h2{font-size:25px;margin-bottom:4px}.recorded-clips p{color:#66756c}.recorded-clips a{color:#176b59;white-space:nowrap} +@media(max-width:600px){.recorded-clips{padding:0 20px 30px}} diff --git a/experiments/sound.html b/experiments/sound.html new file mode 100644 index 0000000000000000000000000000000000000000..c6e4ed8bf51ef00c33d36c1261d84752d5db9722 --- /dev/null +++ b/experiments/sound.html @@ -0,0 +1,40 @@ + + + + + + + + + What does a model sound like? | alphaBell + + + + + + +
α alphaBellAll experiments ↗
+
+

MODEL OBSERVATORY / EXPERIMENT 05

What does a
model sound like?

Turn measured expert routing and recurrent head activity into a small, strange piece of music. Same capture. Same mapping. Same notes.

This is a musical mapping of real numbers, not a recording of a model's voice or thoughts.

+
+
+ + + + +
+
Loading the measured capture…
+ +
Measured activityToken 0 / 0
Every cell is one measured unitlow high
+
+ +
OUTPUT TOKEN…
+

Recorded prompt
Explain why the sky is blue in two sentences.

+
+
+

A COMPOSITION WITH A RECEIPT

The numbers choose the notes.

These are the same real captures available in Model Observatory. No new inference runs here. Switching blocks changes which recorded measurements you hear.

Inspect the original topology ↗

Flint: six routed experts

Each selected expert becomes a sine tone. Its expert ID chooses a note in a repeating, three-octave C pentatonic scale. Its routing weight sets amplitude using 0.11 × √weight. The shared expert and dense blocks are excluded.

Qwen Open: recurrent head output

The six recurrent value heads with the largest output RMS at each token become triangle tones. Head ID chooses a note on the same scale, one octave higher. Amplitude is 0.08 × RMS / the largest head RMS across the chosen block's capture. All 48 heads remain visible.

How to read the comparison

Both use a fixed token clock selected above, not the original inference timing. Pitch repeats every 15 IDs, so different IDs can share a note. Colors show routing weight for Flint and normalized head RMS for Qwen. The two mappings have different units and are not a model-quality comparison.

Mono, 22,050 Hz. Identical settings produce identical samples. A short attack and release soften note boundaries. WAV export includes the selected volume and stays on your device.

+
+

Take a clip with you.

The same mapping, 350 ms per token and 35% volume.

Flint video ↓ · Flint WAV ↓ · Qwen Open video ↓ · Qwen Open WAV ↓
+ + + diff --git a/experiments/sound.js b/experiments/sound.js new file mode 100644 index 0000000000000000000000000000000000000000..40a18688921a7f46ca35969ecc0659e993943a0d --- /dev/null +++ b/experiments/sound.js @@ -0,0 +1,186 @@ +/* alphaBell: deterministic sonification of measured capture data. */ +(function (root) { + 'use strict'; + const PENTATONIC = [0, 2, 4, 7, 9]; + const SAMPLE_RATE = 22050; + function frequency(id, model) { + const note = id % 15; + const midi = (model === 'qwen' ? 60 : 48) + Math.floor(note / 5) * 12 + PENTATONIC[note % 5]; + return 440 * 2 ** ((midi - 69) / 12); + } + function buildScore(trace, layer, seconds = .35) { + if (!trace || !Array.isArray(trace.frames) || !trace.frames.length || trace.frames.length > 256) throw new Error('A complete capture is required.'); + const model = trace.model?.id || (trace.schema === 'flint.trace.v1' ? 'flint' : null); + if (!['flint', 'qwen'].includes(model) || !Number.isInteger(layer) || layer < 0 || !Number.isFinite(seconds) || seconds < .15 || seconds > 1 || trace.frames.length * seconds > 90) throw new Error('Unsupported score settings.'); + const frames = trace.frames.map(frame => { + const block = frame.layers?.[layer]; + if (!block) throw new Error('The chosen block is missing.'); + if (model === 'flint') { + if (!Array.isArray(block.experts) || block.experts.length !== 6 || !Array.isArray(block.weights) || block.weights.length !== 6 || new Set(block.experts).size !== 6 || !block.experts.every(id => Number.isInteger(id) && id >= 0 && id < 256) || !block.weights.every(v => Number.isFinite(v) && v >= 0 && v <= 1)) throw new Error('Choose a Flint block with six measured routed experts.'); + } else if (!Array.isArray(block.retention) || !Array.isArray(block.heads) || block.heads.length !== 48 || !block.heads.every(v => Number.isFinite(v) && v >= 0)) throw new Error('Choose a Qwen recurrent block with 48 measured value heads.'); + return block; + }); + let maximum = 0; + if (model === 'qwen') for (const block of frames) for (const value of block.heads) maximum = Math.max(maximum, value); + const events = frames.map((block, index) => { + let voices; + if (model === 'flint') voices = block.experts.map((id, rank) => ({ id, value: block.weights[rank], normalized: block.weights[rank], amplitude: .11 * Math.sqrt(block.weights[rank]), frequency: frequency(id, model), waveform: 'sine' })); + else voices = block.heads.map((value, id) => ({ id, value, normalized: maximum ? value / maximum : 0 })).sort((a, b) => b.value - a.value || a.id - b.id).slice(0, 6).map(voice => ({ ...voice, amplitude: .08 * voice.normalized, frequency: frequency(voice.id, model), waveform: 'triangle' })); + return { index, text: trace.frames[index].text, start: index * seconds, duration: seconds, voices }; + }); + return { model, layer, seconds, maximum, duration: events.length * seconds, sampleRate: SAMPLE_RATE, events }; + } + function renderPCM(score) { + if (!score || !Number.isFinite(score.duration) || score.duration <= 0 || score.duration > 90 || score.sampleRate !== SAMPLE_RATE || !Array.isArray(score.events)) throw new Error('Unsupported audio duration or sample rate.'); + const pcm = new Float32Array(Math.ceil(score.duration * SAMPLE_RATE)); + for (const event of score.events) { + const start = Math.round(event.start * SAMPLE_RATE), end = Math.min(pcm.length, Math.round((event.start + event.duration) * SAMPLE_RATE)); + const length = end - start, attack = Math.max(1, Math.min(Math.round(.012 * SAMPLE_RATE), length / 4)), release = Math.max(1, Math.min(Math.round(.05 * SAMPLE_RATE), length / 3)); + for (let i = 0; i < length; i++) { + const envelope = Math.min(1, i / attack, (length - 1 - i) / release); + let value = 0; + for (const voice of event.voices) { + const phase = 2 * Math.PI * voice.frequency * i / SAMPLE_RATE; + const wave = voice.waveform === 'triangle' ? 2 / Math.PI * Math.asin(Math.sin(phase)) : Math.sin(phase); + value += wave * voice.amplitude; + } + pcm[start + i] = Math.max(-1, Math.min(1, value * envelope)); + } + } + return pcm; + } + function encodeWAV(pcm, volume = .35) { + if (!(pcm instanceof Float32Array) || pcm.length > SAMPLE_RATE * 90 || !Number.isFinite(volume) || volume < 0 || volume > 1) throw new Error('Unsupported WAV samples or volume.'); + const buffer = new ArrayBuffer(44 + pcm.length * 2), view = new DataView(buffer); + const str = (at, value) => { for (let i = 0; i < value.length; i++) view.setUint8(at + i, value.charCodeAt(i)); }; + str(0, 'RIFF'); view.setUint32(4, buffer.byteLength - 8, true); str(8, 'WAVE'); str(12, 'fmt '); view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, 1, true); view.setUint32(24, SAMPLE_RATE, true); view.setUint32(28, SAMPLE_RATE * 2, true); view.setUint16(32, 2, true); view.setUint16(34, 16, true); str(36, 'data'); view.setUint32(40, pcm.length * 2, true); + for (let i = 0; i < pcm.length; i++) { const value = Math.max(-1, Math.min(1, pcm[i] * volume)); view.setInt16(44 + i * 2, Math.round(value * (value < 0 ? 32768 : 32767)), true); } + return buffer; + } + const API = { frequency, buildScore, renderPCM, encodeWAV, SAMPLE_RATE }; + if (typeof module !== 'undefined' && module.exports) module.exports = API; + root.ObservatorySound = API; + if (typeof document === 'undefined') return; + + const $ = id => document.getElementById(id); + const state = { trace: null, score: null, pcm: null, buffer: null, context: null, gain: null, source: null, playing: false, starting: false, animation: 0, offset: 0, started: 0, frame: 0, request: 0, cache: new Map(), models: null, manifest: null }; + const formatTime = seconds => `${Math.floor(seconds / 60)}:${String(Math.floor(seconds % 60)).padStart(2, '0')}`; + function status(text, error = false) { $('soundStatus').textContent = text; $('soundStatus').classList.toggle('error', error); } + function buttons(ready) { for (const id of ['soundPlay', 'soundStop', 'soundDownload', 'soundScrubber', 'soundLayer']) $(id).disabled = !ready; } + function position() { return state.playing ? Math.min(state.score.duration, state.offset + state.context.currentTime - state.started) : state.offset; } + function pause(reset = false) { + state.animation++; + if (state.playing) state.offset = position(); + state.playing = false; + if (state.source) { state.source.onended = null; state.source.stop(); state.source.disconnect(); state.source = null; } + if (reset) { state.offset = 0; state.frame = 0; } + $('soundPlay').textContent = '▶ Play capture'; + if (state.score) paint(); + } + function remake() { + pause(true); + if (!state.trace) return; + state.score = buildScore(state.trace, Number($('soundLayer').value), Number($('soundSpeed').value)); + state.pcm = renderPCM(state.score); state.buffer = null; + $('soundScrubber').max = state.score.events.length - 1; + buttons(true); + status(`Verified Q8_0 capture · ${state.score.events.length} measured output tokens · ${state.score.duration.toFixed(1)} seconds of mapped audio. Press Play to enable sound.`); + paint(); + } + async function play() { + if (!state.score || state.starting) return; + if (state.playing) { pause(); return; } + state.starting = true; + const score = state.score; + try { + const Audio = root.AudioContext || root.webkitAudioContext; + if (!Audio) throw new Error('Audio playback is unavailable in this browser. You can still download the WAV.'); + if (!state.context) { state.context = new Audio(); state.gain = state.context.createGain(); state.gain.connect(state.context.destination); } + await state.context.resume(); + if (state.score !== score || document.hidden) return; + if (state.offset >= state.score.duration) state.offset = 0; + if (!state.buffer) { state.buffer = state.context.createBuffer(1, state.pcm.length, SAMPLE_RATE); state.buffer.copyToChannel(state.pcm, 0); } + state.gain.gain.setValueAtTime(Number($('soundVolume').value) / 100, state.context.currentTime); + const source = state.context.createBufferSource(); source.buffer = state.buffer; source.connect(state.gain); state.source = source; + state.started = state.context.currentTime; state.playing = true; + source.onended = () => { if (state.source === source) { state.offset = state.score.duration; state.playing = false; state.source = null; source.disconnect(); $('soundPlay').textContent = '▶ Replay capture'; paint(); } }; + source.start(0, state.offset); $('soundPlay').textContent = 'Ⅱ Pause capture'; + const animation = ++state.animation; + const tick = () => { if (!state.playing || state.animation !== animation) return; paint(); requestAnimationFrame(tick); }; requestAnimationFrame(tick); + } catch (error) { pause(); status(error.message, true); } finally { state.starting = false; } + } + function paint() { + if (!state.score) return; + const time = position(), index = Math.min(state.score.events.length - 1, Math.floor(time / state.score.seconds)); state.frame = index; + const event = state.score.events[index], block = state.trace.frames[index].layers[state.score.layer], flint = state.score.model === 'flint'; + $('frameCount').textContent = `Token ${index + 1} / ${state.score.events.length}`; + $('playTime').textContent = `${formatTime(time)} / ${formatTime(state.score.duration)}`; + $('soundScrubber').value = index; + $('currentToken').textContent = event.text.replace(/\n/g, '↵') || '∅'; + $('activityTitle').textContent = `${flint ? 'EXPERT ROUTING' : 'RECURRENT HEAD OUTPUT'} · BLOCK ${state.score.layer}`; + $('gridLegend').textContent = flint ? '256 experts · 6 selected' : '48 heads · top 6 audible'; + const canvas = $('soundCanvas'), width = canvas.clientWidth, height = canvas.clientHeight, ratio = Math.min(2, root.devicePixelRatio || 1); + if (canvas.width !== Math.round(width * ratio) || canvas.height !== Math.round(height * ratio)) { canvas.width = Math.round(width * ratio); canvas.height = Math.round(height * ratio); } + const context = canvas.getContext('2d'); context.setTransform(ratio, 0, 0, ratio, 0, 0); context.clearRect(0, 0, width, height); + const columns = flint ? 16 : 8, rows = flint ? 16 : 6, gap = flint ? 3 : 6, cw = (width - gap * (columns - 1)) / columns, ch = (height - gap * (rows - 1)) / rows; + for (let id = 0; id < columns * rows; id++) { + const rank = flint ? block.experts.indexOf(id) : -1; + const value = flint ? (rank < 0 ? 0 : block.weights[rank]) : (state.score.maximum ? block.heads[id] / state.score.maximum : 0); + const shade = Math.pow(Math.min(1, value * (flint ? 3 : 1)), .6); + const a = [220,230,208], b = [36,102,68]; context.fillStyle = `rgb(${a.map((v, i) => Math.round(v + (b[i] - v) * shade)).join(',')})`; + const x = (id % columns) * (cw + gap), y = Math.floor(id / columns) * (ch + gap); context.fillRect(x, y, cw, ch); + if (!flint) { context.fillStyle = shade > .5 ? '#fffef8' : '#526849'; context.font = '10px ui-monospace,monospace'; context.textAlign = 'center'; context.textBaseline = 'middle'; context.fillText(String(id), x + cw / 2, y + ch / 2); } + if (event.voices.some(voice => voice.id === id)) { context.strokeStyle = '#1d4e35'; context.lineWidth = 1.5; context.strokeRect(x + .7, y + .7, cw - 1.4, ch - 1.4); } + } + if ($('voiceReadings').dataset.frame !== `${state.score.model}:${state.score.layer}:${index}`) { + $('voiceReadings').dataset.frame = `${state.score.model}:${state.score.layer}:${index}`; + $('voiceReadings').replaceChildren(...event.voices.map(voice => { const div = document.createElement('div'); div.className = 'reading'; const label = document.createElement('strong'); label.textContent = `${flint ? 'Expert' : 'Head'} ${voice.id} · ${voice.frequency.toFixed(0)} Hz`; div.append(label, document.createTextNode(flint ? `${(100 * voice.value).toFixed(2)}% weight` : `${voice.value.toPrecision(4)} RMS`)); return div; })); + } + } + async function sha256(bytes) { return Array.from(new Uint8Array(await crypto.subtle.digest('SHA-256', bytes)), b => b.toString(16).padStart(2, '0')).join(''); } + async function load() { + const request = ++state.request, model = $('soundModel').value; + pause(true); buttons(false); state.score = null; state.trace = null; state.pcm = null; state.buffer = null; $('soundRetry').hidden = true; + $('currentToken').textContent = '…'; $('voiceReadings').replaceChildren(); $('voiceReadings').dataset.frame = ''; + status(`Loading ${model === 'flint' ? 'Flint' : 'Qwen'} measured capture and verifying both SHA-256 checksums…`); + try { + if (!state.models || !state.manifest) { + const [modelsResponse, manifestResponse] = await Promise.all([fetch('../models.json'), fetch('../traces/index.json')]); + if (!modelsResponse.ok || !manifestResponse.ok) throw new Error('The capture registry could not be loaded.'); + [state.models, state.manifest] = await Promise.all([modelsResponse.json(), manifestResponse.json()]); + } + let trace = state.cache.get(model); + if (!trace) { + const row = state.manifest.find(item => item.model === model); + if (!row) throw new Error('No capture is available for this model.'); + const response = await fetch('../' + row.file); if (!response.ok) throw new Error('The capture download failed.'); + const compressed = await response.arrayBuffer(); if (await sha256(compressed) !== row.gzip_sha256) throw new Error('Compressed capture checksum did not match. Retry the download.'); + if (typeof DecompressionStream === 'undefined') throw new Error('This browser needs gzip stream support. Open in a current Chrome, Firefox or Safari.'); + const plain = await new Response(new Blob([compressed]).stream().pipeThrough(new DecompressionStream('gzip'))).arrayBuffer(); + if (await sha256(plain) !== row.sha256) throw new Error('Decoded capture checksum did not match. Retry the download.'); + trace = JSON.parse(new TextDecoder().decode(plain)); + root.validateObservatoryTrace(trace, state.models); + state.cache.set(model, trace); + } + if (request !== state.request) return; + state.trace = trace; + const config = state.models.find(item => item.id === model).architecture; + const eligible = config.layer_types.map((type, index) => ({ type, index })).filter(({ type, index }) => model === 'flint' ? index >= (config.first_k_dense_replace || 0) && !(config.mlp_only_layers || []).includes(index) : type === 'linear_attention'); + $('soundLayer').replaceChildren(...eligible.map(({ index }) => { const option = document.createElement('option'); option.value = index; option.textContent = `Block ${index} · ${model === 'flint' ? 'MoE' : 'recurrent'}`; return option; })); + $('soundLayer').value = model === 'flint' ? '12' : '0'; remake(); + } catch (error) { if (request === state.request) { status(error.message + ' Your browser has not started audio.', true); $('soundRetry').hidden = false; } } + } + $('soundModel').addEventListener('change', load); + $('soundRetry').addEventListener('click', load); + $('soundLayer').addEventListener('change', remake); + $('soundSpeed').addEventListener('change', remake); + $('soundPlay').addEventListener('click', play); + $('soundStop').addEventListener('click', () => pause(true)); + $('soundVolume').addEventListener('input', () => { const volume = Number($('soundVolume').value) / 100; $('volumeLabel').textContent = Math.round(volume * 100) + '%'; if (state.gain) state.gain.gain.setTargetAtTime(volume, state.context.currentTime, .02); }); + $('soundScrubber').addEventListener('input', () => { const wasPlaying = state.playing, frame = Number($('soundScrubber').value); pause(); state.offset = frame * state.score.seconds; paint(); if (wasPlaying) play(); }); + $('soundDownload').addEventListener('click', () => { if (!state.pcm) return; const blob = new Blob([encodeWAV(state.pcm, Number($('soundVolume').value) / 100)], { type: 'audio/wav' }); const url = URL.createObjectURL(blob), anchor = document.createElement('a'); anchor.href = url; anchor.download = `alphabell-${state.score.model}-block-${state.score.layer}-${Math.round(state.score.seconds * 1000)}ms.wav`; anchor.click(); setTimeout(() => URL.revokeObjectURL(url), 30000); }); + root.addEventListener('resize', paint); + document.addEventListener('visibilitychange', () => { if (document.hidden) pause(); }); + root.addEventListener('pagehide', () => { pause(); if (state.context) state.context.close(); }); + load(); +})(typeof globalThis !== 'undefined' ? globalThis : this); diff --git a/experiments/sound.test.cjs b/experiments/sound.test.cjs new file mode 100644 index 0000000000000000000000000000000000000000..fd4e0fcd17fff97b911a6d431746004b458e330a --- /dev/null +++ b/experiments/sound.test.cjs @@ -0,0 +1,60 @@ +'use strict'; +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const zlib = require('node:zlib'); +const path = require('node:path'); +const sound = require('./sound.js'); +const traces = Object.fromEntries(['flint','qwen'].map(model => [model, JSON.parse(zlib.gunzipSync(fs.readFileSync(path.join(__dirname, '../traces/' + model + '-sky-q8.json.gz'))))])); +test('Flint score preserves all selected expert IDs and exact measured weights', () => { + const score = sound.buildScore(traces.flint, 12, .2); + assert.equal(score.events.length, 59); + for (const event of score.events) { + const block = traces.flint.frames[event.index].layers[12]; + assert.deepEqual(event.voices.map(voice => voice.id), block.experts); + assert.deepEqual(event.voices.map(voice => voice.value), block.weights); + event.voices.forEach(voice => assert.equal(voice.amplitude, .11 * Math.sqrt(voice.value))); + } +}); +test('Qwen score uses the exact six largest recurrent head RMS values and a capture-wide scale', () => { + const score = sound.buildScore(traces.qwen, 0, .2); + const maximum = Math.max(...traces.qwen.frames.flatMap(frame => frame.layers[0].heads)); + assert.equal(score.maximum, maximum); + for (const event of score.events) { + const heads = traces.qwen.frames[event.index].layers[0].heads; + const expected = heads.map((value,id) => ({value,id})).sort((a,b) => b.value-a.value || a.id-b.id).slice(0,6); + assert.deepEqual(event.voices.map(({value,id}) => ({value,id})), expected); + event.voices.forEach(voice => { assert.equal(voice.normalized, voice.value / maximum); assert.equal(voice.waveform, 'triangle'); }); + } +}); +test('Synthesis is deterministic, non-silent, bounded, and aligned with token duration', () => { + for (const [model, layer] of [['flint',12], ['qwen',0]]) { + const score = sound.buildScore(traces[model], layer, .2), first = sound.renderPCM(score), second = sound.renderPCM(score); + assert.deepEqual(first, second); + assert.equal(first.length, Math.ceil(score.duration * sound.SAMPLE_RATE)); + let peak = 0; for (const value of first) { assert.ok(Number.isFinite(value)); peak = Math.max(peak, Math.abs(value)); } + assert.ok(peak > .001 && peak < 1); + assert.equal(first[0], 0); assert.equal(first.at(-1), 0); + } +}); +test('WAV encodes actual deterministic signed PCM with the chosen volume', () => { + const pcm = new Float32Array([0, .5, -.5, 1, -1]), buffer = sound.encodeWAV(pcm, .5), view = new DataView(buffer); + assert.equal(Buffer.from(buffer).toString('ascii', 0, 4), 'RIFF'); + assert.equal(Buffer.from(buffer).toString('ascii', 8, 12), 'WAVE'); + assert.equal(view.getUint32(24,true), 22050); assert.equal(view.getUint16(22,true), 1); assert.equal(view.getUint16(34,true),16); + assert.equal(view.getUint32(40,true), 10); assert.equal(buffer.byteLength,54); + assert.deepEqual(Array.from({length:5}, (_,i) => view.getInt16(44+i*2,true)), [0,8192,-8192,16384,-16384]); + assert.ok(new Int16Array(sound.encodeWAV(pcm,0),44).every(value => value === 0)); +}); +test('Reject dense Flint, full attention Qwen, excessive duration, and invalid volume', () => { + assert.throws(() => sound.buildScore(traces.flint,0)); + const full = traces.qwen.architecture.layer_types.indexOf('full_attention'); + assert.throws(() => sound.buildScore(traces.qwen,full)); + assert.throws(() => sound.buildScore(traces.flint,12,2)); + assert.throws(() => sound.renderPCM({duration:91,sampleRate:22050,events:[]})); + assert.throws(() => sound.encodeWAV(new Float32Array(1),-1)); +}); +test('ID pitch repeats every 15 IDs and Qwen is one octave above Flint', () => { + assert.equal(sound.frequency(2,'flint'), sound.frequency(17,'flint')); + assert.equal(sound.frequency(2,'qwen'), sound.frequency(2,'flint')*2); +}); diff --git a/index.html b/index.html index 6e1e20ed189ff6f5cd021bd915aad9394f8746ab..cc52679f1bf82557b9d63deef61a9413b81d4481 100644 --- a/index.html +++ b/index.html @@ -3,7 +3,7 @@ Flint & Qwen Model Observatory | alphaBell
αalphaBell/observatory
Loading capture
-
MODEL OBSERVATORY

Inside Flint & Qwen.

+
MODEL OBSERVATORY

Inside Flint & Qwen.

Try the five experiments ↗