Update README.md

README.md

@@ -1,3 +1,243 @@
 ---
 license: mit
+language:
+  - en
+library_name: cflow
+tags:
+  - moe
+  - cpu-inference
+  - rust
+  - custom-architecture
+  - pipeline-native
+  - avx-512
+datasets:
+  - roneneldan/TinyStories
+  - HuggingFaceFW/fineweb-edu
+pipeline_tag: text-generation
+model-index:
+  - name: arch2_4_combined
+    results:
+      - task:
+          type: text-generation
+        dataset:
+          name: TinyStories
+          type: roneneldan/TinyStories
+        metrics:
+          - name: Test Perplexity (114M, 10K steps)
+            type: perplexity
+            value: 6.50
+          - name: Top-1 Accuracy (114M, 10K steps)
+            type: accuracy
+            value: 56.8
+          - name: Val Perplexity (8.34B, 10K steps)
+            type: perplexity
+            value: 4.52
+          - name: Top-1 Accuracy (8.34B, 10K steps)
+            type: accuracy
+            value: 61.4
 ---
+
+# arch2_4_combined — Pipeline-Native MoE for CPU Inference
+
+A custom decoder-only transformer with delayed dense FFN + delayed MoE experts,
+designed so its inter-layer dependency graph permits vertical pipelining on CPU.
+Part of the **cflow** project — a CPU-first streaming inference engine written in
+Rust.
+
+## Key Results
+
+| Metric | Value |
+|---|---|
+| CPU decode throughput (8.34B, Q4, 32 threads) | **5.94 tok/s** |
+| Effective memory bandwidth | 61 GB/s (30% of 204.8 GB/s peak) |
+| Bandwidth reduction from pipelining | **2.00x** (16.50 → 4.50 MB/token) |
+| Test perplexity (114M, TinyStories, 10K steps) | 6.50 |
+| Test perplexity (8.34B, TinyStories, 10K steps) | 4.52 |
+
+### CPU Decode Benchmark (AWS r6i.8xlarge, Ice Lake Xeon, 256 GB DDR4)
+
+| Engine | Model | Quant | tok/s |
+|---|---|---|---|
+| **cflow** | arch2_4_8k_16l (8.34B MoE, ~3–4B active) | Q4 | **5.94** |
+| Ollama (llama.cpp) | Qwen2.5-32B (32B dense) | Q4 GGUF | 4.75 |
+| vLLM CPU | Qwen2.5-32B-Instruct (32B dense) | GPTQ-Int4 | 1.65 |
+
+> **Note:** cflow and the baselines run different models — cflow's 8.34B MoE has
+> ~3–4B active params per token vs 32B dense. The cflow number shows what a
+> co-designed architecture + streaming runtime achieves.
+
+## Model Description
+
+**arch2_4_combined** is a pre-norm decoder-only transformer with a parallel dense
+FFN + sparse MoE block per layer, using delayed residual injection:
+
+- The **dense FFN** reads from a delayed residual (1 layer behind)
+- The **MoE experts** are routed on the current residual but injected 2 layers later
+- This creates a dependency DAG where dense and expert weight reads for layer N
+  can overlap with compute for layer N-1, reducing critical-path memory bandwidth
+
+The architecture was selected from a screen of 5 pipeline-native candidates. It
+is the only design that achieves a measured bandwidth reduction (2.00x) while
+maintaining competitive perplexity.
+
+### Architecture Details
+
+| Parameter | 114M (screening) | 8.34B (scaled) |
+|---|---|---|
+| Hidden dim | 512 | 8,192 |
+| Layers | 6 | 16 |
+| Attention heads | 8 | 128 |
+| Head dim | 64 | 64 |
+| Dense FFN hidden | 2,048 | 32,768 |
+| Expert FFN hidden | 512 | 4,096 |
+| Experts / top-k | 8 / 2 | 8 / 2 |
+| Dense delay | 1 | 1 |
+| Expert delay | 2 | 2 |
+| Vocab | 50,257 (GPT-2 BPE) | 50,257 (GPT-2 BPE) |
+| Max seq len | 512 | 2,048 |
+
+### Per-Layer Forward Pass
+
+```
+attn_out = attention(attn_norm(x))
+x = x + attn_out                              # residual connection
+x = x + dense_ffn(ffn_norm(delayed_x))        # dense reads DELAYED residual
+if queued_expert: x = x + queued_expert        # inject expert from 2 layers ago
+expert_out = moe(ffn_norm(x))                  # router sees CURRENT residual
+# expert_out queued for injection at layer + expert_delay
+```
+
+### Components
+
+- **Attention:** Multi-head (not GQA), Q/K/V/O projections (no bias), standard
+  RoPE (base=10000, half-interleave), causal masking, KV cache
+- **Dense FFN:** GeGLU — `down(gelu(gate(x)) * up(x))`
+- **MoE:** Linear router → top-k selection → softmax over selected → per-expert
+  GeGLU FFN → weighted sum. No auxiliary/load-balancing loss.
+- **Normalization:** RMSNorm (eps=1e-6) at attn input, FFN input, and pre-lm_head
+- **Combine style:** `DelayedSum` — dense and router share `ffn_norm` but read
+  different residual snapshots
+
+## Training
+
+### 114M Screening (5 architectures)
+
+| | |
+|---|---|
+| Dataset | TinyStories (431M train tokens, 24M test tokens) |
+| Tokenizer | GPT-2 BPE (50,257 vocab) |
+| Sequence length | 512 |
+| Optimizer | AdamW (betas=0.9/0.95, eps=1e-8, weight_decay=0.1) |
+| Learning rate | 3e-4 with linear warmup (200 steps) + cosine decay to 1e-5 |
+| Gradient clipping | Global norm 1.0 |
+| Batch size | 8 |
+| Steps | 10,000 |
+| Precision | float32 |
+| Hardware | RTX 3060 12 GB |
+
+### 8.34B Scale-Up
+
+| | |
+|---|---|
+| Dataset | TinyStories (same splits) |
+| Optimizer | 8-bit AdamW (bitsandbytes) |
+| Learning rate | 1e-4 with linear warmup (500 steps) + cosine decay to 1e-6 |
+| Batch size | 4 per GPU (global 32) |
+| Steps | 10,000 |
+| Precision | bf16 |
+| Parallelism | FSDP (FULL_SHARD / ZeRO-3) |
+| Gradient checkpointing | Per `DelayedMoELayer`, non-reentrant |
+| Hardware | 8x A100 SXM4 80 GB (Lambda Cloud) |
+
+### Architecture Comparison (114M, TinyStories, 10K steps)
+
+| Architecture | dense_delay | expert_delay | Test PPL | Top-1 Acc | BW Reduction |
+|---|---|---|---|---|---|
+| arch1_decoupled_streams | 0 | 0 | 7.21 | 54.9% | 1.00x |
+| **arch2_4_combined** | **1** | **2** | **6.50** | **56.8%** | **2.00x** |
+| arch3_pipeline_registers | 0 | 0 | 7.24 | 55.1% | 1.00x |
+| arch4_async_experts | 0 | 2 | **6.26** | **57.6%** | 1.00x |
+| arch5_fixed_point | 0 | 0 | 6.77 | 56.2% | 1.00x |
+
+**Key insight:** Dense delay is the bandwidth knob; expert delay is the quality
+knob. arch4_async_experts gets the best perplexity by routing off pre-dense
+activations (cleaner router signal) but sacrifices the bandwidth win that
+arch2_4 achieves by also delaying the dense read.
+
+## Inference with cflow
+
+cflow is a Rust inference engine that reads `.cflow` (per-layer streaming) or
+`.vflow` (vertical pipeline) weight files. Weights are stored as pre-tiled Q4
+(128x256 tiles, ~18 KB each, sized to fit L2 cache).
+
+```bash
+# Build
+cargo build --release --bin cflow-run
+
+# Convert safetensors → .cflow
+cargo run --release --bin cflow-convert -- \
+  --input checkpoint.safetensors \
+  --output model.cflow \
+  --model arch2_4
+
+# Run inference
+CFLOW_THREADS=32 ./target/release/cflow-run \
+  model.cflow 32 \
+  --prompt "Once upon a time" \
+  --tokenizer tokenizer.json \
+  --temperature 0.8
+```
+
+### SIMD Support
+
+The runtime auto-detects and dispatches to the best available instruction set:
+
+| ISA | Kernel | Notes |
+|---|---|---|
+| AVX-512 + VNNI | Q4×Q8 `vpdpbusd` | Best path (Ice Lake+) |
+| AVX-512F | Q4×f32 FMA | Skylake-X+ |
+| AVX2 + FMA | Q4×f32 FMA | Haswell+ |
+| AVX + SSE4.1 | Q4×f32 | Sandy Bridge+ |
+| Scalar | Q4×f32 | Fallback |
+
+## Limitations
+
+- **Not a general-purpose LLM.** Trained on TinyStories / FineWeb-Edu subsets at
+  10K steps — this is an architecture and runtime research artifact, not a
+  production language model.
+- **Custom architecture.** Cannot be loaded in Hugging Face Transformers, vLLM,
+  or llama.cpp without adaptation. Requires the cflow Rust runtime or the
+  PyTorch reference in `pipeline_native/`.
+- **CPU-only.** The runtime targets x86-64 CPUs with AVX2 or AVX-512. No GPU
+  backend.
+- **Single-token decode optimized.** Batch/prefill throughput is not the focus.
+
+## Thesis Scorecard
+
+The cflow project tests 8 claims about CPU inference optimization:
+
+| # | Claim | Result |
+|---|---|---|
+| 1 | Conditional expert reading (top-k only) | **Proven** |
+| 2 | Tile-streaming L1/L2 cache locality | **Proven** (7.29x fewer L1-d misses, PMU-measured) |
+| 3 | AVX2/AVX-512 Q4 SIMD kernels | **Proven** |
+| 4 | Fused QKV and gate+up projections | **Proven** |
+| 5 | Compute-order file layout | **Proven** |
+| 6 | Software prefetch (`_mm_prefetch`) | **Disproven** (no benefit; slightly harmful) |
+| 7 | Vertical pipeline via delayed dependencies | **Validated** (2.00x bandwidth reduction) |
+| 8 | Stage-major disk layout readahead | **Disproven** (no isolated benefit) |
+
+## Citation
+
+```bibtex
+@software{poperszky2026cflow,
+  author = {Poperszky, Tom},
+  title = {cflow: CPU-First Streaming Inference for Pipeline-Native Transformers},
+  year = {2026},
+  url = {https://github.com/tpoperszky/cflow}
+}
+```
+
+## License
+
+MIT