README.md

---
license: mit
language:
  - en
library_name: cflow
tags:
  - moe
  - cpu-inference
  - rust
  - custom-architecture
  - pipeline-native
  - avx-512
datasets:
  - roneneldan/TinyStories
  - HuggingFaceFW/fineweb-edu
pipeline_tag: text-generation
model-index:
  - name: arch2_4_combined
    results:
      - task:
          type: text-generation
        dataset:
          name: TinyStories
          type: roneneldan/TinyStories
        metrics:
          - name: Test Perplexity (114M, 10K steps)
            type: perplexity
            value: 6.50
          - name: Top-1 Accuracy (114M, 10K steps)
            type: accuracy
            value: 56.8
          - name: Val Perplexity (8.34B / 4-layer, 10K steps)
            type: perplexity
            value: 4.52
          - name: Top-1 Accuracy (8.34B / 4-layer, 10K steps)
            type: accuracy
            value: 61.4
---

# arch2_4_combined — Pipeline-Native MoE for CPU Inference

A custom decoder-only transformer with delayed dense FFN + delayed MoE experts,
designed so its inter-layer dependency graph permits vertical pipelining on CPU.
Part of the **cflow** project — a CPU-first streaming inference engine written in
Rust.

> **Hosted weights:** this repository hosts `model.cflow` (17.39 GB) — the
> **arch2_4_8k_16l** model: 16 layers, hidden 8192, **~31B parameters**
> (top-2-of-8 MoE, ~20B active/token), Q4. This is the model benchmarked at
> 5.94 tok/s below. The **8.34B** figures in this card refer to a *smaller
> 4-layer scale point* (`arch2_4_8k_4l`) used for quality and cache-locality
> validation (val ppl 4.52); that checkpoint is not hosted here.

## Key Results

| Metric | Value |
|---|---|
| CPU decode throughput (~31B / 16-layer, Q4, 32 threads) | **5.94 tok/s** |
| Effective memory bandwidth | 61 GB/s (30% of 204.8 GB/s peak) |
| Bandwidth reduction from pipelining | **2.00x** (9.00 → 4.50 MB/token) |
| Test perplexity (114M, TinyStories, 10K steps) | 6.50 |
| Val perplexity (8.34B / 4-layer, TinyStories, 10K steps) | 4.52 |

### CPU Decode Benchmark (AWS r6i.8xlarge, Ice Lake Xeon, 256 GB DDR4)

| Engine | Model | Quant | tok/s |
|---|---|---|---|
| **cflow** | arch2_4_8k_16l (~31B MoE, ~20B active) | Q4 | **5.94** |
| Ollama (llama.cpp) | Qwen2.5-32B (32B dense) | Q4 GGUF | 4.75 |
| vLLM CPU | Qwen2.5-32B-Instruct (32B dense) | GPTQ-Int4 | 1.65 |

> **Note:** cflow and the baselines run different models — cflow's ~31B MoE has
> ~20B active params per token vs 32B dense. The total parameter counts are
> comparable (31B vs 32B), but the architectures and training differ, so the
> cflow number shows what a co-designed architecture + streaming runtime achieves,
> not a quality-matched result.

## Model Description

**arch2_4_combined** is a pre-norm decoder-only transformer with a parallel dense
FFN + sparse MoE block per layer, using delayed residual injection:

- The **dense FFN** reads from a delayed residual (1 layer behind)
- The **MoE experts** are routed on the current residual but injected 2 layers later
- This creates a dependency DAG where dense and expert weight reads for layer N
  can overlap with compute for layer N-1, reducing critical-path memory bandwidth

The architecture was selected from a screen of 5 pipeline-native candidates. It
is the only design that achieves a measured bandwidth reduction (2.00x) while
maintaining competitive perplexity.

### Architecture Details

| Parameter | 114M (screening) | ~31B (16-layer, hosted) |
|---|---|---|
| Hidden dim | 512 | 8,192 |
| Layers | 6 | 16 |
| Attention heads | 8 | 128 |
| Head dim | 64 | 64 |
| Dense FFN hidden | 2,048 | 32,768 |
| Expert FFN hidden | 512 | 4,096 |
| Experts / top-k | 8 / 2 | 8 / 2 |
| Dense delay | 1 | 1 |
| Expert delay | 2 | 2 |
| Vocab | 50,257 (GPT-2 BPE) | 50,257 (GPT-2 BPE) |
| Max seq len | 512 | 2,048 |

### Per-Layer Forward Pass

```
attn_out = attention(attn_norm(x))
x = x + attn_out                              # residual connection
x = x + dense_ffn(ffn_norm(delayed_x))        # dense reads DELAYED residual
if queued_expert: x = x + queued_expert        # inject expert from 2 layers ago
expert_out = moe(ffn_norm(x))                  # router sees CURRENT residual
# expert_out queued for injection at layer + expert_delay
```

### Components

- **Attention:** Multi-head (not GQA), Q/K/V/O projections (no bias), standard
  RoPE (base=10000, half-interleave), causal masking, KV cache
- **Dense FFN:** GeGLU — `down(gelu(gate(x)) * up(x))`
- **MoE:** Linear router → top-k selection → softmax over selected → per-expert
  GeGLU FFN → weighted sum. No auxiliary/load-balancing loss.
- **Normalization:** RMSNorm (eps=1e-6) at attn input, FFN input, and pre-lm_head
- **Combine style:** `DelayedSum` — dense and router share `ffn_norm` but read
  different residual snapshots

## Training

### 114M Screening (5 architectures)

| | |
|---|---|
| Dataset | TinyStories (431M train tokens, 24M test tokens) |
| Tokenizer | GPT-2 BPE (50,257 vocab) |
| Sequence length | 512 |
| Optimizer | AdamW (betas=0.9/0.95, eps=1e-8, weight_decay=0.1) |
| Learning rate | 3e-4 with linear warmup (200 steps) + cosine decay to 1e-5 |
| Gradient clipping | Global norm 1.0 |
| Batch size | 8 |
| Steps | 10,000 |
| Precision | float32 |
| Hardware | RTX 3060 12 GB |

### 8.34B Scale-Up (4-layer — quality & cache validation)

This is the smaller scale point: `arch2_4_8k_4l`, 4 layers, 8.34B params. It
provides the quality numbers (val ppl 4.52, top-1 61.4%) and the PMU cache-locality
result. The hosted decode-benchmark model (`arch2_4_8k_16l`, ~31B) shares this
per-layer geometry but has 16 layers.

| | |
|---|---|
| Dataset | TinyStories (same splits) |
| Optimizer | 8-bit AdamW (bitsandbytes) |
| Learning rate | 1e-4 with linear warmup (500 steps) + cosine decay to 1e-6 |
| Batch size | 4 per GPU (global 32) |
| Steps | 10,000 |
| Precision | bf16 |
| Parallelism | FSDP (FULL_SHARD / ZeRO-3) |
| Gradient checkpointing | Per `DelayedMoELayer`, non-reentrant |
| Hardware | 8x A100 SXM4 80 GB (Lambda Cloud) |

### Architecture Comparison (114M, TinyStories, 10K steps)

| Architecture | dense_delay | expert_delay | Test PPL | Top-1 Acc | BW Reduction |
|---|---|---|---|---|---|
| arch1_decoupled_streams | 0 | 0 | 7.21 | 54.9% | 1.00x |
| **arch2_4_combined** | **1** | **2** | **6.50** | **56.8%** | **2.00x** |
| arch3_pipeline_registers | 0 | 0 | 7.24 | 55.1% | 1.00x |
| arch4_async_experts | 0 | 2 | **6.26** | **57.6%** | 1.00x |
| arch5_fixed_point | 0 | 0 | 6.77 | 56.2% | 1.00x |

**Key insight:** Dense delay is the bandwidth knob; expert delay is the quality
knob. arch4_async_experts gets the best perplexity by routing off pre-dense
activations (cleaner router signal) but sacrifices the bandwidth win that
arch2_4 achieves by also delaying the dense read.

## Inference with cflow

cflow is a Rust inference engine that reads `.cflow` (per-layer streaming) or
`.vflow` (vertical pipeline) weight files. Weights are stored as pre-tiled Q4
(128x256 tiles, ~18 KB each, sized to fit L2 cache).

```bash
# Build
cargo build --release --bin cflow-run

# Convert safetensors → .cflow
cargo run --release --bin cflow-convert -- \
  --input checkpoint.safetensors \
  --output model.cflow \
  --model arch2_4

# Run inference
CFLOW_THREADS=32 ./target/release/cflow-run \
  model.cflow 32 \
  --prompt "Once upon a time" \
  --tokenizer tokenizer.json \
  --temperature 0.8
```

### SIMD Support

The runtime auto-detects and dispatches to the best available instruction set:

| ISA | Kernel | Notes |
|---|---|---|
| AVX-512 + VNNI | Q4×Q8 `vpdpbusd` | Best path (Ice Lake+) |
| AVX-512F | Q4×f32 FMA | Skylake-X+ |
| AVX2 + FMA | Q4×f32 FMA | Haswell+ |
| AVX + SSE4.1 | Q4×f32 | Sandy Bridge+ |
| Scalar | Q4×f32 | Fallback |

## Limitations

- **Not a general-purpose LLM.** Trained on TinyStories / FineWeb-Edu subsets at
  10K steps — this is an architecture and runtime research artifact, not a
  production language model.
- **Custom architecture.** Cannot be loaded in Hugging Face Transformers, vLLM,
  or llama.cpp without adaptation. Requires the cflow Rust runtime or the
  PyTorch reference in `pipeline_native/`.
- **CPU-only.** The runtime targets x86-64 CPUs with AVX2 or AVX-512. No GPU
  backend.
- **Single-token decode optimized.** Batch/prefill throughput is not the focus.

## Thesis Scorecard

The cflow project tests 8 claims about CPU inference optimization:

| # | Claim | Result |
|---|---|---|
| 1 | Conditional expert reading (top-k only) | **Proven** |
| 2 | Tile-streaming L1/L2 cache locality | **Proven** (7.29x fewer L1-d misses, PMU-measured) |
| 3 | AVX2/AVX-512 Q4 SIMD kernels | **Proven** |
| 4 | Fused QKV and gate+up projections | **Proven** |
| 5 | Compute-order file layout | **Proven** |
| 6 | Software prefetch (`_mm_prefetch`) | **Disproven** (no benefit; slightly harmful) |
| 7 | Vertical pipeline via delayed dependencies | **Validated** (2.00x bandwidth reduction) |
| 8 | Stage-major disk layout readahead | **Disproven** (no isolated benefit) |

## Citation

```bibtex
@software{poperszky2026cflow,
  author = {Poperszky, Tom},
  title = {cflow: CPU-First Streaming Inference for Pipeline-Native Transformers},
  year = {2026}
}
```

## License

MIT