FWKV
Collection
5 items • Updated
FWKV
FWKV (Feed-forward Weighted Key Value, or Floored Weighted Key Value) is a novel efficient language model architecture proposed on May 13, 2026 by Me.
Design Philosophy
FWKV is built on two simple ideas:
- FFNN as the primary compute block — fast, parallelizable, hardware-friendly
- floor(W·K·V) as the sole context mechanism — a multiplicative recurrent state with a learned decay W floored above zero, so the model can never fully forget
Sample
Inference speed (CUDA -- T4)
Training
Factorised tied weights, chunked CE, bf16 training. Best val perplexity: 143.62
- Downloads last month
- 83