🌟 Why BareTorch? (Breaking the Edge & Cloud Kernel Wall)

Modern sub-quadratic architectures (such as Mamba-2, RWKV, and Gated DeltaNet) promise linear scaling, but they suffer from a critical deployment flaw: custom kernel lock-in. They rely on specialized C++/CUDA/Triton extensions (causal-conv1d, flash-linear-attention) that frequently fail to compile on consumer setups and collapse into slow sequential fallbacks on systolic arrays and edge hardware.

Model Rampage solves this through BareTorch and our CS-LRAD (Chunked Selective Low-Rank Associative Delta) engine:

1. 100% Pure-GEMM Formulation: Replaces hardware-locked kernels and 1D convolutions with standard matrix multiplications (GEMMs) and an exact $O(1)$ low-rank recurrent state update for autoregressive decoding.
2. True Universal Portability: Because every AI chip natively accelerates matrix multiplication, BareTorch runs at peak hardware speed out-of-the-box across Apple Silicon (MLX), Google Cloud TPUs (OpenXLA), NVIDIA/AMD/Intel GPUs, WebGPU, and Mobile NPUs—using nothing but standard torch and transformers.
3. Surgical Up-Distillation & <think> Alignment: Our multi-stage Layerwise Up-Distillation Engine converts pretrained weights into 3:1 CS-LRAD hybrids in hours, followed by Top-64 Skew-KD from larger teachers and 4-Pillar <think> Dr. GRPO reinforcement learning—producing models that outperform their own native teachers in both intelligence and hardware speed.

🚀 Model Portfolio

🏆 Spotlight: BareTorch-Qwen3.5-4B-Hybrid & 2B-Hybrid Benchmarks

Surgically distilled from Qwen/Qwen3.5 and aligned via Stage 4B/4C Top-64 Skew-KD (4B $\rightarrow$ 9B and 2B $\rightarrow$ 4B) + 1,500-step <think> Dr. GRPO, our BareTorch-Qwen3.5-4B-Hybrid and BareTorch-Qwen3.5-2B-Hybrid models replace 75% of the backbone's token mixers with pure-PyTorch CS-LRAD layers while surpassing their native Qwen3.5 teachers across both general knowledge and generative reasoning/coding:

1. Executive Benchmark Summary (4B-Hybrid & 2B-Hybrid vs. Native Qwen3.5 Teachers)

2. Hardware Speed & Memory Efficiency (4B, 2B & 0.8B Hybrids on NVIDIA RTX 4090)

Running 100% in standard PyTorch (torch.compile) with zero custom C++/CUDA/Triton packages, BareTorch beats Qwen3.5's native hand-written kernels across every scale (4B, 2B, and 0.8B) in prefill throughput, decoding speed, KV-cache size, and 32K long-context VRAM:

(Full 512 $\rightarrow$ 32,768 context hardware tables for each model size are available directly on the 4B Model Card and 2B Model Card).

🌐 Cross-Platform Edge & Cloud Benchmarks (BareTorch-2.5B-Base From-Scratch Study)

To prove that pure-GEMM sub-quadratic architectures scale from scratch and transfer effortlessly across non-NVIDIA silicon (where Triton/CUDA kernels do not exist), we also trained BareTorch-2.5B-Base on 75B tokens and profiled it across Google Cloud TPUs, Apple Silicon (MLX 4-bit), and NVIDIA CUDA:

• Zero-Shot Sample Efficiency: Trained on 75B tokens, BareTorch-2.5B-Base achieves a 57.87% 7-task zero-shot average—beating Pythia-2.8B (54.94%) and Mamba-1.4B (56.10%) trained on 300B tokens (4x more compute), while outperforming Qwen3.5-2B-Base on PIQA (74.81%) and OpenBookQA (39.40%).
• 13.37x Faster 32K Prefill & 2.33x Faster Decoding on Google Cloud TPU (v5e-1 BF16): Because Qwen3.5-2B lacks systolic-array TPU kernels for GatedDeltaNet, it falls back to sequential execution at 32K context (17.51s TTFT / 1,871 tok/s). BareTorch's pure-GEMM CS-LRAD runs natively on the TPU MXU, completing 32K prefill in 1.31s (25,018 tok/s) and decoding at 43.38–49.02 tok/s (vs. 21.07 tok/s).
• 34% Faster Edge Prefill & 30% Lower 32K RAM on Apple Silicon (M1 MacBook Pro MLX 4-Bit): Processes 32K tokens in 31.9s (1,028 tok/s) vs. 42.6s (770 tok/s) for Qwen3.5-2B, sustains 99.3 tok/s short-context and 75.5 tok/s 32K decoding, and cuts 32K unified RAM by 1.48 GB (3.52 GB vs. 5.00 GB).

(Full 512 $\rightarrow$ 32,768 tables across TPU v5e, Apple M1 MLX, and CUDA are available on the BareTorch-2.5B-Base model card).

🔗 Resources & Citation

• Layerwise Distillation & GRPO Pipeline Repository: github.com/martin-kbcc/BareTorch_Layerwise_Distill
• Core BareTorch Engine (PyTorch & MLX): github.com/martin-kbcc/baretorch
• Technical Whitepaper: Read Technical Report (PDF)

@article{kovacevic2026baretorch,
  title={BareTorch: Challenging State-of-The-Art Sequence Mixing Topologies via Kernel-Free, Pure GEMM-Compliant Architectures},
  author={Kovacevic Buvinic, Martin Ignacio},
  journal={BareTorch Framework Laboratory Technical Report},
  year={2026},
  url={[https://github.com/martin-kbcc/baretorch](https://github.com/martin-kbcc/baretorch)}
}