| --- |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - custom-code |
| - reasoning |
| - code |
| - long-context |
| - flexattention |
| - muon |
| license: other |
| --- |
| |
| # CubicV11 150M Reasoning LM |
|
|
| Custom 156.9M-parameter causal language model trained with the |
| CubicV11 sparse long-context layer and the Cubic Muon optimizer. This revision |
| was exported at optimizer step **6,500** from |
| `kabsis/AM-DeepSeek-R1-0528-Distilled/code.jsonl` using assistant-only loss over |
| the teacher's `<think>` and `<answer>` output. |
|
|
| ## Architecture |
|
|
| - 14 layers, width 896, 14 query heads and 7 KV heads |
| - 4,096-token causal local attention window |
| - causal pooled global summaries every 256 tokens |
| - compressed cross-layer depth memory, rank 224 |
| - RoPE base 500,000, SwiGLU and tied embeddings |
| - trained context length: 32,768 tokens |
| - ByteLevel BPE vocabulary: 16,384 tokens |
|
|
| This is custom PyTorch code, not a drop-in Transformers model. Use the included |
| `cubic_v11_chat_gradio.py` application: |
|
|
| ```bash |
| python cubic_v11_chat_gradio.py |
| ``` |
|
|
| It downloads this repository, loads `model.safetensors`, uses incremental local, |
| global-summary and depth caches, supports a 32K total context window, reserves |
| up to 4,096 tokens for the answer and launches a public Gradio chat link. |
|
|
| `training_checkpoint.pt` contains model and optimizer state for continuing training. |
| `cubic_v11_deepseek_distill_32k.py` is the matching trainer. |
|
|
| ## Limitations and use |
|
|
| This is an experimental research checkpoint. It can produce incorrect, |
| repetitive or unsafe code and reasoning. Generated code must be reviewed and |
| tested before use. The source distillation dataset and its upstream model have |
| their own terms; users are responsible for checking those terms. No claim is |
| made that this checkpoint is production-ready. |
|
|
