library_name: pytorch
pipeline_tag: text-generation
tags:
- custom-code
- reasoning
- code
- long-context
- flexattention
- muon
license: other
CubicV11 150M Reasoning LM
Custom 156.9M-parameter causal language model trained with the
CubicV11 sparse long-context layer and the Cubic Muon optimizer. This revision
was exported at optimizer step 6,500 from
kabsis/AM-DeepSeek-R1-0528-Distilled/code.jsonl using assistant-only loss over
the teacher's <think> and <answer> output.
Architecture
- 14 layers, width 896, 14 query heads and 7 KV heads
- 4,096-token causal local attention window
- causal pooled global summaries every 256 tokens
- compressed cross-layer depth memory, rank 224
- RoPE base 500,000, SwiGLU and tied embeddings
- trained context length: 32,768 tokens
- ByteLevel BPE vocabulary: 16,384 tokens
This is custom PyTorch code, not a drop-in Transformers model. Use the included
cubic_v11_chat_gradio.py application:
python cubic_v11_chat_gradio.py
It downloads this repository, loads model.safetensors, uses incremental local,
global-summary and depth caches, supports a 32K total context window, reserves
up to 4,096 tokens for the answer and launches a public Gradio chat link.
training_checkpoint.pt contains model and optimizer state for continuing training.
cubic_v11_deepseek_distill_32k.py is the matching trainer.
Limitations and use
This is an experimental research checkpoint. It can produce incorrect, repetitive or unsafe code and reasoning. Generated code must be reviewed and tested before use. The source distillation dataset and its upstream model have their own terms; users are responsible for checking those terms. No claim is made that this checkpoint is production-ready.