license: apache-2.0
language:
- code
tags:
- code-generation
- multi-scale-transformer
- cpu-optimized
- cubic
- pytorch
- llama
- gguf
- byte-level
pipeline_tag: text-generation
library_name: transformers
datasets:
- bigcode/starcoderdata
- theblackcat102/evol-codealpaca-v1
widget:
- text: |-
# Find GCD of two numbers
def gcd(a, b):
model-index:
- name: AXL-Reasoning-70M
results:
- task:
type: text-generation
metrics:
- name: Perplexity (byte-level)
type: perplexity
value: 1.93
AXL-Reasoning-70M
CoT reasoning (SGD). 70M params. PPL 1.93 Part of the AXL model family by CubicLabs.
Model Details
| Property | Value |
|---|---|
| Developed by | CubicLabs |
| Architecture | Multi-Scale Transformer |
| Parameters | 70M |
| Optimizer | SGD |
| Attention | SDPA |
| Vocab Size | 258 (byte-level) |
| Context Window | 512 bytes |
| d_model | 512 |
| Attention Heads | 4 |
| Layers per Scale | 5 |
| Downsample Factors | [1, 2, 4] |
| License | Apache 2.0 |
Sources
Uses
Direct Use
Chain-of-thought reasoning (SGD baseline).
import torch
from multiscale_transformer.model.model import MultiScaleTransformer
from multiscale_transformer.training.tokenizer import ByteTokenizer
ckpt = torch.load("axl_reasoning_70m.pt", map_location="cpu")
model = MultiScaleTransformer(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
tokenizer = ByteTokenizer()
ids = torch.tensor([tokenizer.encode("def hello():")], dtype=torch.long)
with torch.no_grad():
out = model.generate(ids, max_new_tokens=50, temperature=0.8)
print(tokenizer.decode(out[0].tolist()))
Out-of-Scope Use
Not for production code generation. Use the Lion version for better results. For integration with tools like Continue.dev, LlamaIndex, or LangChain, use the Python API server which provides OpenAI-compatible endpoints.
Bias, Risks, and Limitations
Byte-level perplexity is not comparable to BPE-level perplexity. SGD baseline. Use AXL-Reasoning-Lion for better results. Note: GGUF files for Ollama use a simplified single-stack encoder. For full AXL quality, use the Python API server.
Recommendations
- Use for prototyping and experimentation, not production code generation.
- Byte-level perplexity (258 vocab) is not comparable to BPE-level perplexity (32K vocab).
- For better results, use the Lion-optimized version if available.
Training Details
Training Data
SGD 60 min. 1478 steps. Lion version achieves PPL 1.93 in 20 min.
Preprocessing
Byte-level tokenization with vocabulary size 258 (256 bytes + BOS + EOS). No vocabulary training required.
Speeds, Sizes, Times
| Metric | Value |
|---|---|
| Training Steps | 1478 |
| Training Time | 60 min |
| Final Loss | 0.6384 |
Evaluation
Metrics
Perplexity on held-out Python code using byte-level tokenization.
Results
| Metric | Value |
|---|---|
| Perplexity (byte-level) | 1.93 |
| Final Loss | 0.6384 |
| Training Steps | 1478 |
| Training Time | 60 min |
Summary: SGD reasoning baseline. Lion version is significantly better.
Environmental Impact
| Property | Value |
|---|---|
| Hardware | AMD Ryzen 5 5600G |
| Hours Used | 1.001 |
| Carbon Emitted | 0.0420 kg CO2 |
| Cloud Provider | None (local CPU) |
Technical Specifications
Model Architecture
Multi-Scale Transformer with three parallel encoder stacks at resolution scales 1x, 2x, and 4x. Cross-scale attention connects all scale pairs. Adaptive gating fusion. SwiGLU feed-forward. RoPE positional encoding.
Compute Infrastructure
| Property | Value |
|---|---|
| Hardware | AMD Ryzen 5 5600G (6 cores, 12 threads) |
| RAM | 16 GB |
| GPU | None (CPU-only) |
Citation
@misc{axl_2026,
title={AXL: AXL-Reasoning-70M - Multi-Scale Transformer for CPU Code Generation},
author={Cubic},
year={2026},
url={[https://huggingface.co/CubicLabs](https://huggingface.co/CubicLabs)}
}
How to Get Started
With Ollama
ollama create axl-reasoning-70m -f Modelfile
ollama run axl-reasoning-70m "def fibonacci():"
With Python
import torch
from multiscale_transformer.model.config import load_config
from multiscale_transformer.model.model import MultiScaleTransformer
from multiscale_transformer.training.tokenizer import ByteTokenizer
config = load_config("config.json")
model = MultiScaleTransformer(config)
ckpt = torch.load("axl_reasoning_70m.pt", map_location="cpu")
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
tokenizer = ByteTokenizer()
prompt = "def fibonacci():"
ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)
with torch.no_grad():
out = model.generate(ids, max_new_tokens=100, temperature=0.8, top_k=40)
print(tokenizer.decode(out[0].tolist()))