AXL-Chat-10M

Conversational AI. 9.9M params. PPL 1.02. Context 512 bytes. Part of the AXL model family by CubicLabs.

Model Details

Property	Value
Developed by	CubicLabs
Architecture	Multi-Scale Transformer
Parameters	10M
Optimizer	Lion
Attention	SDPA
Vocab Size	258 (byte-level)
Context Window	512 bytes
d_model	224
Attention Heads	4
Layers per Scale	3
Downsample Factors	[1, 2, 4]
License	Apache 2.0

Sources

• Repository: https://github.com/Cubic/AXL
• Organization: https://huggingface.co/CubicLabs

Uses

Direct Use

Conversational AI for programming Q&A.

Example Usage: import torch from multiscale_transformer.model.model import MultiScaleTransformer from multiscale_transformer.training.tokenizer import ByteTokenizer ckpt = torch.load("axl_chat_10m.pt", map_location="cpu") model = MultiScaleTransformer(config) model.load_state_dict(ckpt["model_state_dict"]) model.eval() tokenizer = ByteTokenizer() ids = torch.tensor([tokenizer.encode("def hello():")], dtype=torch.long) with torch.no_grad(): out = model.generate(ids, max_new_tokens=50, temperature=0.8) print(tokenizer.decode(out[0].tolist()))

Out-of-Scope Use

Not for production code generation. Not for non-code NLP tasks. For integration with tools like Continue.dev, LlamaIndex, or LangChain, use the Python API server which provides OpenAI-compatible endpoints.

Bias, Risks, and Limitations

Byte-level perplexity is not comparable to BPE-level perplexity. Max context 512 bytes. Note: GGUF files for Ollama use a simplified single-stack encoder. For full AXL quality, use the Python API server.

Recommendations

• Use for prototyping and experimentation, not production code generation.
• Byte-level perplexity (258 vocab) is not comparable to BPE-level perplexity (32K vocab).
• For better results, use the Lion-optimized version if available.

Training Details

Training Data

Retrained with Lion on 10MB chat pairs. 216 steps in 10 min. Covers code Q&A, general knowledge.

Preprocessing

Byte-level tokenization with vocabulary size 258 (256 bytes + BOS + EOS). No vocabulary training required.

Evaluation

Metrics

Perplexity on held-out Python code using byte-level tokenization.

Results

Perplexity (byte-level): 1.02 Final Loss: 0.3650 Training Steps: 216 Training Time: 10 min

Environmental Impact

Hardware: AMD Ryzen 5 5600G Hours Used: 0.167 Carbon Emitted: 0.0070 kg CO2 Cloud Provider: None (local CPU)

Citation

@misc{axl_2026, title={AXL: AXL-Chat-10M - Multi-Scale Transformer for CPU Code Generation}, author={Cubic}, year={2026}, url={https://huggingface.co/CubicLabs} }

How to Get Started

With Ollama

ollama create axl-chat-10m -f Modelfile ollama run axl-chat-10m "def fibonacci():"

With Python

import torch from multiscale_transformer.model.config import load_config from multiscale_transformer.model.model import MultiScaleTransformer from multiscale_transformer.training.tokenizer import ByteTokenizer config = load_config("config.json") model = MultiScaleTransformer(config) ckpt = torch.load("axl_chat_10m.pt", map_location="cpu") model.load_state_dict(ckpt["model_state_dict"]) model.eval() tokenizer = ByteTokenizer() prompt = "def fibonacci():" ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long) with torch.no_grad(): out = model.generate(ids, max_new_tokens=100, temperature=0.8, top_k=40) print(tokenizer.decode(out[0].tolist()))