Orpheus-Zero Model Card
Orpheus-Zero is a 350-million parameter auto-regressive transformer decoder trained from scratch. It is the baseline model in the Orpheus model family developed by MixlyGames (OwlNestTeam).
The primary motivation of the project is to overcome the structural limitations inherent in commercial and large-scale open-source LLMs, such as unconfigurable safety alignments embedded directly into the model weights and behavioral reversion under single system prompt modifications.
Model Specifications
- Developer: MixlyGames (OwlNestTeam)
- Model Type: Causal Language Modeling (Transformer Decoder)
- Architecture: GPT-style with modern optimizations
- Languages: Russian (Primary), English
- License: Apache-2.0
Architectural Features
Orpheus-Zero incorporates modern architectural designs to optimize training efficiency and inference throughput:
- Attention: Grouped Query Attention (GQA) with 16 query heads and 8 key-value heads.
- Position Embeddings: Rotary Position Embeddings (RoPE) configured with a base theta of 500,000.
- Activation Function: SwiGLU.
- Normalization: RMSNorm.
| Parameter | Value |
|---|---|
| hidden_size | 1024 |
| num_hidden_layers | 18 |
| intermediate_size | 2816 |
| max_position_embeddings | 4096 |
| vocab_size | 32,000 |
| Total Parameters | ~350M |
Training Details
Dataset and Tokenization
The vocabulary is built using a Byte-Pair Encoding (BPE) tokenizer with byte-level fallback to eliminate out-of-vocabulary token loss. The tokenizer was trained on 136,308 documents.
The pre-training dataset comprises approximately 30 GB of cleaned text (~400,000 chunks). Documents are formatted using explicit <|bos|> and <|eos|> boundary tokens and segmented into sequences of 4,096 tokens.
The dataset focuses on gaming lore, fiction, and lyric domains, primarily in the Russian language:
- Gaming Domains: Extensive extractions via MediaWiki API (Fandom) covering Project Moon universes (Lobotomy Corporation, Limbus Company), classic series (Half-Life, Portal, Crysis, Detroit: Become Human), and prominent indie titles (Minecraft, Terraria with Calamity Mod, Doki Doki Literature Club, Undertale, OMORI, Don't Starve Together, Celeste, Kinito Pet, Until Then).
- Literary & Lyric Domains: General fiction corpus and a specialized reference playlist (including lyrics from artists such as Mili, Kikuo, and classic rock compositions).
Training Environment & Hyperparameters
The model was trained on a single local GPU node using Ubuntu 22.04 via WSL2.
- Hardware: NVIDIA GeForce RTX 4060 Laptop GPU (8 GB VRAM)
- Software Stack: PyTorch 2.6.0, Transformers 4.57.6, Unsloth 2026.2.1
- Optimizer: AdamW 8-bit (via
bitsandbytes) - Learning Rate: 1e-4 with a Cosine Decay schedule (100 warmup steps)
- Effective Batch Size: 16 (Batch size = 1, Gradient Accumulation = 16)
- Precision: bfloat16
- Gradient Clipping: 1.0
- Steps per Epoch: 25,699 (~60 hours per epoch)
Evaluation and Training Dynamics
During the initial phase of pre-training, cross-entropy loss tracking demonstrated stable convergence:
- Step 10: ~5.3
- Step 1500: ~4.8
Validation and text generation metrics are monitored every 50 steps to verify structural coherence.
Quickstart
Inference Example
To execute text generation using Orpheus-Zero, install the required dependencies and run the initialization script below.
pip install transformers torch accelerate
How to Use
### Inference
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "MixlyGames/Orpheus_Zero"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto"
)
prompt = "В глубинах корпорации Лоботомия произошло"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=256,
do_sample=True,
temperature=0.7,
top_p=0.9,
bos_token_id=tokenizer.bos_token_id,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Roadmap and Future Work
- Pre-training Completion:** Finish the baseline 3–5 training epochs of the Orpheus-Zero foundation model.
- Supervised Fine-Tuning (SFT):** Curate an instruction dataset of 5,000–10,000 high-quality dialog turns for instruction tuning using Unsloth and LoRA.
- Multimodality: Integrate Vision Transformers (ViT) along with audio and video encoders to expand the model into a multimodal system.
- Scale-up: Leverage cloud infrastructure (NVIDIA A100/H100 clusters) to train the larger iterations of the family: Orpheus-KAISER (4–16B parameters) and Orpheus-NeiKos (17B+ parameters).