| --- |
| license: apache-2.0 |
| language: |
| - ru |
| - en |
| new_version: MixlyGames/Orpheus_Zero-base-0.3b |
| pipeline_tag: text-generation |
| library_name: transformers |
| tags: |
| - gpt |
| - causal-lm |
| - pytorch |
| - unsloth |
| - rope |
| - gqa |
| - transformers |
| --- |
| |
| # Orpheus-Zero Model Card |
|
|
| Orpheus-Zero is a 350-million parameter auto-regressive transformer decoder trained from scratch. It is the baseline model in the Orpheus model family developed by MixlyGames (OwlNestTeam). |
|
|
| The primary motivation of the project is to overcome the structural limitations inherent in commercial and large-scale open-source LLMs, such as unconfigurable safety alignments embedded directly into the model weights and behavioral reversion under single system prompt modifications. |
|
|
| ## Model Specifications |
|
|
| * **Developer:** MixlyGames (OwlNestTeam) |
| * **Model Type:** Causal Language Modeling (Transformer Decoder) |
| * **Architecture:** GPT-style with modern optimizations |
| * **Languages:** Russian (Primary), English |
| * **License:** Apache-2.0 |
|
|
| ### Architectural Features |
|
|
| Orpheus-Zero incorporates modern architectural designs to optimize training efficiency and inference throughput: |
| * **Attention:** Grouped Query Attention (GQA) with 16 query heads and 8 key-value heads. |
| * **Position Embeddings:** Rotary Position Embeddings (RoPE) configured with a base theta of 500,000. |
| * **Activation Function:** SwiGLU. |
| * **Normalization:** RMSNorm. |
|
|
| | Parameter | Value | |
| | :--- | :--- | |
| | **hidden_size** | 1024 | |
| | **num_hidden_layers** | 18 | |
| | **intermediate_size** | 2816 | |
| | **max_position_embeddings** | 4096 | |
| | **vocab_size** | 32,000 | |
| | **Total Parameters** | ~350M | |
| |
| --- |
| |
| ## Training Details |
| |
| ### Dataset and Tokenization |
| |
| The vocabulary is built using a Byte-Pair Encoding (BPE) tokenizer with byte-level fallback to eliminate out-of-vocabulary token loss. The tokenizer was trained on 136,308 documents. |
| |
| The pre-training dataset comprises approximately 30 GB of cleaned text (~400,000 chunks). Documents are formatted using explicit `<|bos|>` and `<|eos|>` boundary tokens and segmented into sequences of 4,096 tokens. |
| |
| The dataset focuses on gaming lore, fiction, and lyric domains, primarily in the Russian language: |
| * **Gaming Domains:** Extensive extractions via MediaWiki API (Fandom) covering *Project Moon* universes (Lobotomy Corporation, Limbus Company), classic series (Half-Life, Portal, Crysis, Detroit: Become Human), and prominent indie titles (Minecraft, Terraria with Calamity Mod, Doki Doki Literature Club, Undertale, OMORI, Don't Starve Together, Celeste, Kinito Pet, Until Then). |
| * **Literary & Lyric Domains:** General fiction corpus and a specialized reference playlist (including lyrics from artists such as Mili, Kikuo, and classic rock compositions). |
| |
| ### Training Environment & Hyperparameters |
| |
| The model was trained on a single local GPU node using Ubuntu 22.04 via WSL2. |
| |
| * **Hardware:** NVIDIA GeForce RTX 4060 Laptop GPU (8 GB VRAM) |
| * **Software Stack:** PyTorch 2.6.0, Transformers 4.57.6, Unsloth 2026.2.1 |
| * **Optimizer:** AdamW 8-bit (via `bitsandbytes`) |
| * **Learning Rate:** 1e-4 with a Cosine Decay schedule (100 warmup steps) |
| * **Effective Batch Size:** 16 (Batch size = 1, Gradient Accumulation = 16) |
| * **Precision:** bfloat16 |
| * **Gradient Clipping:** 1.0 |
| * **Steps per Epoch:** 25,699 (~60 hours per epoch) |
| |
| --- |
| |
| ## Evaluation and Training Dynamics |
| |
| During the initial phase of pre-training, cross-entropy loss tracking demonstrated stable convergence: |
| * Step 10: ~5.3 |
| * Step 1500: ~4.8 |
| |
| Validation and text generation metrics are monitored every 50 steps to verify structural coherence. |
| |
| --- |
| |
| ## Quickstart |
| |
| ### Inference Example |
| |
| To execute text generation using Orpheus-Zero, install the required dependencies and run the initialization script below. |
| |
| ```bash |
| pip install transformers torch accelerate |
| |
| How to Use |
| |
| ### Inference |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| model_id = "MixlyGames/Orpheus_Zero" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| torch_dtype=torch.float16, |
| device_map="auto" |
| ) |
| |
| prompt = "В глубинах корпорации Лоботомия произошло" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=256, |
| do_sample=True, |
| temperature=0.7, |
| top_p=0.9, |
| bos_token_id=tokenizer.bos_token_id, |
| eos_token_id=tokenizer.eos_token_id, |
| pad_token_id=tokenizer.pad_token_id |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
| |
| --- |
| |
| ## Roadmap and Future Work |
| |
| 1. Pre-training Completion:** Finish the baseline 3–5 training epochs of the Orpheus-Zero foundation model. |
| 2. Supervised Fine-Tuning (SFT):** Curate an instruction dataset of 5,000–10,000 high-quality dialog turns for instruction tuning using Unsloth and LoRA. |
| 3. Multimodality: Integrate Vision Transformers (ViT) along with audio and video encoders to expand the model into a multimodal system. |
| 4. Scale-up: Leverage cloud infrastructure (NVIDIA A100/H100 clusters) to train the larger iterations of the family: **Orpheus-KAISER** (4–16B parameters) and **Orpheus-NeiKos** (17B+ parameters). |
