|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- bigcode/the-stack |
|
|
language: |
|
|
- es |
|
|
- en |
|
|
base_model: |
|
|
- openai-community/gpt2 |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- code |
|
|
new_version: OpceanAI/Yuuki-the-best-model |
|
|
library_name: transformers |
|
|
--- |
|
|
⚠️ ## Notice on Current Model Scope |
|
|
|
|
|
Please note that **Yuuki**, in its current state, represents **approximately 3.7%** of the total training planned for **version v0.1**. |
|
|
|
|
|
At this stage, Yuuki should be considered an **early and incomplete snapshot** of the model. The full **v0.1 release**, which will include the remaining training stages, additional refinements, and stabilization, will be released at a later time. |
|
|
|
|
|
As such, performance, behavior, or capability assessments based on the current version of Yuuki **do not reflect** the final characteristics of the v0.1 model. |
|
|
|
|
|
Further updates will be provided as development progresses. |
|
|
|
|
|
🌸 Yuuki v0.1 - The $0 Code LLM |
|
|
|
|
|
> ⚠️ WORK IN PROGRESS - Currently training on mobile CPU (Day 3/42) |
|
|
|
|
|
|
|
|
|
|
|
🎯 The Mission |
|
|
|
|
|
Prove that you DON'T need expensive GPUs to train LLMs. |
|
|
|
|
|
Yuuki is a code generation model trained entirely on a $150 Android phone with: |
|
|
|
|
|
❌ No cloud compute |
|
|
|
|
|
❌ No GPU |
|
|
|
|
|
❌ No data center |
|
|
|
|
|
✅ Just determination and time |
|
|
|
|
|
|
|
|
The Setup |
|
|
|
|
|
Hardware: Snapdragon 685 (8-core ARM CPU) |
|
|
RAM: 6GB |
|
|
Storage: 128GB |
|
|
NPU: Hexagon 686 (1 TOPS) |
|
|
GPU: Adreno 610 (243 GFLOPS) - NOT USED for training |
|
|
Cost: $0 in compute |
|
|
|
|
|
📊 Current Status |
|
|
|
|
|
Metric Value |
|
|
|
|
|
Progress 1,417 / 37,500 steps (3.78%) |
|
|
Epoch 0.08 / 2.0 |
|
|
Current Loss ~1.70 - 2.23 |
|
|
Best Loss 1.7053 ⭐ |
|
|
Training Time ~3 days |
|
|
ETA ~39 days remaining |
|
|
Speed ~100 sec/step |
|
|
|
|
|
|
|
|
Loss Progression |
|
|
|
|
|
Step 0: Loss 3.35 (baseline) |
|
|
Step 500: Loss 2.50 ↓ -25% |
|
|
Step 1000: Loss 2.00 ↓ -40% |
|
|
Step 1265: Loss 1.83 ↓ -45% |
|
|
Step 1292: Loss 1.71 ↓ -49% ⭐ RECORD |
|
|
Step 1417: Loss 2.23 (current, oscillating 1.7-2.3) |
|
|
|
|
|
🎓 What Yuuki Knows (So Far) |
|
|
|
|
|
Due to alphabetically-ordered dataset: |
|
|
|
|
|
Language Exposure Quality Status |
|
|
|
|
|
Agda High 85/100 ✅ Excellent |
|
|
C Starting 30/100 ⏳ Learning |
|
|
Assembly Low 5/100 🌱 Minimal |
|
|
Python None 0/100 ❌ Not reached yet |
|
|
|
|
|
|
|
|
Example Output (Step 1,300) |
|
|
|
|
|
Agda prompt: module Main where |
|
|
|
|
|
module Main where (x, f) in a |
|
|
|
|
|
open import Cubical.Sigma |
|
|
open import Cubical.Sigma.Core |
|
|
open import Cubical.Foundations.H |
|
|
|
|
|
✅ Real Agda libraries! The model learned actual Cubical type theory modules. |
|
|
|
|
|
🛠️ Training Configuration |
|
|
Model: DistilGPT-2 (82M parameters) |
|
|
Dataset: The Stack (75,000 examples) |
|
|
Batch size: 1 |
|
|
Gradient accumulation: 4 |
|
|
Effective batch: 4 |
|
|
Learning rate: 5e-5 |
|
|
Max length: 256 tokens |
|
|
Optimizer: AdamW |
|
|
Epochs: 2 |
|
|
Total tokens: ~30M (2 epochs) |
|
|
|
|
|
Why so slow? |
|
|
100 seconds/step × 37,500 steps = 3,750,000 seconds |
|
|
= 1,042 hours |
|
|
= 43.4 days |
|
|
= ~6 weeks of continuous training |
|
|
No GPU acceleration. Pure CPU grinding. 💪 |
|
|
|
|
|
📈 Roadmap |
|
|
|
|
|
v0.1 (Current - Proof of Concept) |
|
|
|
|
|
[x] Setup training pipeline |
|
|
|
|
|
[x] Start training (Step 0) |
|
|
|
|
|
[x] Reach Step 1,000 |
|
|
|
|
|
[x] Break loss 2.0 barrier |
|
|
|
|
|
[x] Break loss 1.8 barrier ⭐ |
|
|
|
|
|
[ ] Checkpoint 2,500 (7%) |
|
|
|
|
|
[ ] Checkpoint 5,000 (13%) |
|
|
|
|
|
[ ] Checkpoint 10,000 (27%) |
|
|
|
|
|
[ ] Checkpoint 18,750 (50% - Epoch 1 complete) |
|
|
|
|
|
[ ] Checkpoint 37,500 (100% - DONE) |
|
|
|
|
|
[ ] Quantize to INT8 |
|
|
|
|
|
[ ] Convert to ONNX |
|
|
|
|
|
[ ] Publish final model |
|
|
|
|
|
ETA: Mid-March 2026 |
|
|
|
|
|
|
|
|
v0.2 (The Full Dataset) |
|
|
|
|
|
Dataset: 786,387 examples (full Stack) |
|
|
|
|
|
Duration: 418 days (~14 months) |
|
|
|
|
|
Epochs: 2.0 |
|
|
|
|
|
Total tokens: ~314M |
|
|
|
|
|
Dataset fix: SHUFFLED (not alphabetical) |
|
|
|
|
|
Languages: All 80+ languages balanced |
|
|
|
|
|
Start: March 2026 |
|
|
|
|
|
End: May 2027 |
|
|
|
|
|
|
|
|
v0.3+ (PC Era) |
|
|
|
|
|
Hardware upgrade: RTX 4060/4070 |
|
|
|
|
|
Larger models: 350M-1B parameters |
|
|
|
|
|
Faster training: ~30x speedup |
|
|
|
|
|
Advanced techniques: LoRA, QLoRA, etc. |
|
|
|
|
|
|
|
|
|
|
|
💡 Philosophy |
|
|
"The barrier to AI isn't money. It's mindset." |
|
|
This project demonstrates: ✅ You CAN train LLMs without GPUs |
|
|
✅ Patience > Hardware |
|
|
✅ $0 budget is enough to start |
|
|
✅ Limited resources inspire creativity |
|
|
✅ Anyone can contribute to AI |
|
|
|
|
|
🚀 Usage (After Training Completes) |
|
|
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
# Load model |
|
|
model = AutoModelForCausalLM.from_pretrained("OpceanAI/Yuuki") |
|
|
tokenizer = AutoTokenizer.from_pretrained("OpceanAI/Yuuki") |
|
|
|
|
|
# Generate code |
|
|
prompt = "def fibonacci(n):" |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_length=100) |
|
|
code = tokenizer.decode(outputs[0]) |
|
|
print(code) |
|
|
|
|
|
Quantized (4x faster, 4x smaller) |
|
|
|
|
|
Coming after training completes |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
"OpceanAI/Yuuki", |
|
|
subfolder="yuuki-v0.1-int8" |
|
|
) |
|
|
|
|
|
⚠️ Known Limitations |
|
|
|
|
|
Dataset order: Alphabetical (not shuffled) - learns early languages best |
|
|
|
|
|
Token count: Only ~30M tokens (vs GPT-2's 40B) |
|
|
|
|
|
Training speed: Very slow (~100 sec/step) |
|
|
|
|
|
Model size: Small (82M params) |
|
|
|
|
|
Language coverage: Incomplete due to alphabetical ordering |
|
|
These will be addressed in v0.2 with shuffled dataset. |
|
|
|
|
|
|
|
|
🔬 Technical Details |
|
|
|
|
|
CPU Training (100 sec/step): |
|
|
|
|
|
Forward pass: 40 sec |
|
|
|
|
|
Backward pass: 40 sec |
|
|
|
|
|
Optimizer: 20 sec |
|
|
|
|
|
Total: ~100 sec |
|
|
|
|
|
|
|
|
vs GPU Training (0.5 sec/step): |
|
|
|
|
|
200x faster |
|
|
|
|
|
But costs $0.50-$2.00/hour |
|
|
|
|
|
42 days = $500-$2,000 |
|
|
|
|
|
|
|
|
Mobile: FREE but SLOW |
|
|
|
|
|
GPU: FAST but EXPENSIVE |
|
|
|
|
|
For proof of concept: Mobile wins. 🏆 |
|
|
|
|
|
|
|
|
📊 Benchmarks (Post-Training) |
|
|
|
|
|
Coming soon after training completes (~March 2026). |
|
|
Expected performance: |
|
|
|
|
|
Agda: 85-95/100 (primary language) |
|
|
|
|
|
C: 85-92/100 (secondary language) |
|
|
|
|
|
Assembly: 75-85/100 (tertiary) |
|
|
|
|
|
Python: 10-20/100 (barely seen due to alphabet order) |
|
|
|
|
|
|
|
|
|
|
|
🙏 Acknowledgments |
|
|
|
|
|
HuggingFace: Infrastructure and transformers library |
|
|
|
|
|
BigCode: The Stack dataset |
|
|
|
|
|
The ML community: For saying "you need GPUs" - best motivation 😏 |
|
|
|
|
|
|
|
|
📜 License |
|
|
|
|
|
Apache 2.0 - See LICENSE file. You can use Yuuki commercially, modify it, distribute it. Just give credit. ✅ |
|
|
|
|
|
|
|
|
🔗 Links |
|
|
|
|
|
GitHub: (https://github.com/aguitauwu) |
|
|
|
|
|
Discord: (https://discord.gg/j8zV2u8k) |
|
|
|
|
|
Progress updates: Check this model card |
|
|
|
|
|
|
|
|
📅 Updates |
|
|
|
|
|
2026-01-29: Training started |
|
|
|
|
|
2026-01-29: Step 1,000 reached - Loss 2.00 |
|
|
|
|
|
2026-01-29: Step 1,292 - NEW RECORD Loss 1.7053 |
|
|
|
|
|
2026-01-29: Repository created on HuggingFace |
|
|
|
|
|
Last updated: 2026-01-29 |
|
|
|
|
|
|
|
|
Follow the journey of training an LLM with $0 budget. One step at a time. 🌸 |
