Update README.md

README.md

@@ -1,3 +1,113 @@
 ---
 license: mit
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- code
+- python
+- gguf
+- small-model
+- pretrained-from-scratch
+- gpt2
+- from-scratch
+- coding
+- SLM
 ---
+
+# PyBlissa-Coder-50M
+
+A 50M-parameter Python code generation model trained from scratch on a single RTX 5080. Built as part of the **PRIME** lineup of small, locally-runnable AI systems.
+
+Despite its size, PyBlissa punches well above its weight on Python instruction-following tasks. Trained near-Chinchilla optimal (~13 tokens/parameter) for maximum capacity utilization.
+
+[loss_visualization.png]
+
+## Stats
+
+| | |
+|---|---|
+| Parameters | 50.2M |
+| Architecture | Decoder-only transformer (GPT-2 style) |
+| Context length | 1024 tokens |
+| Vocab size | 16,000 (custom ByteLevel BPE) |
+| Train tokens | 166M |
+| Final val loss | 0.474 |
+| Training time | 73 minutes (RTX 5080) |
+
+## Architecture
+
+```
+d_model:      640
+n_layer:      8
+n_head:       8
+d_ff:         2560
+block_size:   1024
+tied embeddings, pre-LN, no bias, GELU MLP, SDPA attention
+```
+
+## Training data
+
+Two-source code-instruction corpus, 425k samples → 166M tokens after BPE tokenization:
+
+- **`nvidia/OpenCodeInstruct`** — 400k high-quality instruction-code pairs
+- **`flytech/python-codes-25k`** — 25k Python-focused instruction-code pairs
+
+Trained for 4 epochs with cosine LR schedule (3e-4 → 3e-5), bf16 autocast, batch size 20.
+
+## Prompt format
+
+Trained on a strict prefix structure:
+
+```
+PROMPT: <your instruction>
+CODE:
+<generated code>
+```
+
+Anything else is out-of-distribution. The Modelfile in this repo handles the formatting automatically.
+
+## Usage — Ollama (recommended)
+
+```bash
+ollama run hf.co/Rohanify/PyBlissa-Coder-50M
+```
+
+Or pull the GGUF directly and create locally:
+
+```bash
+ollama create pyblissa-coder -f Modelfile
+ollama run pyblissa-coder "write a function to merge two sorted lists"
+```
+
+## Usage — llama.cpp
+
+```bash
+./llama-cli -m PyBlissa-Coder-50M-F32.gguf \
+  -p "PROMPT: write a function to reverse a string\nCODE:\n" \
+  --temp 0.8 --top-k 50 --top-p 0.95 -n 256
+```
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `PyBlissa-Coder-50M-F32.gguf` | Full-precision GGUF weights |
+| `Modelfile` | Ollama config (prompt template, stop tokens, sampling) |
+| `tokenizer.json` | Custom 16k BPE tokenizer |
+
+## Limitations
+
+- Python-only — other languages weren't in training data
+- 1024-token context — longer programs get truncated
+- Small flytech subset (~6% of training data) contains code with unescaped quote bugs; the model occasionally inherits this pattern
+- No safety tuning, no RLHF — base model only
+
+## Acknowledgments
+
+Datasets by NVIDIA and flytech. Built using the nanoGPT-style training recipe with custom tokenization. Tooling: PyTorch, HuggingFace `tokenizers`, llama.cpp for GGUF conversion.
+
+---
+
+Made by Rohan. Also known as ElectroPlayin on YouTube