README.md

---
license: mit
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- code
- python
- gguf
- small-model
- pretrained-from-scratch
- gpt2
- from-scratch
- coding
- SLM
---

# 🚀 PyBlissa-Coder-50M
# !! BENCHMARK ON HumaEval DATASET: 10.4% !!

A 50M-parameter Python code generation model trained from scratch on a single RTX 5080. Built as part of the **PRIME** lineup of small, locally-runnable AI systems.

Despite its size, PyBlissa punches well above its weight on Python instruction-following tasks. Trained near-Chinchilla optimal (~13 tokens/parameter) for maximum capacity utilization.

This model punched a solid 10.4% score in OpenAI's HumanEval dataset, which is an amazing number for this model's size!
However, this model technically *can* generate bad outputs. You'd need to tweak the temperature. But that's a rare case!

![Benchmark](pyblissa_humaneval_benchmark.png)


![Loss Visualization](loss_visualization.png)



## Stats

| | |
|---|---|
| Parameters | 50.2M |
| Architecture | Decoder-only transformer (GPT-2 style) |
| Context length | 1024 tokens |
| Vocab size | 16,000 (custom ByteLevel BPE) |
| Train tokens | 166M |
| Final val loss | 0.474 |
| Training time | 73 minutes (RTX 5080) |

## Architecture

```
d_model:      640
n_layer:      8
n_head:       8
d_ff:         2560
block_size:   1024
tied embeddings, pre-LN, no bias, GELU MLP, SDPA attention
```

## Training data

Two-source code-instruction corpus, 425k samples → 166M tokens after BPE tokenization:

- **`nvidia/OpenCodeInstruct`** — 400k high-quality instruction-code pairs
- **`flytech/python-codes-25k`** — 25k Python-focused instruction-code pairs

Trained for 4 epochs with cosine LR schedule (3e-4 → 3e-5), bf16 autocast, batch size 20.

## Prompt format

Trained on a strict prefix structure:

```
PROMPT: <your instruction>
CODE:
<generated code>
```

Anything else is out-of-distribution. The Modelfile in this repo handles the formatting automatically.

## Usage — Ollama (recommended)

```bash
ollama run hf.co/Rohanify/PyBlissa-Coder-50M:F32
```

Or pull the GGUF directly and create locally:

```bash
ollama create pyblissa-coder -f Modelfile
ollama run pyblissa-coder "write a function to merge two sorted lists"
```

## Usage — llama.cpp

```bash
./llama-cli -m PyBlissa-Coder-50M-F32.gguf \
  -p "PROMPT: write a function to reverse a string\nCODE:\n" \
  --temp 0.8 --top-k 50 --top-p 0.95 -n 256
```

## Files

| File | Purpose |
|---|---|
| `PyBlissa-Coder-50M-F32.gguf` | Full-precision GGUF weights |
| `Modelfile` | Ollama config (prompt template, stop tokens, sampling) |
| `tokenizer.json` | Custom 16k BPE tokenizer |

## Limitations

- Mostly Python — other languages weren't that much in training data
- 1024-token context — longer programs get truncated
- Small flytech subset (~6% of training data) contains code with unescaped quote bugs; the model occasionally inherits this pattern
- No safety tuning, no RLHF — base model only

## Acknowledgments

Datasets by NVIDIA and flytech. Built using the nanoGPT-style training recipe with custom tokenization. Tooling: PyTorch, HuggingFace `tokenizers`, llama.cpp for GGUF conversion.

---

Made by Rohan. Also known as ElectroPlayin on YouTube