GrimmCoder_v2 / README.md
Bayhakl1's picture
Update README.md
f7ca116 verified
|
Raw History Blame Contribute Delete
6.54 kB
---
language:
- code
- en
license: apache-2.0
datasets:
- Avelina/python-edu-cleaned
- bigcode/starcoderdata
library_name: transformers
pipeline_tag: text-generation
tags:
- code
- python
- code-completion
- fill-in-the-middle
- fim
- llama
---
# GrimmCoder_v2
**GrimmCoder_v2** is a ~107M-parameter LLaMA-style decoder-only transformer trained from scratch exclusively on Python source code for both **left-to-right completion** and **Fill-in-the-Middle (FIM)** code completion.
This release represents **Phase 2 finetuning** on `Avelina/python-edu-cleaned` (774.1M tokens, 6,000 steps with cosine decay `1e-4 → 1e-5`), initialized from the 8B-token `bigcode/starcoderdata` pretrain base checkpoint.
## Model Summary
| Property | Value |
|---|---|
| Architecture | LLaMA (`LlamaForCausalLM`, pre-norm, RoPE, SwiGLU, GQA, tied embeddings) |
| Parameters | ~106.98M total (~100.69M non-embedding) |
| Context Length | 2048 tokens |
| Hidden Size (`d_model`) | 768 |
| Layers | 16 |
| Attention Heads | 12 query heads / 4 KV heads (GQA) |
| FFN Intermediate Size | 2048 (SwiGLU) |
| Vocabulary | 8192 (byte-level BPE trained on Python) |
| Training Data | `Avelina/python-edu-cleaned` (finetuned from `bigcode/starcoderdata` pretrain) |
| Tokens / Checkpoint | ~8.70B tokens total (7.93B pretrain + 774.1M finetune, step 6,000, val loss 1.2260) |
## Evaluation (HumanEval)
Evaluated on [`openai/openai_humaneval`](https://huggingface.co/datasets/openai/openai_humaneval) (164 Python problems) using `transformers` (`bfloat16`) on CUDA:
| Model Version | Mode & Decoding | `pass@1` | Solved | Syntax Valid |
|---|---|---|---|---|
| **GrimmCoder_v2** | Left-to-Right: greedy (`do_sample=False`) | **9.15%** | **15 / 164** | **137 / 164 (83.5%)** |
| **GrimmCoder_v2** | Left-to-Right: `t=0.2, rep=1.1` | **9.15%** | **15 / 164** | **158 / 164 (96.3%)** |
| **GrimmCoder_v2** | **FIM (PSM): greedy (`do_sample=False`)** | **9.15%** | **15 / 164** | **151 / 164 (92.1%)** |
| **GrimmCoder_v2** | **FIM (PSM): `t=0.2, rep=1.1`** | **8.54%** | **14 / 164** | **153 / 164 (93.3%)** |
| GrimmCoder (Base Pretrain) | Left-to-Right: greedy (`do_sample=False`) | 5.49% | 9 / 164 | 126 / 164 (76.8%) |
| GrimmCoder (Base Pretrain) | Left-to-Right: `t=0.2, rep=1.1` | 4.88% | 8 / 164 | 155 / 164 (94.5%) |
| GrimmCoder (Base Pretrain) | FIM (PSM): greedy (`do_sample=False`) | 4.27% | 7 / 164 | 122 / 164 (74.4%) |
### Solved Problems
- **L2R Greedy (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `34`, `45`, `48`, `50`, `52`, `53`, `55`, `60`
- **L2R t0.2_rep1.1 (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `34`, `45`, `48`, `50`, `53`, `58`, `60`, `61`
- **FIM Greedy (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `42`, `45`, `48`, `50`, `53`, `55`, `58`, `60`
- **FIM t0.2_rep1.1 (14)**: `HumanEval/12`, `16`, `22`, `23`, `28`, `31`, `34`, `35`, `42`, `45`, `48`, `53`, `58`, `60`
- **Overall Distinct Problems Solved Across All Evaluated Modes**: **20 / 164** (`HumanEval/12, 16, 22, 23, 28, 29, 30, 31, 34, 35, 42, 45, 48, 50, 52, 53, 55, 58, 60, 61`)
## Special Tokens
| Token | ID | Purpose |
|---|---|---|
| `<\|endoftext\|>` | `0` | BOS / EOS / document delimiter |
| `<\|fim_prefix\|>` | `1` | FIM prefix marker |
| `<\|fim_middle\|>` | `2` | FIM middle marker (model generates from here) |
| `<\|fim_suffix\|>` | `3` | FIM suffix marker |
| `<\|pad\|>` | `4` | Padding token |
## Usage (`transformers`)
### 1. Left-to-Right Code Completion
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Bayhakl1/GrimmCoder_v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
)
prompt = """def binary_search(arr, target):
lo, hi = 0, len(arr) - 1
"""
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.2,
top_p=0.95,
eos_token_id=[0, 1, 2, 3, 4],
pad_token_id=4,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
### 2. Fill-in-the-Middle (FIM)
GrimmCoder_v2 natively supports Fill-in-the-Middle (FIM) to insert code between a **prefix** (code before the cursor) and a **suffix** (code after the cursor). The model was trained 50/50 on **PSM** and **SPM** formats.
> [!NOTE]
> Special tokens require vertical pipe characters: `<|fim_prefix|>`, `<|fim_suffix|>`, and `<|fim_middle|>`.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Bayhakl1/GrimmCoder_v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
)
prefix = "def add(a: int, b: int) -> int:\n "
suffix = "\n\nprint(add(2, 3))\n"
# --- Method A: String Template (PSM format) ---
# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
inputs = tokenizer(fim_prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
# --- Method B (Alternative): Direct Token IDs ---
# pre_id, mid_id, suf_id = 1, 2, 3
# p_ids = tokenizer.encode(prefix, add_special_tokens=False)
# s_ids = tokenizer.encode(suffix, add_special_tokens=False)
# input_ids = torch.tensor([[pre_id] + p_ids + [suf_id] + s_ids + [mid_id]], device=model.device)
# inputs = {"input_ids": input_ids, "attention_mask": torch.ones_like(input_ids)}
outputs = model.generate(
**inputs,
max_new_tokens=64,
do_sample=True,
temperature=0.2,
top_p=0.95,
eos_token_id=[0, 1, 2, 3, 4],
pad_token_id=4,
)
# Extract and decode only the generated middle tokens
middle_ids = outputs[0][inputs["input_ids"].shape[1]:]
middle = tokenizer.decode(middle_ids, skip_special_tokens=True)
# Reconstruct complete code
print("--- Generated Middle ---")
print(repr(middle))
print("\n--- Complete Code ---")
print(prefix + middle + suffix)
```
#### SPM Mode (Suffix-Prefix-Middle)
If you prefer SPM format, format the prompt as:
```python
fim_prompt = f"<|fim_prefix|><|fim_suffix|>{suffix}<|fim_middle|>{prefix}"
```
## Limitations
- Trained exclusively on Python source code; not intended as a general natural-language chat or instruction-following model.
- Base / completion model: responds best to code prompts and FIM markers rather than conversational Q&A.