GrimmCoder_v2

GrimmCoder_v2 is a ~107M-parameter LLaMA-style decoder-only transformer trained from scratch exclusively on Python source code for both left-to-right completion and Fill-in-the-Middle (FIM) code completion.

This release represents Phase 2 finetuning on Avelina/python-edu-cleaned (774.1M tokens, 6,000 steps with cosine decay 1e-4 → 1e-5), initialized from the 8B-token bigcode/starcoderdata pretrain base checkpoint.

Model Summary

Property Value
Architecture LLaMA (LlamaForCausalLM, pre-norm, RoPE, SwiGLU, GQA, tied embeddings)
Parameters 106.98M total (100.69M non-embedding)
Context Length 2048 tokens
Hidden Size (d_model) 768
Layers 16
Attention Heads 12 query heads / 4 KV heads (GQA)
FFN Intermediate Size 2048 (SwiGLU)
Vocabulary 8192 (byte-level BPE trained on Python)
Training Data Avelina/python-edu-cleaned (finetuned from bigcode/starcoderdata pretrain)
Tokens / Checkpoint ~8.70B tokens total (7.93B pretrain + 774.1M finetune, step 6,000, val loss 1.2260)

Evaluation (HumanEval)

Evaluated on openai/openai_humaneval (164 Python problems) using transformers (bfloat16) on CUDA:

Model Version Mode & Decoding pass@1 Solved Syntax Valid
GrimmCoder_v2 Left-to-Right: greedy (do_sample=False) 9.15% 15 / 164 137 / 164 (83.5%)
GrimmCoder_v2 Left-to-Right: t=0.2, rep=1.1 9.15% 15 / 164 158 / 164 (96.3%)
GrimmCoder_v2 FIM (PSM): greedy (do_sample=False) 9.15% 15 / 164 151 / 164 (92.1%)
GrimmCoder_v2 FIM (PSM): t=0.2, rep=1.1 8.54% 14 / 164 153 / 164 (93.3%)
GrimmCoder (Base Pretrain) Left-to-Right: greedy (do_sample=False) 5.49% 9 / 164 126 / 164 (76.8%)
GrimmCoder (Base Pretrain) Left-to-Right: t=0.2, rep=1.1 4.88% 8 / 164 155 / 164 (94.5%)
GrimmCoder (Base Pretrain) FIM (PSM): greedy (do_sample=False) 4.27% 7 / 164 122 / 164 (74.4%)

Solved Problems

  • L2R Greedy (15): HumanEval/16, 22, 23, 28, 29, 30, 31, 34, 45, 48, 50, 52, 53, 55, 60
  • L2R t0.2_rep1.1 (15): HumanEval/16, 22, 23, 28, 29, 30, 31, 34, 45, 48, 50, 53, 58, 60, 61
  • FIM Greedy (15): HumanEval/16, 22, 23, 28, 29, 30, 31, 42, 45, 48, 50, 53, 55, 58, 60
  • FIM t0.2_rep1.1 (14): HumanEval/12, 16, 22, 23, 28, 31, 34, 35, 42, 45, 48, 53, 58, 60
  • Overall Distinct Problems Solved Across All Evaluated Modes: 20 / 164 (HumanEval/12, 16, 22, 23, 28, 29, 30, 31, 34, 35, 42, 45, 48, 50, 52, 53, 55, 58, 60, 61)

Special Tokens

Token ID Purpose
<|endoftext|> 0 BOS / EOS / document delimiter
<|fim_prefix|> 1 FIM prefix marker
<|fim_middle|> 2 FIM middle marker (model generates from here)
<|fim_suffix|> 3 FIM suffix marker
<|pad|> 4 Padding token

Usage (transformers)

1. Left-to-Right Code Completion

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Bayhakl1/GrimmCoder_v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = """def binary_search(arr, target):
    lo, hi = 0, len(arr) - 1
"""
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.2,
    top_p=0.95,
    eos_token_id=[0, 1, 2, 3, 4],
    pad_token_id=4,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Fill-in-the-Middle (FIM)

GrimmCoder_v2 natively supports Fill-in-the-Middle (FIM) to insert code between a prefix (code before the cursor) and a suffix (code after the cursor). The model was trained 50/50 on PSM and SPM formats.

Special tokens require vertical pipe characters: <|fim_prefix|>, <|fim_suffix|>, and <|fim_middle|>.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Bayhakl1/GrimmCoder"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prefix = "def add(a: int, b: int) -> int:\n    "
suffix = "\n\nprint(add(2, 3))\n"

# --- Method A: String Template (PSM format) ---
# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
inputs = tokenizer(fim_prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

# --- Method B (Alternative): Direct Token IDs ---
# pre_id, mid_id, suf_id = 1, 2, 3
# p_ids = tokenizer.encode(prefix, add_special_tokens=False)
# s_ids = tokenizer.encode(suffix, add_special_tokens=False)
# input_ids = torch.tensor([[pre_id] + p_ids + [suf_id] + s_ids + [mid_id]], device=model.device)
# inputs = {"input_ids": input_ids, "attention_mask": torch.ones_like(input_ids)}

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=True,
    temperature=0.2,
    top_p=0.95,
    eos_token_id=[0, 1, 2, 3, 4],
    pad_token_id=4,
)

# Extract and decode only the generated middle tokens
middle_ids = outputs[0][inputs["input_ids"].shape[1]:]
middle = tokenizer.decode(middle_ids, skip_special_tokens=True)

# Reconstruct complete code
print("--- Generated Middle ---")
print(repr(middle))
print("\n--- Complete Code ---")
print(prefix + middle + suffix)

SPM Mode (Suffix-Prefix-Middle)

If you prefer SPM format, format the prompt as:

fim_prompt = f"<|fim_prefix|><|fim_suffix|>{suffix}<|fim_middle|>{prefix}"

Limitations

  • Trained exclusively on Python source code; not intended as a general natural-language chat or instruction-following model.
  • Base / completion model: responds best to code prompts and FIM markers rather than conversational Q&A.
Downloads last month
140
Safetensors
Model size
0.1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train Bayhakl1/GrimmCoder_v2