--- language: - code - en license: apache-2.0 datasets: - Avelina/python-edu-cleaned - bigcode/starcoderdata library_name: transformers pipeline_tag: text-generation tags: - code - python - code-completion - fill-in-the-middle - fim - llama --- # GrimmCoder_v2 **GrimmCoder_v2** is a ~107M-parameter LLaMA-style decoder-only transformer trained from scratch exclusively on Python source code for both **left-to-right completion** and **Fill-in-the-Middle (FIM)** code completion. This release represents **Phase 2 finetuning** on `Avelina/python-edu-cleaned` (774.1M tokens, 6,000 steps with cosine decay `1e-4 → 1e-5`), initialized from the 8B-token `bigcode/starcoderdata` pretrain base checkpoint. ## Model Summary | Property | Value | |---|---| | Architecture | LLaMA (`LlamaForCausalLM`, pre-norm, RoPE, SwiGLU, GQA, tied embeddings) | | Parameters | ~106.98M total (~100.69M non-embedding) | | Context Length | 2048 tokens | | Hidden Size (`d_model`) | 768 | | Layers | 16 | | Attention Heads | 12 query heads / 4 KV heads (GQA) | | FFN Intermediate Size | 2048 (SwiGLU) | | Vocabulary | 8192 (byte-level BPE trained on Python) | | Training Data | `Avelina/python-edu-cleaned` (finetuned from `bigcode/starcoderdata` pretrain) | | Tokens / Checkpoint | ~8.70B tokens total (7.93B pretrain + 774.1M finetune, step 6,000, val loss 1.2260) | ## Evaluation (HumanEval) Evaluated on [`openai/openai_humaneval`](https://huggingface.co/datasets/openai/openai_humaneval) (164 Python problems) using `transformers` (`bfloat16`) on CUDA: | Model Version | Mode & Decoding | `pass@1` | Solved | Syntax Valid | |---|---|---|---|---| | **GrimmCoder_v2** | Left-to-Right: greedy (`do_sample=False`) | **9.15%** | **15 / 164** | **137 / 164 (83.5%)** | | **GrimmCoder_v2** | Left-to-Right: `t=0.2, rep=1.1` | **9.15%** | **15 / 164** | **158 / 164 (96.3%)** | | **GrimmCoder_v2** | **FIM (PSM): greedy (`do_sample=False`)** | **9.15%** | **15 / 164** | **151 / 164 (92.1%)** | | **GrimmCoder_v2** | **FIM (PSM): `t=0.2, rep=1.1`** | **8.54%** | **14 / 164** | **153 / 164 (93.3%)** | | GrimmCoder (Base Pretrain) | Left-to-Right: greedy (`do_sample=False`) | 5.49% | 9 / 164 | 126 / 164 (76.8%) | | GrimmCoder (Base Pretrain) | Left-to-Right: `t=0.2, rep=1.1` | 4.88% | 8 / 164 | 155 / 164 (94.5%) | | GrimmCoder (Base Pretrain) | FIM (PSM): greedy (`do_sample=False`) | 4.27% | 7 / 164 | 122 / 164 (74.4%) | ### Solved Problems - **L2R Greedy (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `34`, `45`, `48`, `50`, `52`, `53`, `55`, `60` - **L2R t0.2_rep1.1 (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `34`, `45`, `48`, `50`, `53`, `58`, `60`, `61` - **FIM Greedy (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `42`, `45`, `48`, `50`, `53`, `55`, `58`, `60` - **FIM t0.2_rep1.1 (14)**: `HumanEval/12`, `16`, `22`, `23`, `28`, `31`, `34`, `35`, `42`, `45`, `48`, `53`, `58`, `60` - **Overall Distinct Problems Solved Across All Evaluated Modes**: **20 / 164** (`HumanEval/12, 16, 22, 23, 28, 29, 30, 31, 34, 35, 42, 45, 48, 50, 52, 53, 55, 58, 60, 61`) ## Special Tokens | Token | ID | Purpose | |---|---|---| | `<\|endoftext\|>` | `0` | BOS / EOS / document delimiter | | `<\|fim_prefix\|>` | `1` | FIM prefix marker | | `<\|fim_middle\|>` | `2` | FIM middle marker (model generates from here) | | `<\|fim_suffix\|>` | `3` | FIM suffix marker | | `<\|pad\|>` | `4` | Padding token | ## Usage (`transformers`) ### 1. Left-to-Right Code Completion ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Bayhakl1/GrimmCoder_v2" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, dtype=torch.bfloat16, device_map="auto", ) prompt = """def binary_search(arr, target): lo, hi = 0, len(arr) - 1 """ inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device) outputs = model.generate( **inputs, max_new_tokens=128, do_sample=True, temperature=0.2, top_p=0.95, eos_token_id=[0, 1, 2, 3, 4], pad_token_id=4, ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` ### 2. Fill-in-the-Middle (FIM) GrimmCoder_v2 natively supports Fill-in-the-Middle (FIM) to insert code between a **prefix** (code before the cursor) and a **suffix** (code after the cursor). The model was trained 50/50 on **PSM** and **SPM** formats. > [!NOTE] > Special tokens require vertical pipe characters: `<|fim_prefix|>`, `<|fim_suffix|>`, and `<|fim_middle|>`. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Bayhakl1/GrimmCoder_v2" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, dtype=torch.bfloat16, device_map="auto", ) prefix = "def add(a: int, b: int) -> int:\n " suffix = "\n\nprint(add(2, 3))\n" # --- Method A: String Template (PSM format) --- # Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|> fim_prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>" inputs = tokenizer(fim_prompt, return_tensors="pt", add_special_tokens=False).to(model.device) # --- Method B (Alternative): Direct Token IDs --- # pre_id, mid_id, suf_id = 1, 2, 3 # p_ids = tokenizer.encode(prefix, add_special_tokens=False) # s_ids = tokenizer.encode(suffix, add_special_tokens=False) # input_ids = torch.tensor([[pre_id] + p_ids + [suf_id] + s_ids + [mid_id]], device=model.device) # inputs = {"input_ids": input_ids, "attention_mask": torch.ones_like(input_ids)} outputs = model.generate( **inputs, max_new_tokens=64, do_sample=True, temperature=0.2, top_p=0.95, eos_token_id=[0, 1, 2, 3, 4], pad_token_id=4, ) # Extract and decode only the generated middle tokens middle_ids = outputs[0][inputs["input_ids"].shape[1]:] middle = tokenizer.decode(middle_ids, skip_special_tokens=True) # Reconstruct complete code print("--- Generated Middle ---") print(repr(middle)) print("\n--- Complete Code ---") print(prefix + middle + suffix) ``` #### SPM Mode (Suffix-Prefix-Middle) If you prefer SPM format, format the prompt as: ```python fim_prompt = f"<|fim_prefix|><|fim_suffix|>{suffix}<|fim_middle|>{prefix}" ``` ## Limitations - Trained exclusively on Python source code; not intended as a general natural-language chat or instruction-following model. - Base / completion model: responds best to code prompts and FIM markers rather than conversational Q&A.