HIMANSHUKUMARJHA/minicpm5-1b-code-lora

LoRA adapter for openbmb/MiniCPM5-1B, fine-tuned for coding assistance.

Trained by AutoTune, a scheduled fine-tuning pipeline on Modal that only publishes an adapter when it beats the previous best on a held-out slice.

Results

Metric	Value
Eval loss (200 held-out rows)	0.8707
Previous published adapter	0.9127

The eval gate published this version only because it beat the stored previous best on the same fixed holdout.

Caveat on the metric. Eval loss measures token prediction, not whether generated code runs. Treat it as a training-health signal rather than a capability benchmark; pass@1 on HumanEval or MBPP would be the meaningful measure and is not yet wired up.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "openbmb/MiniCPM5-1B"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "HIMANSHUKUMARJHA/minicpm5-1b-code-lora")

messages = [{'role': 'user', 'content': 'Write a Python function that reverses a linked list.'}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
inputs.pop("token_type_ids", None)   # this architecture's generate() rejects it
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Requires transformers>=4.51, which is the first version that reads MiniCPM's standalone chat_template.jinja.

Training

Base model	openbmb/MiniCPM5-1B
Dataset	m-a-p/CodeFeedback-Filtered-Instruction
Training rows	60,000
Steps	6,000 (~1.6 epochs at effective batch 16)
LoRA rank / alpha	32 / 64
Target modules	attention + MLP projections
LR schedule	0.0002 cosine, 3% warmup
Precision	bfloat16
Hardware	1x A100

Reproduced with AutoTune:

modal run finetune.py --profile code

Data handling

The dataset is shuffled with a fixed seed, then a 200-row holdout is taken before training. Rows the formatter cannot parse are dropped and exact duplicates removed, because these datasets contain repeats that would otherwise leak holdout examples into training.

12,515 exact duplicates were removed from this dataset before training.

Note on the metric. Eval loss measures token prediction, not whether generated code runs. Treat it as a training-health signal, not a capability benchmark. Pass@1 on HumanEval or MBPP would be the meaningful measure.

Limitations

This is a 1B parameter model. It is useful for coding assistance at small scale and for on-device or cost-sensitive settings, but it will not match a large general model. Outputs should be validated before use. The adapter inherits any bias present in the training dataset.