File size: 3,848 Bytes
bc1c1fa | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 | ---
base_model: microsoft/Phi-4-mini-instruct
library_name: peft
license: mit
pipeline_tag: text-generation
tags:
- lora
- peft
- small-language-model
- enterprise-benchmark
- code-generation
- base_model:adapter:microsoft/Phi-4-mini-instruct
---
# phi-4-mini-instruct — Code generation adapter
A LoRA adapter that specialises `microsoft/Phi-4-mini-instruct` (3.80 B parameters) for a single enterprise task: it completes a Python function so that it passes the reference unit tests.
It was produced for the MSc thesis *Fine-Tune or Pay Per Token? An Enterprise Benchmark of Small Language Models* (SRH University Hamburg), which measures fine-tuned small models against frontier provider APIs on accuracy, latency, cost, privacy exposure and return-on-investment breakeven volume. The adapter is released so that the benchmark can be independently verified.
## Measured performance
| Metric | Value |
|---|---|
| pass@1 | **0.4287** |
| Mean latency, batch 1 | 2200 ms |
| Cost per 1M generated tokens | USD 9.53 |
Measured on a single NVIDIA H200 (141 GB) at batch size one and full utilisation, priced at an imputed USD 3.99 per GPU-hour. Latency excludes network transit. Scores are not comparable across tasks — each task carries its own metric. Evaluation ran on 5 July 2026; the complete matrix is at [`results/benchmark_matrix.csv`](https://github.com/Yusifnuri/slm-benchmark/blob/main/results/benchmark_matrix.csv).
## Training
| | |
|---|---|
| Method | LoRA |
| Dataset | HumanEval (`openai/openai_humaneval`) |
| Dataset licence | MIT |
| Training examples | 5,000 (500 held out for checkpoint selection) |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| Learning rate | 2e-4, cosine schedule, 3% warmup |
| Epochs | 3 |
| Effective batch size | 16 (4 x 4 gradient accumulation) |
| Max sequence length | 512 tokens |
| Optimiser | AdamW |
| Seed | 42 |
Hyperparameters were held constant across every model and task rather than tuned per cell, so these figures are a conservative lower bound on attainable performance.
## Usage
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("microsoft/Phi-4-mini-instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "<your-hf-username>/phi-4-mini-instruct_code_generation")
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-4-mini-instruct")
```
The adapter was trained on this prompt format and expects it at inference:
```text
Complete the following Python function:
{text}
```
## Limitations
- Trained once, with a single seed. Reported differences confound model quality with initialisation variance.
- Specialised to one task on one public corpus. It is not a general-purpose assistant and should not be treated as one.
- The evaluation corpora are long-standing public benchmarks and are plausibly present in the base model's pretraining data, which inflates absolute scores.
- Evaluation used 200 held-out instances (all 164 problems for code generation), so detectable effect sizes are bounded at roughly ten percentage points.
## Links
- Code, configurations and evaluation harness: [https://github.com/Yusifnuri/slm-benchmark](https://github.com/Yusifnuri/slm-benchmark)
- Full benchmark matrix: [`results/benchmark_matrix.csv`](https://github.com/Yusifnuri/slm-benchmark/blob/main/results/benchmark_matrix.csv)
- Per-request cost analysis: [`results/cost_per_request.csv`](https://github.com/Yusifnuri/slm-benchmark/blob/main/results/cost_per_request.csv)
## Citation
```bibtex
@mastersthesis{nuri2026finetune,
title = {Fine-Tune or Pay Per Token? An Enterprise Benchmark of Small Language Models},
author = {Nuri, Yusif},
school = {SRH University Hamburg},
year = {2026}
}
```
|