Blame for README.md

---
license: mit
language:
- en
base_model: microsoft/Phi-4-mini-instruct
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
datasets:
- samadpls/torchlight-threads
tags:
- pytorch
- code
- debugging
- lora
- mlx
- phi3
---

<p align="center">
  <img alt="torchlight" src="https://huggingface.co/samadpls/torchlight-phi4-mini/resolve/main/assets/hero.png" width="100%">
</p>

<p align="center">
  <a href="https://github.com/samadpls/torchlight"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-samadpls%2Ftorchlight-181717?logo=github"></a>
  <a href="https://huggingface.co/datasets/samadpls/torchlight-threads"><img alt="Dataset" src="https://img.shields.io/badge/dataset-torchlight--threads-FFD21E?logo=huggingface&logoColor=000"></a>
  <a href="https://huggingface.co/microsoft/Phi-4-mini-instruct"><img alt="Base model: Phi-4-mini-instruct" src="https://img.shields.io/badge/base-Phi--4--mini--instruct-555"></a>
  <img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-EE4C2C">
  <a href="https://samadpls.com"><img alt="samadpls.com" src="https://img.shields.io/badge/by-samadpls.com-EE4C2C"></a>
</p>

# torchlight-phi4-mini

> This repo has the full model, with the fine-tuning merged in.

## Model Overview

torchlight is a small model that helps with PyTorch questions. It is [Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct) (3.8B parameters), fine-tuned on 6,918 chats from the [PyTorch forums](https://discuss.pytorch.org). Most end with the answer that solved the thread; the rest teach it to ask for missing details.

It was trained not to guess: when a question has enough detail, suggest a fix; when it doesn't, ask for the code or the full error.

- **Trained on real answers.** Each answer is the post that solved the thread. Most were written by [ptrblck](https://discuss.pytorch.org/u/ptrblck).
- **Tested on new questions.** 260 forum questions that are newer than anything the model saw in training.
- **Trained on a laptop.** 6.3 hours on a MacBook Pro (M3 Pro).

## Model Details

| | |
|---|---|
| **Base model** | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) |
| **Size** | 3.8B parameters (11.5M trained with LoRA) |
| **Training data** | [`samadpls/torchlight-threads`](https://huggingface.co/datasets/samadpls/torchlight-threads) |
| **Language** | English |
| **License** | MIT |
| **Author** | [Abdul Samad](https://samadpls.com) |
| **Code** | [github.com/samadpls/torchlight](https://github.com/samadpls/torchlight) |

The model comes in three versions:

| Repo | What it is | Use it with |
|---|---|---|
| [`samadpls/torchlight-phi4-mini`](https://huggingface.co/samadpls/torchlight-phi4-mini) | Full model (16-bit) | transformers, mlx-lm |
| [`samadpls/torchlight-phi4-mini-lora`](https://huggingface.co/samadpls/torchlight-phi4-mini-lora) | LoRA adapter only | PEFT, mlx-lm |
| [`samadpls/torchlight-phi4-mini-GGUF`](https://huggingface.co/samadpls/torchlight-phi4-mini-GGUF) | GGUF files (Q4_K_M, Q8_0) | llama.cpp, Ollama, LM Studio |

## ⚡ Quickstart

### Transformers

```python
from transformers import pipeline

SYSTEM_PROMPT = "You are a careful PyTorch helper. Give specific, correct guidance with code where it helps. If a question lacks the details needed to answer it, ask for them, for example a minimal executable code snippet or the full error message, instead of guessing."

chat = pipeline("text-generation", model="samadpls/torchlight-phi4-mini", dtype="auto", device_map="auto")
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "How do I check which device a tensor is on?"},
]
print(chat(messages, max_new_tokens=512)[0]["generated_text"][-1]["content"])
```

### mlx-lm (Apple Silicon)

```bash
pip install mlx-lm
mlx_lm.generate --model samadpls/torchlight-phi4-mini --max-tokens 512 \
  --system-prompt "You are a careful PyTorch helper. Give specific, correct guidance with code where it helps. If a question lacks the details needed to answer it, ask for them, for example a minimal executable code snippet or the full error message, instead of guessing." \
  --prompt "How do I check which device a tensor is on?"
```

Use the same system prompt as in the examples. The model was trained with it.

## 📊 Evaluation

We asked 260 forum questions from 2025-01-01 to 2026-10-08 to four setups. Claude Opus 5.5, acting as judge, scored each answer from 1 to 5 by comparing it with the answer that solved the thread.

| Model | Average grade (out of 5) | Graded 4 or 5 | Wrong answers (graded 1) | Code with only real torch APIs | Asked for details, when they were missing | Asked for details, when not needed | Searches per question |
|---|---|---|---|---|---|---|---|
| Phi-4-mini (base model) | 1.88 | 5% | 39% | 170 of 175 | 0 of 7 | 0% | 0.0 |
| Phi-4-mini + forum search | 1.75 | 5% | 45% | 145 of 157 | 1 of 7 | 2% | 0.1 |
| torchlight-phi4-mini (fine-tuned) | 1.97 | 7% | 27% | 9 of 10 | 0 of 7 | 7% | 0.0 |
| torchlight-phi4-mini + forum search | 1.95 | 6% | 26% | 4 of 4 | 0 of 7 | 2% | 0.0 |

**What the results show**

- **Far fewer wrong answers.** 27% of torchlight-phi4-mini's answers were graded 1 (wrong), against 39% for the base model: 71 against 102. That gap is real, not chance (p = 0.003).
- **Better than giving the base model forum search.** torchlight-phi4-mini averages 1.97 against 1.75 (95% interval +0.09 to +0.35). Search pulled the small base model off topic: on the 23 questions where it searched, it averaged 1.35.
- **Short and fast.** The median answer is 31 tokens, against 690 for the base model, and it answered all 260 questions in 9 minutes, against 32.
- **The test is hard by design.** The questions are real forum posts from 2025 and 2026, after the model's knowledge ends, and a 4 or 5 needs the same fix that solved the thread. Every small setup gets a 4 or 5 on only 5–7% of questions. torchlight-phi4-mini raises the average grade from 1.88 to 1.97 and good answers from 14 to 19, but those gaps are too small to be sure of.
- **torchlight-phi4-mini never uses the search tool.** None of its training chats included tools, so adding search changes nothing.
- **A bigger model helps.** The same training on Qwen3-8B (8.2B parameters) scored higher on the newest questions. See [`torchlight-qwen3-8b`](https://huggingface.co/samadpls/torchlight-qwen3-8b).

<details>
<summary><b>How the test works</b></summary>

- **Setups.** Phi-4-mini (the base model) and torchlight, each with and without forum search. Search covers 7,438 solved forum threads, up to twice per question.
- **Settings.** The local models use greedy decoding (no sampling) and up to 1,024 new tokens.
- **Scoring.** The judge sees all the answers to a question together, shuffled and without names. It follows [`prompts/judge.md`](https://github.com/samadpls/torchlight/blob/main/prompts/judge.md).
- **Valid torch APIs.** A script checks that every `torch.*` name in an answer's code exists in PyTorch 2.15.
- **Asks for details.** How often an answer asks for more information. This is measured on the 7 questions where the expert also asked, and on the other 253, where asking less is better.
- **Reference answer.** For each question, the judge also scored an answer from Claude Sonnet 5.5 with the same search. It isn't in the table, which compares models of the same size.

</details>

## Training Details

**Data.** 6,918 chats from forum threads created before 2024-07-01. Each chat ends with the answer that solved the thread. 7 chats longer than 4,096 tokens were left out. The 169 validation chats come from 2024-07-01 to 2025-01-01. The test questions are newer than both.

**Settings.**

| | |
|---|---|
| Method | LoRA on the top 16 of 32 layers; the base weights stay frozen |
| LoRA rank / alpha | 16 / 32 |
| Learning rate | 2e-4, cosine schedule, 43 warmup steps |
| Batch | 2 chats, 4 accumulation steps (8 chats per update) |
| Epochs | 1 (3,459 batches) |
| Max length | 4,096 tokens |
| Loss | On the final answer only |
| Optimizer | AdamW, weight decay 0.01 |
| Hardware | MacBook Pro (M3 Pro), 6.3 hours |
| Software | mlx-lm 0.32.2, MLX 0.32.4 |

**Training results.**

<p align="center">
  <img alt="Training and validation loss" src="https://huggingface.co/samadpls/torchlight-phi4-mini/resolve/main/assets/training-loss.png" width="100%">
</p>

| Epoch | Batch | Training loss | Validation loss |
|---|---|---|---|
| 0.00 | 0 | - | 3.280 |
| 0.14 | 500 | 1.995 | - |
| 0.29 | 1,000 | 1.738 | 1.554 |
| 0.43 | 1,500 | 1.713 | - |
| 0.58 | 2,000 | 1.674 | 1.511 |
| 0.72 | 2,500 | 1.665 | - |
| 0.87 | 3,000 | 1.623 | 1.497 |
| 1.00 | 3,459 | 1.687 | 1.495 |

Training loss is the average since the previous row. Batch 0 is the base model before training.

## Intended Use

**Good for:** PyTorch questions about errors, tensor shapes, training loops, data loading and CUDA.

**Not for:** code you run without reading, other frameworks, PyTorch features from after mid-2024, or anything where a wrong answer can cause harm.

## ⚠️ Limitations

- It is a small model and can be wrong while sounding sure. Check its code.
- On new forum questions it rarely finds the real fix on its own (see Evaluation). Use it for quick first ideas, not final answers.
- It asks for missing details less often than it should.
- With greedy decoding, about 1 answer in 65 gets stuck repeating a phrase. A repetition penalty or sampling avoids this.
- It learned the forum's style, so its answers are short. Sometimes they are too short to help on their own.
- It knows PyTorch up to mid-2024, not version 2.4 or later.
- The training data leans toward ptrblck's topics: debugging, CUDA, training loops and shape errors.
- English only.

## License and Credits

The model is released under the MIT license, like the base model. The training data, [`samadpls/torchlight-threads`](https://huggingface.co/datasets/samadpls/torchlight-threads), is under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The answers in it were written by the [PyTorch forum](https://discuss.pytorch.org) community, above all [ptrblck](https://discuss.pytorch.org/u/ptrblck), and every row links to its thread. This model is not made or endorsed by PyTorch or the Linux Foundation.

## Citation

```bibtex
@misc{samad2026torchlight_model,
  author = {Abdul Samad},
  title = {torchlight: a PyTorch helper fine-tuned on resolved forum threads},
  year = {2026},
  url = {https://huggingface.co/samadpls/torchlight-phi4-mini}
}
```

## Contact

Made by **Abdul Samad** ([samadpls.com](https://samadpls.com) · [GitHub @samadpls](https://github.com/samadpls)). Questions or corrections: abdulsamadsid1@gmail.com.