Point to the Qwen3-8B fine-tune in the results

cad58d4 verified 2 days ago

10.8 kB

---
license: mit
language:
- en
base_model: microsoft/Phi-4-mini-instruct
base_model_relation: adapter
library_name: peft
pipeline_tag: text-generation
datasets:
- samadpls/torchlight-threads
tags:
- pytorch
- code
- debugging
- lora
- mlx
- phi3
---
<p align="center">
  <img alt="torchlight" src="https://huggingface.co/samadpls/torchlight-phi4-mini-lora/resolve/main/assets/hero.png" width="100%">
</p>

<p align="center">
  <a href="https://github.com/samadpls/torchlight"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-samadpls%2Ftorchlight-181717?logo=github"></a>
  <a href="https://huggingface.co/datasets/samadpls/torchlight-threads"><img alt="Dataset" src="https://img.shields.io/badge/dataset-torchlight--threads-FFD21E?logo=huggingface&logoColor=000"></a>
  <a href="https://huggingface.co/microsoft/Phi-4-mini-instruct"><img alt="Base model: Phi-4-mini-instruct" src="https://img.shields.io/badge/base-Phi--4--mini--instruct-555"></a>
  <img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-EE4C2C">
  <a href="https://samadpls.com"><img alt="samadpls.com" src="https://img.shields.io/badge/by-samadpls.com-EE4C2C"></a>
</p>

# torchlight-phi4-mini-lora

> This repo has only the LoRA adapter, in PEFT format. The mlx-lm version is in `mlx/`. For the full model, see [`samadpls/torchlight-phi4-mini`](https://huggingface.co/samadpls/torchlight-phi4-mini).

## Model Overview

torchlight is a small model that helps with PyTorch questions. It is [Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct) (3.8B parameters), fine-tuned on 6,918 chats from the [PyTorch forums](https://discuss.pytorch.org). Most end with the answer that solved the thread; the rest teach it to ask for missing details.

It was trained not to guess: when a question has enough detail, suggest a fix; when it doesn't, ask for the code or the full error.

- **Trained on real answers.** Each answer is the post that solved the thread. Most were written by [ptrblck](https://discuss.pytorch.org/u/ptrblck).
- **Tested on new questions.** 260 forum questions that are newer than anything the model saw in training.
- **Trained on a laptop.** 6.3 hours on a MacBook Pro (M3 Pro).

## Model Details

| | |
|---|---|
| **Base model** | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) |
| **Size** | 3.8B parameters (11.5M trained with LoRA) |
| **Training data** | [`samadpls/torchlight-threads`](https://huggingface.co/datasets/samadpls/torchlight-threads) |
| **Language** | English |
| **License** | MIT |
| **Author** | [Abdul Samad](https://samadpls.com) |
| **Code** | [github.com/samadpls/torchlight](https://github.com/samadpls/torchlight) |

The model comes in three versions:

| Repo | What it is | Use it with |
|---|---|---|
| [`samadpls/torchlight-phi4-mini`](https://huggingface.co/samadpls/torchlight-phi4-mini) | Full model (16-bit) | transformers, mlx-lm |
| [`samadpls/torchlight-phi4-mini-lora`](https://huggingface.co/samadpls/torchlight-phi4-mini-lora) | LoRA adapter only | PEFT, mlx-lm |
| [`samadpls/torchlight-phi4-mini-GGUF`](https://huggingface.co/samadpls/torchlight-phi4-mini-GGUF) | GGUF files (Q4_K_M, Q8_0) | llama.cpp, Ollama, LM Studio |
## ⚡ Quickstart
### PEFT
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("microsoft/Phi-4-mini-instruct", dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "samadpls/torchlight-phi4-mini-lora")
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-4-mini-instruct")
model = model.merge_and_unload()  # optional: merge the adapter into the model
```
### mlx-lm (Apple Silicon)
```bash
hf download samadpls/torchlight-phi4-mini-lora --include "mlx/*" --local-dir torchlight-lora
mlx_lm.generate --model microsoft/Phi-4-mini-instruct --adapter-path torchlight-lora/mlx --max-tokens 512 \
  --system-prompt "You are a careful PyTorch helper. Give specific, correct guidance with code where it helps. If a question lacks the details needed to answer it, ask for them, for example a minimal executable code snippet or the full error message, instead of guessing." \
  --prompt "How do I check which device a tensor is on?"
```
Use the same system prompt as in the examples. The model was trained with it.
## 📊 Evaluation
We asked 260 forum questions from 2025-01-01 to 2026-10-08 to four setups. Claude Opus 5.5, acting as judge, scored each answer from 1 to 5 by comparing it with the answer that solved the thread.
| Model | Average grade (out of 5) | Graded 4 or 5 | Wrong answers (graded 1) | Code with only real torch APIs | Asked for details, when they were missing | Asked for details, when not needed | Searches per question |
|---|---|---|---|---|---|---|---|
| Phi-4-mini (base model) | 1.88 | 5% | 39% | 170 of 175 | 0 of 7 | 0% | 0.0 |
| Phi-4-mini + forum search | 1.75 | 5% | 45% | 145 of 157 | 1 of 7 | 2% | 0.1 |
| torchlight-phi4-mini (fine-tuned) | 1.97 | 7% | 27% | 9 of 10 | 0 of 7 | 7% | 0.0 |
| torchlight-phi4-mini + forum search | 1.95 | 6% | 26% | 4 of 4 | 0 of 7 | 2% | 0.0 |
**What the results show**
- **Far fewer wrong answers.** 27% of torchlight-phi4-mini's answers were graded 1 (wrong), against 39% for the base model: 71 against 102. That gap is real, not chance (p = 0.003).
- **Better than giving the base model forum search.** torchlight-phi4-mini averages 1.97 against 1.75 (95% interval +0.09 to +0.35). Search pulled the small base model off topic: on the 23 questions where it searched, it averaged 1.35.
- **Short and fast.** The median answer is 31 tokens, against 690 for the base model, and it answered all 260 questions in 9 minutes, against 32.
- **The test is hard by design.** The questions are real forum posts from 2025 and 2026, after the model's knowledge ends, and a 4 or 5 needs the same fix that solved the thread. Every small setup gets a 4 or 5 on only 5–7% of questions. torchlight-phi4-mini raises the average grade from 1.88 to 1.97 and good answers from 14 to 19, but those gaps are too small to be sure of.
- **torchlight-phi4-mini never uses the search tool.** None of its training chats included tools, so adding search changes nothing.
- **A bigger model helps.** The same training on Qwen3-8B (8.2B parameters) scored higher on the newest questions. See [`torchlight-qwen3-8b`](https://huggingface.co/samadpls/torchlight-qwen3-8b).
<details>
<summary><b>How the test works</b></summary>
- **Setups.** Phi-4-mini (the base model) and torchlight, each with and without forum search. Search covers 7,438 solved forum threads, up to twice per question.
- **Settings.** The local models use greedy decoding (no sampling) and up to 1,024 new tokens.
- **Scoring.** The judge sees all the answers to a question together, shuffled and without names. It follows [`prompts/judge.md`](https://github.com/samadpls/torchlight/blob/main/prompts/judge.md).
- **Valid torch APIs.** A script checks that every `torch.*` name in an answer's code exists in PyTorch 2.15.
- **Asks for details.** How often an answer asks for more information. This is measured on the 7 questions where the expert also asked, and on the other 253, where asking less is better.
- **Reference answer.** For each question, the judge also scored an answer from Claude Sonnet 5.5 with the same search. It isn't in the table, which compares models of the same size.
</details>
## Training Details
**Data.** 6,918 chats from forum threads created before 2024-07-01. Each chat ends with the answer that solved the thread. 7 chats longer than 4,096 tokens were left out. The 169 validation chats come from 2024-07-01 to 2025-01-01. The test questions are newer than both.
**Settings.**
| | |
|---|---|
| Method | LoRA on the top 16 of 32 layers; the base weights stay frozen |
| LoRA rank / alpha | 16 / 32 |
| Learning rate | 2e-4, cosine schedule, 43 warmup steps |
| Batch | 2 chats, 4 accumulation steps (8 chats per update) |
| Epochs | 1 (3,459 batches) |
| Max length | 4,096 tokens |
| Loss | On the final answer only |
| Optimizer | AdamW, weight decay 0.01 |
| Hardware | MacBook Pro (M3 Pro), 6.3 hours |
| Software | mlx-lm 0.32.2, MLX 0.32.4 |
**Training results.**
<p align="center">
  <img alt="Training and validation loss" src="https://huggingface.co/samadpls/torchlight-phi4-mini-lora/resolve/main/assets/training-loss.png" width="100%">
</p>
| Epoch | Batch | Training loss | Validation loss |
|---|---|---|---|
| 0.00 | 0 | - | 3.280 |
| 0.14 | 500 | 1.995 | - |
| 0.29 | 1,000 | 1.738 | 1.554 |
| 0.43 | 1,500 | 1.713 | - |
| 0.58 | 2,000 | 1.674 | 1.511 |
| 0.72 | 2,500 | 1.665 | - |
| 0.87 | 3,000 | 1.623 | 1.497 |
| 1.00 | 3,459 | 1.687 | 1.495 |
Training loss is the average since the previous row. Batch 0 is the base model before training.
## Intended Use
**Good for:** PyTorch questions about errors, tensor shapes, training loops, data loading and CUDA.
**Not for:** code you run without reading, other frameworks, PyTorch features from after mid-2024, or anything where a wrong answer can cause harm.
## ⚠️ Limitations
- It is a small model and can be wrong while sounding sure. Check its code.
- On new forum questions it rarely finds the real fix on its own (see Evaluation). Use it for quick first ideas, not final answers.
- It asks for missing details less often than it should.
- With greedy decoding, about 1 answer in 65 gets stuck repeating a phrase. A repetition penalty or sampling avoids this.
- It learned the forum's style, so its answers are short. Sometimes they are too short to help on their own.
- It knows PyTorch up to mid-2024, not version 2.4 or later.
- The training data leans toward ptrblck's topics: debugging, CUDA, training loops and shape errors.
- English only.
## License and Credits
The model is released under the MIT license, like the base model. The training data, [`samadpls/torchlight-threads`](https://huggingface.co/datasets/samadpls/torchlight-threads), is under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The answers in it were written by the [PyTorch forum](https://discuss.pytorch.org) community, above all [ptrblck](https://discuss.pytorch.org/u/ptrblck), and every row links to its thread. This model is not made or endorsed by PyTorch or the Linux Foundation.
## Citation
```bibtex
@misc{samad2026torchlight_model,
  author = {Abdul Samad},
  title = {torchlight: a PyTorch helper fine-tuned on resolved forum threads},
  year = {2026},
  url = {https://huggingface.co/samadpls/torchlight-phi4-mini}
}
```
## Contact
Made by **Abdul Samad** ([samadpls.com](https://samadpls.com) · [GitHub @samadpls](https://github.com/samadpls)). Questions or corrections: abdulsamadsid1@gmail.com.

	---
	license: mit
	language:
	- en
	base_model: microsoft/Phi-4-mini-instruct
	base_model_relation: adapter
	library_name: peft
	pipeline_tag: text-generation
	datasets:
	- samadpls/torchlight-threads
	tags:
	- pytorch
	- code
	- debugging
	- lora
	- mlx
	- phi3
	---

	<p align="center">
	<img alt="torchlight" src="https://huggingface.co/samadpls/torchlight-phi4-mini-lora/resolve/main/assets/hero.png" width="100%">
	</p>

	<p align="center">
	<a href="https://github.com/samadpls/torchlight"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-samadpls%2Ftorchlight-181717?logo=github"></a>
	<a href="https://huggingface.co/datasets/samadpls/torchlight-threads"><img alt="Dataset" src="https://img.shields.io/badge/dataset-torchlight--threads-FFD21E?logo=huggingface&logoColor=000"></a>
	<a href="https://huggingface.co/microsoft/Phi-4-mini-instruct"><img alt="Base model: Phi-4-mini-instruct" src="https://img.shields.io/badge/base-Phi--4--mini--instruct-555"></a>
	<img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-EE4C2C">
	<a href="https://samadpls.com"><img alt="samadpls.com" src="https://img.shields.io/badge/by-samadpls.com-EE4C2C"></a>
	</p>

	# torchlight-phi4-mini-lora

	> This repo has only the LoRA adapter, in PEFT format. The mlx-lm version is in `mlx/`. For the full model, see [`samadpls/torchlight-phi4-mini`](https://huggingface.co/samadpls/torchlight-phi4-mini).

	## Model Overview

	torchlight is a small model that helps with PyTorch questions. It is [Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct) (3.8B parameters), fine-tuned on 6,918 chats from the [PyTorch forums](https://discuss.pytorch.org). Most end with the answer that solved the thread; the rest teach it to ask for missing details.

	It was trained not to guess: when a question has enough detail, suggest a fix; when it doesn't, ask for the code or the full error.

	- Trained on real answers. Each answer is the post that solved the thread. Most were written by [ptrblck](https://discuss.pytorch.org/u/ptrblck).
	- Tested on new questions. 260 forum questions that are newer than anything the model saw in training.
	- Trained on a laptop. 6.3 hours on a MacBook Pro (M3 Pro).

	## Model Details

	\| \| \|
	\|---\|---\|
	\| Base model \| [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) \|
	\| Size \| 3.8B parameters (11.5M trained with LoRA) \|
	\| Training data \| [`samadpls/torchlight-threads`](https://huggingface.co/datasets/samadpls/torchlight-threads) \|
	\| Language \| English \|
	\| License \| MIT \|
	\| Author \| [Abdul Samad](https://samadpls.com) \|
	\| Code \| [github.com/samadpls/torchlight](https://github.com/samadpls/torchlight) \|

	The model comes in three versions:

	\| Repo \| What it is \| Use it with \|
	\|---\|---\|---\|
	\| [`samadpls/torchlight-phi4-mini`](https://huggingface.co/samadpls/torchlight-phi4-mini) \| Full model (16-bit) \| transformers, mlx-lm \|
	\| [`samadpls/torchlight-phi4-mini-lora`](https://huggingface.co/samadpls/torchlight-phi4-mini-lora) \| LoRA adapter only \| PEFT, mlx-lm \|
	\| [`samadpls/torchlight-phi4-mini-GGUF`](https://huggingface.co/samadpls/torchlight-phi4-mini-GGUF) \| GGUF files (Q4_K_M, Q8_0) \| llama.cpp, Ollama, LM Studio \|

	## ⚡ Quickstart

	### PEFT

	```python
	from peft import PeftModel
	from transformers import AutoModelForCausalLM, AutoTokenizer

	base = AutoModelForCausalLM.from_pretrained("microsoft/Phi-4-mini-instruct", dtype="auto", device_map="auto")
	model = PeftModel.from_pretrained(base, "samadpls/torchlight-phi4-mini-lora")
	tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-4-mini-instruct")
	model = model.merge_and_unload() # optional: merge the adapter into the model
	```

	### mlx-lm (Apple Silicon)

	```bash
	hf download samadpls/torchlight-phi4-mini-lora --include "mlx/*" --local-dir torchlight-lora
	mlx_lm.generate --model microsoft/Phi-4-mini-instruct --adapter-path torchlight-lora/mlx --max-tokens 512 \
	--system-prompt "You are a careful PyTorch helper. Give specific, correct guidance with code where it helps. If a question lacks the details needed to answer it, ask for them, for example a minimal executable code snippet or the full error message, instead of guessing." \
	--prompt "How do I check which device a tensor is on?"
	```

	Use the same system prompt as in the examples. The model was trained with it.

	## 📊 Evaluation

	We asked 260 forum questions from 2025-01-01 to 2026-10-08 to four setups. Claude Opus 5.5, acting as judge, scored each answer from 1 to 5 by comparing it with the answer that solved the thread.

	\| Model \| Average grade (out of 5) \| Graded 4 or 5 \| Wrong answers (graded 1) \| Code with only real torch APIs \| Asked for details, when they were missing \| Asked for details, when not needed \| Searches per question \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| Phi-4-mini (base model) \| 1.88 \| 5% \| 39% \| 170 of 175 \| 0 of 7 \| 0% \| 0.0 \|
	\| Phi-4-mini + forum search \| 1.75 \| 5% \| 45% \| 145 of 157 \| 1 of 7 \| 2% \| 0.1 \|
	\| torchlight-phi4-mini (fine-tuned) \| 1.97 \| 7% \| 27% \| 9 of 10 \| 0 of 7 \| 7% \| 0.0 \|
	\| torchlight-phi4-mini + forum search \| 1.95 \| 6% \| 26% \| 4 of 4 \| 0 of 7 \| 2% \| 0.0 \|

	What the results show

	- Far fewer wrong answers. 27% of torchlight-phi4-mini's answers were graded 1 (wrong), against 39% for the base model: 71 against 102. That gap is real, not chance (p = 0.003).
	- Better than giving the base model forum search. torchlight-phi4-mini averages 1.97 against 1.75 (95% interval +0.09 to +0.35). Search pulled the small base model off topic: on the 23 questions where it searched, it averaged 1.35.
	- Short and fast. The median answer is 31 tokens, against 690 for the base model, and it answered all 260 questions in 9 minutes, against 32.
	- The test is hard by design. The questions are real forum posts from 2025 and 2026, after the model's knowledge ends, and a 4 or 5 needs the same fix that solved the thread. Every small setup gets a 4 or 5 on only 5–7% of questions. torchlight-phi4-mini raises the average grade from 1.88 to 1.97 and good answers from 14 to 19, but those gaps are too small to be sure of.
	- torchlight-phi4-mini never uses the search tool. None of its training chats included tools, so adding search changes nothing.
	- A bigger model helps. The same training on Qwen3-8B (8.2B parameters) scored higher on the newest questions. See [`torchlight-qwen3-8b`](https://huggingface.co/samadpls/torchlight-qwen3-8b).

	<details>
	<summary><b>How the test works</b></summary>

	- Setups. Phi-4-mini (the base model) and torchlight, each with and without forum search. Search covers 7,438 solved forum threads, up to twice per question.
	- Settings. The local models use greedy decoding (no sampling) and up to 1,024 new tokens.
	- Scoring. The judge sees all the answers to a question together, shuffled and without names. It follows [`prompts/judge.md`](https://github.com/samadpls/torchlight/blob/main/prompts/judge.md).
	- Valid torch APIs. A script checks that every `torch.*` name in an answer's code exists in PyTorch 2.15.
	- Asks for details. How often an answer asks for more information. This is measured on the 7 questions where the expert also asked, and on the other 253, where asking less is better.
	- Reference answer. For each question, the judge also scored an answer from Claude Sonnet 5.5 with the same search. It isn't in the table, which compares models of the same size.

	</details>

	## Training Details

	Data. 6,918 chats from forum threads created before 2024-07-01. Each chat ends with the answer that solved the thread. 7 chats longer than 4,096 tokens were left out. The 169 validation chats come from 2024-07-01 to 2025-01-01. The test questions are newer than both.

	Settings.

	\| \| \|
	\|---\|---\|
	\| Method \| LoRA on the top 16 of 32 layers; the base weights stay frozen \|
	\| LoRA rank / alpha \| 16 / 32 \|
	\| Learning rate \| 2e-4, cosine schedule, 43 warmup steps \|
	\| Batch \| 2 chats, 4 accumulation steps (8 chats per update) \|
	\| Epochs \| 1 (3,459 batches) \|
	\| Max length \| 4,096 tokens \|
	\| Loss \| On the final answer only \|
	\| Optimizer \| AdamW, weight decay 0.01 \|
	\| Hardware \| MacBook Pro (M3 Pro), 6.3 hours \|
	\| Software \| mlx-lm 0.32.2, MLX 0.32.4 \|

	Training results.

	<p align="center">
	<img alt="Training and validation loss" src="https://huggingface.co/samadpls/torchlight-phi4-mini-lora/resolve/main/assets/training-loss.png" width="100%">
	</p>

	\| Epoch \| Batch \| Training loss \| Validation loss \|
	\|---\|---\|---\|---\|
	\| 0.00 \| 0 \| - \| 3.280 \|
	\| 0.14 \| 500 \| 1.995 \| - \|
	\| 0.29 \| 1,000 \| 1.738 \| 1.554 \|
	\| 0.43 \| 1,500 \| 1.713 \| - \|
	\| 0.58 \| 2,000 \| 1.674 \| 1.511 \|
	\| 0.72 \| 2,500 \| 1.665 \| - \|
	\| 0.87 \| 3,000 \| 1.623 \| 1.497 \|
	\| 1.00 \| 3,459 \| 1.687 \| 1.495 \|

	Training loss is the average since the previous row. Batch 0 is the base model before training.

	## Intended Use

	Good for: PyTorch questions about errors, tensor shapes, training loops, data loading and CUDA.

	Not for: code you run without reading, other frameworks, PyTorch features from after mid-2024, or anything where a wrong answer can cause harm.

	## ⚠️ Limitations

	- It is a small model and can be wrong while sounding sure. Check its code.
	- On new forum questions it rarely finds the real fix on its own (see Evaluation). Use it for quick first ideas, not final answers.
	- It asks for missing details less often than it should.
	- With greedy decoding, about 1 answer in 65 gets stuck repeating a phrase. A repetition penalty or sampling avoids this.
	- It learned the forum's style, so its answers are short. Sometimes they are too short to help on their own.
	- It knows PyTorch up to mid-2024, not version 2.4 or later.
	- The training data leans toward ptrblck's topics: debugging, CUDA, training loops and shape errors.
	- English only.

	## License and Credits

	The model is released under the MIT license, like the base model. The training data, [`samadpls/torchlight-threads`](https://huggingface.co/datasets/samadpls/torchlight-threads), is under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The answers in it were written by the [PyTorch forum](https://discuss.pytorch.org) community, above all [ptrblck](https://discuss.pytorch.org/u/ptrblck), and every row links to its thread. This model is not made or endorsed by PyTorch or the Linux Foundation.

	## Citation

	```bibtex
	@misc{samad2026torchlight_model,
	author = {Abdul Samad},
	title = {torchlight: a PyTorch helper fine-tuned on resolved forum threads},
	year = {2026},
	url = {https://huggingface.co/samadpls/torchlight-phi4-mini}
	}
	```

	## Contact

	Made by Abdul Samad ([samadpls.com](https://samadpls.com) · [GitHub @samadpls](https://github.com/samadpls)). Questions or corrections: abdulsamadsid1@gmail.com.