---
license: apache-2.0
base_model: Qwen/Qwen3-8B
library_name: peft
tags:
- peft
- lora
- qwen3
- math
- gsm8k
- supervised-fine-tuning
datasets:
- sumitdotml/lora-and-friends-dataset
---

# LoRA and Friends

This repository contains the six retained PEFT LoRA adapter exports from the
`lora-and-friends` target-module comparison on `Qwen/Qwen3-8B`.

The study compared two adapter scopes on the same rendered math SFT dataset,
using three seeds per condition. All adapters here are the selected
step-3169 checkpoints, chosen by the frozen validation-NLL rule before GSM8K
evaluation.

## Files

```text
checkpoints/best-checkpoints/
  attention_only/seed-0/step-3169/
  attention_only/seed-1/step-3169/
  attention_only/seed-2/step-3169/
  all_layer/seed-0/step-3169/
  all_layer/seed-1/step-3169/
  all_layer/seed-2/step-3169/
```

Each checkpoint directory contains:

- `adapter_config.json`
- `adapter_model.safetensors`
- `checkpoint_complete`

## Conditions

| Condition | Intended adapter scope | Seeds | Selected step |
| --- | --- | ---: | ---: |
| `attention_only` | attention projections only | 0, 1, 2 | 3169 |
| `all_layer` | attention and MLP projections | 0, 1, 2 | 3169 |

The exported PEFT adapter configs record `r=8`, `lora_alpha=32`, and
`lora_dropout=0`.

## GSM8K Results

| Condition | Seed 0 | Seed 1 | Seed 2 | Mean |
| --- | ---: | ---: | ---: | ---: |
| `attention_only` | 0.904473 | 0.906748 | 0.905231 | 0.905484 |
| `all_layer` | 0.899166 | 0.902199 | 0.901440 | 0.900935 |

The untouched `Qwen/Qwen3-8B` baseline in the retained evaluation scored
`0.845337` on the same 1,319-example GSM8K test setup.

## Dataset

The frozen raw and rendered training files are published at:

- https://huggingface.co/datasets/sumitdotml/lora-and-friends-dataset

Use the rendered dataset split for reproduction:

- `rendered/openmath_original_clean_qwen3_disable_thinking/train.jsonl`
- `rendered/openmath_original_clean_qwen3_disable_thinking/val.jsonl`

## Project Article

The technical write-up is published on the project site:

- https://sumit.ml/research/lora-and-friends/