README.md

# DynRank: Compressed Model Checkpoints

Hardened SVD-compressed LLM checkpoints from the DynRank paper:
**"DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression"**

## Checkpoint Format

Each `.pt` file is a dictionary:
```python
{
    "step": 1000,
    "ranks": [r_1, r_2, ..., r_224],  # per-sublayer integer ranks
    "state_dict": {...},               # model weights (CompressedLinear format)
    "val_loss": float
}
```

Weight keys use `CompressedLinear` format:
- `model.layers.{i}.self_attn.q_proj.first.weight` — B matrix (r × n)
- `model.layers.{i}.self_attn.q_proj.second.weight` — A matrix (d × r)

## Loading

```python
import torch
ckpt = torch.load("path/to/v8_hardened_step1000.pt", map_location="cpu")
ranks = ckpt["ranks"]        # list of 224 integers
state_dict = ckpt["state_dict"]
```

To use with SVD-LLM's `CompressedLinear`:
```python
from src.model.replace import CompressedLinear
# See https://github.com/zhc1212/SVD-LLM for full loading code
```

## Directory Structure

```
baselines/           # Rank profile JSONs (ASVD, Dobi-SVD proxy, SVD-LLM V2)
phase5a/             # Early TinyLlama logs (no model weights)
phase5b_llama2/      # Main experiments (LLaMA-2-7B, LLaMA-7B, Mistral-7B, LLaMA-2-13B)
phase5e_qwen3/       # Qwen3-8B experiments
```

### phase5b_llama2 key models

| Directory | Model | Method | Ratio | Notes |
|-----------|-------|--------|-------|-------|
| `v9_r04` | LLaMA-2-7B | DynRank | 0.4 | Primary result (seed=42) |
| `static_r04` | LLaMA-2-7B | Static uniform | 0.4 | Matched baseline |
| `v9_r04_s3/s4_refill/s5` | LLaMA-2-7B | DynRank | 0.4 | Multi-seed (43,44,47) |
| `v9_r02/r08` | LLaMA-2-7B | DynRank | 0.2/0.8 | Ratio sweep |
| `llama7b_v9_r04` | LLaMA-7B | DynRank | 0.4 | Cross-model |
| `mistral_v9_r04` | Mistral-7B | DynRank | 0.4 | Cross-model |
| `llama2_13b_v9_r04` | LLaMA-2-13B | DynRank | 0.4 | Scale test |
| `baseline_dobi_r04` | LLaMA-2-7B | Dobi-SVD profile | 0.4 | Baseline comparison |
| `baseline_asvd_r04_v2` | LLaMA-2-7B | ASVD-STRS profile | 0.4 | Baseline comparison |
| `h2_oracle_llama2_r04` | LLaMA-2-7B | Oracle static | 0.4 | H2 ablation |
| `gsm8k_dynrank_r04` | LLaMA-2-7B | DynRank (GSM8K calib) | 0.4 | Domain analysis |

### JSON files

`cross_eval_*.json` — perplexity evaluation results (WT2, C4, LAMBADA)
`downstream7_*.json` — zero-shot accuracy on 6 tasks

## Citation

```bibtex
@article{zhang2026dynrank,
  title={DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression},
  author={Zhang, Huicheng},
  year={2026}
}
```

## License

Model weights are derivatives of:
- LLaMA / LLaMA-2 (Meta Community License)
- Mistral-7B (Apache 2.0)
- Qwen3-8B (Qwen License)

Use subject to the original model licenses.