---
base_model:
- meta-llama/Llama-3.1-8B
library_name: transformers
license: llama3.1
tags:
- pruning
- sparsity
- 2:4-sparsity
- patch
- maskllm
- mask
pipeline_tag: text-generation
---

<div align="center">
<img src="./PATCH-Logo.png" alt="PATCH" width="360">
</div>

# llama_3.1_8b-PATCH-35Sparse

> **This checkpoint** (LLaMA-3.1 8B, PATCH-Tile, 35% sparsity): **55.28%** average zero-shot accuracy, **7.89** WikiText2 perplexity.

[![Paper](https://img.shields.io/badge/arXiv-2509.23410-b31b1b.svg)](https://arxiv.org/abs/2509.23410) [![GitHub](https://img.shields.io/badge/GitHub-Paramathic%2Fpatch-black.svg?logo=github)](https://github.com/Paramathic/patch)

This repository hosts a **mask only** release for the paper
**[PATCH: Learnable Tile-level Hybrid Sparsity for LLMs](https://arxiv.org/abs/2509.23410)**.
PATCH (Pruning with a Learnable Tile-level Configuration for Hybrid Sparsity) learns a structured mask on **frozen** pretrained weights, assigning each tile as dense (0% sparsity) or 2:4 sparse (50% sparsity) to hit a flexible global sparsity target while staying hardware-friendly.

Because PATCH/MaskLLM keep the base weights **frozen**, we distribute *only the
binary keep/prune mask* (bit-packed in `mask.npz`) - **no weight values**. You
recover the sparse model by downloading the original base model and applying the
mask.

- Base model: [`meta-llama/Llama-3.1-8B`](https://huggingface.co/meta-llama/Llama-3.1-8B)
- Method: **PATCH-Tile** &nbsp;|&nbsp; Target sparsity: **35%** &nbsp;|&nbsp; Pattern: **Dense / 2:4 tiles**
- Measured mask sparsity: **35.00%** over 224 Linear layers (6,979,321,856 weights).

<div align="center">
<img src="./PATCH-Pipeline.svg" alt="PATCH pipeline" width="760">
</div>

## Results (LLaMA-3.1 8B)

| Sparsity | Method | Pattern | Avg Acc (% ↑) | WikiText2 PPL (↓) |
|---|---|---|---|---|
| 0% | Dense | - | 60.31 | 5.84 |
| 50% | Magnitude | 2:4 | 35.93 | 765.92 |
| 50% | Wanda | 2:4 | 41.77 | 21.29 |
| 50% | SparseGPT | 2:4 | 45.53 | 15.11 |
| 50% | Thanos | 2:4 | 45.72 | 16.09 |
| 50% | ProxSparse | 2:4 | 45.14 | 15.17 |
| 50% | MaskLLM | 2:4 | 52.80 | 8.58 |
| 45% | PATCH-Tile | Dense/2:4 | 53.60 | 8.20 |
| 35% | **PATCH-Tile** ⭐ | Dense/2:4 | **55.28** | **7.89** |
| 25% | PATCH-Tile | Dense/2:4 | 56.48 | 7.34 |


Per-task zero-shot accuracy (%) for this checkpoint:

| MMLU | PIQA | ARC-E | ARC-C | WinoG. | OBQA | RACE | HellaS. | **Average** |
|---|---|---|---|---|---|---|---|---|
| 51.15 | 77.97 | 76.14 | 42.41 | 69.46 | 31.40 | 38.18 | 55.54 | **55.28** |

All numbers are from the PATCH paper ([arXiv:2509.23410](https://arxiv.org/abs/2509.23410)); accuracy
is the average over MMLU, PIQA, ARC-Easy, ARC-Challenge, Winogrande, OpenBookQA,
RACE and HellaSwag, evaluated with the LM-Evaluation-Harness. PPL is WikiText2.

## Training hyper-parameters

| Hyper-parameter | Value |
|---|---|
| Fine-tuning dataset | SlimPajama (2B tokens) |
| Training steps | 2000 |
| Global batch size | 256 |
| Sequence length | 4096 |
| Mask tile size | 128 x 128 (hardware tiles: 128x128 / 128x64 / 64x128 / 64x64) |
| Logits init. | N(0, 0.014) |
| Tile-logit prior | SparseGPT (strength 3) |
| Regularization scope | Global (single target density) |
| Evaluation | LM-Eval-Harness (8 zero-shot tasks) + WikiText2 PPL @ seqlen 4096 |
| Hardware | 1 node x 4 GPUs, data parallel (HuggingFace Trainer) |
| Optimizer | Adam |
| Learning rate | 1e-4 |
| Gumbel scaling (kappa) | 100 -> 500 |
| Gumbel temp (tau) | 2 -> 0.05 |
| Sparsity reg. (lambda1) | 3 |
| Weight reg. (lambda2) | 0.1 |

## How to use

```python
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch
from load_patch_mask import apply_patch_mask  # shipped in this repo

npz = hf_hub_download(repo_id="mohammad-mozaffari/llama_3.1_8b-PATCH-35Sparse", filename="mask.npz")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B", torch_dtype=torch.bfloat16)
apply_patch_mask(model, npz)   # zeroes the pruned weights in place
```

Or from the command line:

```bash
python load_patch_mask.py --base_model meta-llama/Llama-3.1-8B --mask_repo mohammad-mozaffari/llama_3.1_8b-PATCH-35Sparse
```

Speedup on real hardware requires a 2:4-aware / hybrid sparse kernel; see the
[GitHub repository](https://github.com/Paramathic/patch) and [STOICC](https://github.com/Paramathic/stoicc).

## License

The released mask is a derivative of the base model and is distributed under the
base model's license (**`llama3.1`**). You must comply with that license and
obtain access to the base model separately.

> Built with Llama. Use governed by the Llama 3.1 Community License.

The mask-generation code is released under the MIT license (see the
[PATCH repository](https://github.com/Paramathic/patch)).

## Citation

```bibtex
@article{hourri2025patch,
    title  = {PATCH: Learnable Tile-level Hybrid Sparsity for LLMs},
    author = {Hourri, Younes and Mozaffari, Mohammad and Mehri Dehnavi, Maryam},
    year   = 2025,
    journal = {arXiv preprint arXiv:2509.23410}
}
```