README.md

---
library_name: transformers
tags:
- safetensors
- pruna_pro-ai
- pruna-ai
---

# Model Card for loulou2/tiny_llama_hqq

This model was created using the [pruna](https://github.com/PrunaAI/pruna) library. Pruna is a model optimization framework built for developers, enabling you to deliver more efficient models with minimal implementation overhead.

## Usage

First things first, you need to install the pruna library:

```bash
pip install pruna_pro
```

You can [use the transformers library to load the model](https://huggingface.co/loulou2/tiny_llama_hqq?library=transformers) but this might not include all optimizations by default.

To ensure that all optimizations are applied, use the pruna library to load the model using the following code:

```python
from pruna_pro import PrunaProModel

loaded_model = PrunaProModel.from_pretrained(
    "loulou2/tiny_llama_hqq"
)
# we can then run inference using the methods supported by the base model
```

 Alternatively, you can visit [the Pruna documentation](https://docs.pruna.ai/en/stable/) for more information.

## Smash Configuration

The compression configuration of the model is stored in the `smash_config.json` file, which describes the optimization methods that were applied to the model.

```bash
{
    "batcher": null,
    "cacher": null,
    "compiler": null,
    "distiller": null,
    "distributer": null,
    "enhancer": null,
    "factorizer": null,
    "kernel": null,
    "pruner": null,
    "quantizer": "hqq",
    "recoverer": null,
    "hqq_backend": "torchao_int4",
    "hqq_compute_dtype": "torch.bfloat16",
    "hqq_force_hf_implementation": true,
    "hqq_group_size": 64,
    "hqq_use_torchao_kernels": false,
    "hqq_weight_bits": 4,
    "batch_size": 1,
    "device": "cuda",
    "device_map": null,
    "save_fns": [],
    "load_fns": [
        "transformers"
    ],
    "reapply_after_load": {
        "factorizer": null,
        "pruner": null,
        "quantizer": null,
        "distiller": null,
        "kernel": null,
        "cacher": null,
        "recoverer": null,
        "distributer": null,
        "compiler": null,
        "batcher": null,
        "enhancer": null
    }
}
```

## 🌍 Join the Pruna AI community!

[![Twitter](https://img.shields.io/twitter/follow/PrunaAI?style=social)](https://twitter.com/PrunaAI)
[![GitHub](https://img.shields.io/github/followers/PrunaAI?label=Follow%20%40PrunaAI&style=social)](https://github.com/PrunaAI)
[![LinkedIn](https://img.shields.io/badge/LinkedIn-Connect-blue)](https://www.linkedin.com/company/93832878/admin/feed/posts/?feedType=following)
[![Discord](https://img.shields.io/badge/Discord-Join%20Us-blue?style=social&logo=discord)](https://discord.gg/JFQmtFKCjd)
[![Reddit](https://img.shields.io/reddit/subreddit-subscribers/PrunaAI?style=social)](https://www.reddit.com/r/PrunaAI/)