Update README.md
Browse files
README.md
CHANGED
|
@@ -4,4 +4,53 @@ library_name: transformers
|
|
| 4 |
tags:
|
| 5 |
- custom_generate
|
| 6 |
- sampling
|
| 7 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
tags:
|
| 5 |
- custom_generate
|
| 6 |
- sampling
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
# DeepCONF Custom Generation Strategy
|
| 11 |
+
|
| 12 |
+
This repository implements the DeepCONF (Deep Confidence-based Early Stopping) generation strategy for Hugging Face Transformers models, following the [Deep Think with Confidence](https://jiaweizzhao.github.io/deepconf/) approach from the paper [Deep Think with Confidence](https://huggingface.co/papers/2508.15260).
|
| 13 |
+
|
| 14 |
+
## Overview
|
| 15 |
+
|
| 16 |
+
DeepCONF monitors the confidence of generated tokens and stops generation when confidence falls below a threshold.
|
| 17 |
+
|
| 18 |
+
## Parameters
|
| 19 |
+
|
| 20 |
+
- `enable_conf` (bool): Whether to enable the DeepCONF strategy. Defaults to `False`.
|
| 21 |
+
- `window_size` (int): Size of the sliding window for confidence calculation. Defaults to `2048`.
|
| 22 |
+
- `threshold` (float): Confidence threshold for early stopping. Defaults to `17.0`.
|
| 23 |
+
- `output_confidences` (bool): If `True` and `return_dict_in_generate=True`, returns a per-step confidence tensor alongside generated sequences for debugging/visualization.
|
| 24 |
+
|
| 25 |
+
## Usage
|
| 26 |
+
|
| 27 |
+
To use this custom generation strategy, you can pass it directly to the `generate` method:
|
| 28 |
+
|
| 29 |
+
```python
|
| 30 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 31 |
+
|
| 32 |
+
model = AutoModelForCausalLM.from_pretrained("your-model")
|
| 33 |
+
tokenizer = AutoTokenizer.from_pretrained("your-model")
|
| 34 |
+
|
| 35 |
+
inputs = tokenizer("Hello, world!", return_tensors="pt")
|
| 36 |
+
|
| 37 |
+
# Generate with DeepCONF (Hub repo)
|
| 38 |
+
outputs = model.generate(
|
| 39 |
+
**inputs,
|
| 40 |
+
enable_conf=True,
|
| 41 |
+
window_size=2048,
|
| 42 |
+
threshold=17.0,
|
| 43 |
+
output_confidences=True, # request confidences
|
| 44 |
+
return_dict_in_generate=True, # required to get tensors
|
| 45 |
+
max_new_tokens=100,
|
| 46 |
+
custom_generate="kashif/DeepConf", # Hugging Face Hub repo
|
| 47 |
+
trust_remote_code=True
|
| 48 |
+
)
|
| 49 |
+
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
## Requirements
|
| 53 |
+
|
| 54 |
+
- PyTorch >= 1.13.0
|
| 55 |
+
- Transformers >= 4.35.0
|
| 56 |
+
|