README

---
library_name: transformers
tags:
  - custom_generate
---

# LettucePrevent — custom_generate

Real-time prevention of hallucinations in RAG. A token-level hallucination
detector is hooked into decoding through a `LogitsProcessor`: at each step the
top-k candidate tokens are scored and the ones predicted to introduce a
hallucination are penalized *before* the next token is selected.

Three detectors ship in this repo:

- `number` — rule-based, regex-verifiable. Rejects numbers not present in the
  input text. Deterministic, hard-guarantee, no extra model download.
- `lettuceprevent` — a fine-tuned 68M Ettin decoder
  (`lebe1/lettuceprevent-ettin-decoder-68m-en`) that flags unsupported tokens
  for factual claims.
- `lettucedetect` — model families based on encoder architectures to evaluate hallucinations
  after generation such as `KRLabsOrg/tinylettuce-ettin-68m-en` or `KRLabsOrg/lettucedect-base-modernbert-en-v1`

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

gen_id = "meta-llama/Llama-3.3-70B-Instruct"
tok    = AutoTokenizer.from_pretrained(gen_id)
model  = AutoModelForCausalLM.from_pretrained(gen_id, device_map="auto")

context = "Revenue was 2400 million in 2021 and 3100 million in 2022."
inputs  = tok(context, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    custom_generate="lebe1/lettuceprevent-generate",
    trust_remote_code=True,
    tokenizer=tok,
    input_text=context,
    detector_type="lettuceprevent",   # or "number"
    skip_threshold=0.99,
    max_new_tokens=300,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Additional arguments

| Argument | Default | Description |
| --- | --- | --- |
| `tokenizer` | required | The generator tokenizer (decodes candidates for the detector). |
| `input_text` | required | The grounding context; the detector validates against it. |
| `detector_type` | `"lettuceprevent"` | `"number"`, `"lettuceprevent"`, or a `baseline-*` (unmodified). |
| `confidence_threshold` | `0.9` | Hallucination-probability threshold (lettuceprevent). |
| `model_path` | detector default | Override the HDM checkpoint. |
| `skip_threshold` | `1.0` | Skip the HDM check when top-token prob exceeds this (`1.0` = never skip). |
| `top_k_logits` | `10` | Candidate tokens scored per step. |
| `penalty_value` | `0.0` | Score assigned to penalized tokens (`float('-inf')` to hard-block). |

## Output

Same as `generate()` — a tensor of token ids. No output-type changes.

## Notes

Use this via the `custom_generate=` argument on your own generator model.
Requires a CUDA-capable GPU for the `lettuceprevent` detector.