README.md

# tokenizers-decoder-panic-poc

**PoC for:** Denial of Service via Rust Panic in HuggingFace tokenizers Decoder Deserialization

**Impact:** Loading this `tokenizer.json` file causes an unrecoverable Rust panic (SIGABRT) at `decoders/mod.rs:90`, crashing any application that uses `huggingface/tokenizers`.

**File:** `tokenizer.json` — contains malformed `decoder` field that triggers `.expect("Helper")` panic.

**Affected versions:** 0.1.0 through latest (0.23.2-dev.0)

**Root cause:** `decoders/mod.rs` line 90 uses `.expect("Helper")` instead of proper `?` error handling.

**PoC:**
```python
from tokenizers import Tokenizer
Tokenizer.from_str('{"decoder":[}}}')
# Result: thread 'main' panicked at tokenizers/src/decoders/mod.rs:90:63:
# "Helper: Error(\"expected value\", line: 1, column: 13)"
```