pragnyanramtha/tokenizer-json-regex-dos-poc

Hugging Face Tokenizer JSON Regex DoS PoC

This is a benign proof of concept for an artifact-carried denial of service in tokenizer preprocessing.

The included tokenizer.json contains a Replace normalizer with the regex (a+)+$. Safe input tokenizes normally. Trigger input made of repeated a characters followed by ! causes current Python and JavaScript tokenizer runtimes to fail or hang in different ways:

• Python transformers==5.8.0 with tokenizers==0.22.2 raises pyo3_runtime.PanicException: Onig: Regex search error: retry-limit-in-match over.
• Standalone tokenizers==0.23.1 was also locally verified to raise the same PanicException.
• Current @huggingface/transformers==4.2.0 times out in tokenizer preprocessing for short trigger strings such as a{24}!.

This PoC does not execute code, access files, or make network callbacks.

Reproduction: Python

python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python verify_python_tokenizer_panic.py

Expected result includes:

{
  "exception_type": "PanicException",
  "message": "Onig: Regex search error: retry-limit-in-match over",
  "is_exception_subclass": false
}

Reproduction: Transformers.js

npm install
npm run verify:js

The JavaScript verifier loads the same tokenizer files from tokenizer_artifact/ so Node package metadata in the repository root does not affect local tokenizer loading.

Expected result includes:

{
  "safeCaseOk": true,
  "triggerTimedOut": true
}

Impact

An attacker-controlled tokenizer artifact can make tokenizer preprocessing unavailable for trigger inputs. In applications that load untrusted tokenizer artifacts and process user input through them, this can produce an inference-preprocessing denial of service. The Python panic is not an Exception subclass, so normal except Exception handlers do not catch it.

Severity estimate: Medium. This is DoS only, not arbitrary code execution or data disclosure.

Hashes

f60ef96afb2ee2395beea0fec06ecb235668c7e9bc44ff7bf1672d1b23e62c1a  tokenizer.json
d983072f0f98ea839304df73ac6e631fe64531e7511a760035f08d345c7638ac  tokenizer_config.json
397a46ba357683730e76c164f010d4dc4776298172f7e325ab9ae420f8b0e057  special_tokens_map.json

Prior local outputs are included in evidence/.

ModelScan 0.8.8 reports zero issues and skips the tokenizer JSON files as unsupported; those outputs are included as supporting evidence only.