README.md

metadata

title: Exllamav2 Patch
sdk: static
license: mit
short_description: Patch for ExLlamaV2 to support Unigram-based tokenizers

Unigram Tokenizer Patch for ExLlamaV2

This repository provides a patch to support Unigram-based tokenizers with ExLlamaV2. By default, using a Unigram tokenizer may result in an error during inference. This patch resolves that issue.

Files

• hf.py.patch A patch file that modifies hf.py in the ExLlamaV2 repository to support Unigram tokenizers.

• apply_patch.sh A shell script to apply the patch.

Usage

You can apply the patch with the following command:

$ ./apply_patch.sh EXLLAMAV2_DIR

Replace EXLLAMAV2_DIR with the path to your local clone of ExLlamaV2.

Example:

$ ./apply_patch.sh ~/repos/exllamav2

Purpose

The patch is intended to:

• Prevent runtime errors when using tokenizers.models.Unigram.
• Add fallback handling for missing unk_token attributes.
• Improve compatibility with models trained using SentencePiece + Unigram tokenizers.

Notes

• This patch is intended to be applied to the hf.py file in the ExLlamaV2 repository.
• Tested Versions: v0.2.6 through v0.2.8
• If the patch fails, please ensure your copy of hf.py matches the original before applying.