shawnbro12345 commited on
Commit
ed7c5c0
·
verified ·
1 Parent(s): 6736003

Add T5Gemma SentencePiece tokenizer.model

Browse files

Adds the SentencePiece `tokenizer.model` for the T5Gemma text encoder already published here under `tflite/t5gemma/encoder_fp16.tflite`.

**Why:** the encoder is here but the tokenizer that pairs with it is not, so anything consuming the TFLite text-conditioning path can't tokenise without going to `google/t5gemma-b-b-ul2` directly — which is gated (manual approval) and 401s unauthenticated. That defeats the purpose of re-hosting these weights, which was so users wouldn't have to accept Google's licence on HF themselves.

Concretely this is blocking the Stable Audio plugin's release pipeline: it can build the on-device model installers from this repo except for the tokenizer, which currently has to be copied off a laptop.

**Provenance:** byte-identical to `google/t5gemma-b-b-ul2`'s copy, verified before upload.

```
size: 4241003
sha256: 61a7b147390c64585d6c3543dd6fc636906c9af3865a5548f27f31aee1d4c8e2
```

Unmodified — no conversion, no re-export.

**Licence:** same class of redistribution as the encoder already in this repo. `LICENSE_GEMMA.md` and `NOTICE` are present and cover it, so no licensing changes needed.

Root rather than under `tflite/`, since the tokenizer is shared across the MLX, tensorRT, cpu-amx and TFLite backends — happy to move it if you'd rather it sat elsewhere.

Files changed (1) hide show
  1. tokenizer.model +3 -0
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:61a7b147390c64585d6c3543dd6fc636906c9af3865a5548f27f31aee1d4c8e2
3
+ size 4241003