ggunio
/

intelligent-tokenizer-v6

intelligent_tokenizer

neural-tokenizer

pattern-learning

vocabulary-free

Eval Results (legacy)

Model card Files Files and versions

intelligent-tokenizer-v6

Ctrl+K

Ctrl+K

1 contributor

History: 19 commits

ggunio's picture

Update README.md

a1634b9 verified 8 months ago

core
Upload core/unified_model.py with huggingface_hub 8 months ago
src
Upload src/core/byte_tokenizer_v6.py with huggingface_hub 8 months ago
.gitattributes

1.52 kB
initial commit 8 months ago
README.md

2.87 kB
Update README.md 8 months ago
app.py

13.9 kB
Upload app.py with huggingface_hub 8 months ago
config.json

394 Bytes
Upload config.json with huggingface_hub 8 months ago
demo_poc.py

9.88 kB
Upload demo_poc.py with huggingface_hub 8 months ago
inference.py

9.67 kB
Upload inference.py with huggingface_hub 8 months ago
pytorch_model.bin
Detected Pickle imports (3)
- "torch.FloatStorage",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict"
What is a pickle import?
425 MB
xet

Upload pytorch_model.bin with huggingface_hub 8 months ago
requirements.txt

34 Bytes
Upload requirements.txt with huggingface_hub 8 months ago