Update README.md
Browse files
README.md
CHANGED
|
@@ -19,7 +19,7 @@ datasets:
|
|
| 19 |
|
| 20 |
# BinaryLLM (Proof of Concept)
|
| 21 |
|
| 22 |
-
Tokenizer-free /
|
| 23 |
|
| 24 |
This repo uses `trust_remote_code=True` because it ships a custom `modeling_*.py` / `configuration_*.py`.
|
| 25 |
|
|
@@ -88,7 +88,7 @@ Avi
|
|
| 88 |
|
| 89 |
This repo includes a minimal inference script that:
|
| 90 |
|
| 91 |
-
* encodes the prompt to
|
| 92 |
* runs a manual token-by-token loop (no `generate`),
|
| 93 |
* decodes the generated bits back to text (best-effort strict decode).
|
| 94 |
|
|
|
|
| 19 |
|
| 20 |
# BinaryLLM (Proof of Concept)
|
| 21 |
|
| 22 |
+
Tokenizer-free / radix-2 (vocab_size=4) proof of concept.
|
| 23 |
|
| 24 |
This repo uses `trust_remote_code=True` because it ships a custom `modeling_*.py` / `configuration_*.py`.
|
| 25 |
|
|
|
|
| 88 |
|
| 89 |
This repo includes a minimal inference script that:
|
| 90 |
|
| 91 |
+
* encodes the prompt to radix-2 bits (UTF-8, MSB→LSB),
|
| 92 |
* runs a manual token-by-token loop (no `generate`),
|
| 93 |
* decodes the generated bits back to text (best-effort strict decode).
|
| 94 |
|