README: link the standalone code repo (github.com/AbstractEyes/geolip-bytelex) + git+ install line; regeneration section now points at the library, not alephllm
Browse files
README.md
CHANGED
|
@@ -9,6 +9,11 @@ tags:
|
|
| 9 |
|
| 10 |
# geolip-bytelex — the universal byte-information relational system
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
A **model-free translation matrix between tokenizer languages and byte
|
| 13 |
language**. Foreign tokens are not the same language as bytes — but any
|
| 14 |
real token *is* a byte string underneath, and the statistics of byte
|
|
@@ -76,8 +81,9 @@ the same matrix serves every teacher and every ByteLM student.
|
|
| 76 |
|
| 77 |
## Regeneration
|
| 78 |
|
| 79 |
-
The consuming/producing code is
|
| 80 |
-
[
|
|
|
|
| 81 |
build a `ByteLexicon(GramSchema(...))` over any corpus stream, `feed`
|
| 82 |
raw bytes, `save`; project any `vocab_*.jsonl` through
|
| 83 |
`ByteLexicon.profile`. New alphabet, new corpus, new gram views, new
|
|
|
|
| 9 |
|
| 10 |
# geolip-bytelex — the universal byte-information relational system
|
| 11 |
|
| 12 |
+
**Code:** [github.com/AbstractEyes/geolip-bytelex](https://github.com/AbstractEyes/geolip-bytelex) · install:
|
| 13 |
+
```bash
|
| 14 |
+
pip install "geolip-bytelex @ git+https://github.com/AbstractEyes/geolip-bytelex"
|
| 15 |
+
```
|
| 16 |
+
|
| 17 |
A **model-free translation matrix between tokenizer languages and byte
|
| 18 |
language**. Foreign tokens are not the same language as bytes — but any
|
| 19 |
real token *is* a byte string underneath, and the statistics of byte
|
|
|
|
| 81 |
|
| 82 |
## Regeneration
|
| 83 |
|
| 84 |
+
The consuming/producing code is the standalone
|
| 85 |
+
[geolip-bytelex library](https://github.com/AbstractEyes/geolip-bytelex)
|
| 86 |
+
(pure-stdlib core, shared geolip namespace, 18-test suite):
|
| 87 |
build a `ByteLexicon(GramSchema(...))` over any corpus stream, `feed`
|
| 88 |
raw bytes, `save`; project any `vocab_*.jsonl` through
|
| 89 |
`ByteLexicon.profile`. New alphabet, new corpus, new gram views, new
|