AbstractPhil commited on
Commit
b4e42d9
·
verified ·
1 Parent(s): 3d25c99

README: link the standalone code repo (github.com/AbstractEyes/geolip-bytelex) + git+ install line; regeneration section now points at the library, not alephllm

Browse files
Files changed (1) hide show
  1. README.md +8 -2
README.md CHANGED
@@ -9,6 +9,11 @@ tags:
9
 
10
  # geolip-bytelex — the universal byte-information relational system
11
 
 
 
 
 
 
12
  A **model-free translation matrix between tokenizer languages and byte
13
  language**. Foreign tokens are not the same language as bytes — but any
14
  real token *is* a byte string underneath, and the statistics of byte
@@ -76,8 +81,9 @@ the same matrix serves every teacher and every ByteLM student.
76
 
77
  ## Regeneration
78
 
79
- The consuming/producing code is `geolip.bytelex` in
80
- [AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm):
 
81
  build a `ByteLexicon(GramSchema(...))` over any corpus stream, `feed`
82
  raw bytes, `save`; project any `vocab_*.jsonl` through
83
  `ByteLexicon.profile`. New alphabet, new corpus, new gram views, new
 
9
 
10
  # geolip-bytelex — the universal byte-information relational system
11
 
12
+ **Code:** [github.com/AbstractEyes/geolip-bytelex](https://github.com/AbstractEyes/geolip-bytelex) · install:
13
+ ```bash
14
+ pip install "geolip-bytelex @ git+https://github.com/AbstractEyes/geolip-bytelex"
15
+ ```
16
+
17
  A **model-free translation matrix between tokenizer languages and byte
18
  language**. Foreign tokens are not the same language as bytes — but any
19
  real token *is* a byte string underneath, and the statistics of byte
 
81
 
82
  ## Regeneration
83
 
84
+ The consuming/producing code is the standalone
85
+ [geolip-bytelex library](https://github.com/AbstractEyes/geolip-bytelex)
86
+ (pure-stdlib core, shared geolip namespace, 18-test suite):
87
  build a `ByteLexicon(GramSchema(...))` over any corpus stream, `feed`
88
  raw bytes, `save`; project any `vocab_*.jsonl` through
89
  `ByteLexicon.profile`. New alphabet, new corpus, new gram views, new