ffli commited on
Commit
ad1cd80
·
verified ·
1 Parent(s): 7b154db

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -30,10 +30,10 @@ python filter_k5.py --in shard.parquet --out kept.parquet
30
 
31
  | Path | Size | Used by | What it is |
32
  |------|------|---------|------------|
33
- | `data/aoa.parquet` | 0.5 MB | Stage 1 | Age-of-Acquisition norms, 51,694 words |
34
- | `data/word_log_odds.parquet` | 1.9 MB | Stage 5 | Beyond-K–5 association scores, 129,719 words. `beyond_score` is `log2(rate_beyond / rate_k5)`; positive means characteristic of Beyond-K–5 material. |
35
- | `models/fasttext_grade.bin` | 57 MB | Stage 2 | fastText classifier, 4 classes |
36
- | `models/modernbert_grade/` | 299 MB | Stage 3 | ModernBERT classifier, 149.6M params, 4 classes |
37
 
38
  Both classifiers predict one of `K5`, `K8`, `K12`, `OOS`
39
  (`id2label = {0: "K5", 1: "K8", 2: "K12", 3: "OOS"}`); the filter retains
@@ -48,7 +48,7 @@ annotation of FineWeb-Edu would have been prohibitively expensive, which is
48
  what motivates the cascaded design: a cheap lexical stage, then fastText, then
49
  the ~50× more expensive ModernBERT.
50
 
51
- ## Limitations
52
 
53
  **Trained for web prose.** Labels come from FineWeb-Edu documents. On
54
  substantially different distributions — conversational text, worked solutions,
 
30
 
31
  | Path | Size | Used by | What it is |
32
  |------|------|---------|------------|
33
+ | `data/aoa.parquet` | 0.5 MB | Stage 1 | Age-of-Acquisition norms|
34
+ | `data/word_log_odds.parquet` | 1.9 MB | Stage 5 | Beyond-K–5 association scores |
35
+ | `models/fasttext_grade.bin` | 57 MB | Stage 2 | fastText classifier |
36
+ | `models/modernbert_grade/` | 299 MB | Stage 3 | ModernBERT classifier |
37
 
38
  Both classifiers predict one of `K5`, `K8`, `K12`, `OOS`
39
  (`id2label = {0: "K5", 1: "K8", 2: "K12", 3: "OOS"}`); the filter retains
 
48
  what motivates the cascaded design: a cheap lexical stage, then fastText, then
49
  the ~50× more expensive ModernBERT.
50
 
51
+ ## Intended Usage
52
 
53
  **Trained for web prose.** Labels come from FineWeb-Edu documents. On
54
  substantially different distributions — conversational text, worked solutions,