Update README.md
Browse files
README.md
CHANGED
|
@@ -30,10 +30,10 @@ python filter_k5.py --in shard.parquet --out kept.parquet
|
|
| 30 |
|
| 31 |
| Path | Size | Used by | What it is |
|
| 32 |
|------|------|---------|------------|
|
| 33 |
-
| `data/aoa.parquet` | 0.5 MB | Stage 1 | Age-of-Acquisition norms
|
| 34 |
-
| `data/word_log_odds.parquet` | 1.9 MB | Stage 5 | Beyond-K–5 association scores
|
| 35 |
-
| `models/fasttext_grade.bin` | 57 MB | Stage 2 | fastText classifier
|
| 36 |
-
| `models/modernbert_grade/` | 299 MB | Stage 3 | ModernBERT classifier
|
| 37 |
|
| 38 |
Both classifiers predict one of `K5`, `K8`, `K12`, `OOS`
|
| 39 |
(`id2label = {0: "K5", 1: "K8", 2: "K12", 3: "OOS"}`); the filter retains
|
|
@@ -48,7 +48,7 @@ annotation of FineWeb-Edu would have been prohibitively expensive, which is
|
|
| 48 |
what motivates the cascaded design: a cheap lexical stage, then fastText, then
|
| 49 |
the ~50× more expensive ModernBERT.
|
| 50 |
|
| 51 |
-
##
|
| 52 |
|
| 53 |
**Trained for web prose.** Labels come from FineWeb-Edu documents. On
|
| 54 |
substantially different distributions — conversational text, worked solutions,
|
|
|
|
| 30 |
|
| 31 |
| Path | Size | Used by | What it is |
|
| 32 |
|------|------|---------|------------|
|
| 33 |
+
| `data/aoa.parquet` | 0.5 MB | Stage 1 | Age-of-Acquisition norms|
|
| 34 |
+
| `data/word_log_odds.parquet` | 1.9 MB | Stage 5 | Beyond-K–5 association scores |
|
| 35 |
+
| `models/fasttext_grade.bin` | 57 MB | Stage 2 | fastText classifier |
|
| 36 |
+
| `models/modernbert_grade/` | 299 MB | Stage 3 | ModernBERT classifier |
|
| 37 |
|
| 38 |
Both classifiers predict one of `K5`, `K8`, `K12`, `OOS`
|
| 39 |
(`id2label = {0: "K5", 1: "K8", 2: "K12", 3: "OOS"}`); the filter retains
|
|
|
|
| 48 |
what motivates the cascaded design: a cheap lexical stage, then fastText, then
|
| 49 |
the ~50× more expensive ModernBERT.
|
| 50 |
|
| 51 |
+
## Intended Usage
|
| 52 |
|
| 53 |
**Trained for web prose.** Labels come from FineWeb-Edu documents. On
|
| 54 |
substantially different distributions — conversational text, worked solutions,
|