Update model card with v2 training results (98.87% type accuracy)
Browse files
README.md
CHANGED
|
@@ -38,15 +38,44 @@ outputs = model.generate(**inputs, max_length=128)
|
|
| 38 |
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
| 39 |
```
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
## Training Data
|
| 42 |
|
| 43 |
Derived from STEPBible TAGNT (Translators Amalgamated Greek NT), which marks each NT word
|
| 44 |
with its presence across 6 critical editions: NA27/28, Textus Receptus, SBLGNT, Byzantine,
|
| 45 |
Westcott-Hort, and THGNT.
|
| 46 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
## Part of bible.systems
|
| 48 |
|
| 49 |
-
This is model 4 of
|
| 50 |
|
| 51 |
---
|
| 52 |
*For God so loved the world...* — John 3:16
|
|
|
|
| 38 |
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
| 39 |
```
|
| 40 |
|
| 41 |
+
## Results (v2)
|
| 42 |
+
|
| 43 |
+
Trained on Apple M4 Pro (MPS), mT5-small (300M params), 9 epochs:
|
| 44 |
+
|
| 45 |
+
| Metric | Score |
|
| 46 |
+
|---|---|
|
| 47 |
+
| Eval loss (best) | **0.0868** (epoch 9) |
|
| 48 |
+
| Variant type accuracy | **98.87%** |
|
| 49 |
+
| Exact match (full output) | 37.5% |
|
| 50 |
+
| Training steps | 9,558 |
|
| 51 |
+
|
| 52 |
+
### Per-Epoch Progression
|
| 53 |
+
|
| 54 |
+
| Epoch | Loss | Exact Match | Type Accuracy |
|
| 55 |
+
|-------|------|-------------|---------------|
|
| 56 |
+
| 1 | 0.8033 | 0% | 69.3% |
|
| 57 |
+
| 2 | 0.2577 | 1.0% | 67.5% |
|
| 58 |
+
| 3 | 0.1465 | 12.1% | 94.5% |
|
| 59 |
+
| 4 | 0.1159 | 25.9% | 96.0% |
|
| 60 |
+
| 5 | 0.1039 | 29.3% | 98.3% |
|
| 61 |
+
| 6 | 0.0963 | 32.7% | 98.8% |
|
| 62 |
+
| 7 | 0.0914 | 34.7% | 97.8% |
|
| 63 |
+
| 8 | 0.0888 | 35.6% | 98.8% |
|
| 64 |
+
| 9 | 0.0868 | **37.5%** | **98.9%** |
|
| 65 |
+
|
| 66 |
## Training Data
|
| 67 |
|
| 68 |
Derived from STEPBible TAGNT (Translators Amalgamated Greek NT), which marks each NT word
|
| 69 |
with its presence across 6 critical editions: NA27/28, Textus Receptus, SBLGNT, Byzantine,
|
| 70 |
Westcott-Hort, and THGNT.
|
| 71 |
|
| 72 |
+
- Training set: 8,497 examples
|
| 73 |
+
- Validation set: 1,062 examples
|
| 74 |
+
- Test set: 1,062 examples
|
| 75 |
+
|
| 76 |
## Part of bible.systems
|
| 77 |
|
| 78 |
+
This is model 4 of 8 in the [bible.systems](https://bible.systems) ML pipeline.
|
| 79 |
|
| 80 |
---
|
| 81 |
*For God so loved the world...* — John 3:16
|