Text Generation
PyTorch
Safetensors
Polish
English
polish
causal-lm
from-scratch
Maggio33 commited on
Commit
3add7eb
·
verified ·
1 Parent(s): ed13d53

Fix the byte perplexity range sentence (from step 21000 on)

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -100,7 +100,7 @@ lm-evaluation-harness benchmark used in the Results table of 1.00–1.03.
100
  | **22327 (final)** | **77.90** | **58.67** | **2.1837** | **24.92** | **98.84** | **46.93** | CPU |
101
 
102
  ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
103
- checkpoints are within noise. From 20000 on, WikiText-2 byte perplexity moves within about ±0.005 around 2.18 instead of
104
  falling steadily (lowest 2.1794 at 22000; 2.1837 at the final step): the run used a constant learning rate without decay,
105
  so the last checkpoint is where training stopped, not a converged minimum. The final checkpoint has the highest English
106
  ARC-Easy accuracy of the run (58.67, +2.0 points over 19000). Each point is a single measurement. The training loss in the figure is measured
 
100
  | **22327 (final)** | **77.90** | **58.67** | **2.1837** | **24.92** | **98.84** | **46.93** | CPU |
101
 
102
  ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
103
+ checkpoints are within noise. From 21000 on, WikiText-2 byte perplexity stays between 2.179 and 2.187 instead of
104
  falling steadily (lowest 2.1794 at 22000; 2.1837 at the final step): the run used a constant learning rate without decay,
105
  so the last checkpoint is where training stopped, not a converged minimum. The final checkpoint has the highest English
106
  ARC-Easy accuracy of the run (58.67, +2.0 points over 19000). Each point is a single measurement. The training loss in the figure is measured