Serdar404 commited on
Commit
bef6a07
·
verified ·
1 Parent(s): 0c66a68

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -4
README.md CHANGED
@@ -11,11 +11,11 @@ tags:
11
  - custom_code
12
  ---
13
 
14
- # BabyLM Challenge 2026 submission | RecGPT-100M-Fixed
15
 
16
- RecGPT-100M-Fixed is a 124.03M-parameter recursive causal language model trained for the BabyLM 2026 Strict track. It was trained for 10 epochs on a custom 100M-word English corpus using a 32,768-token BPE vocabulary.
17
 
18
- The model applies a shared Transformer block recursively for 24 iterations. Its hidden size is 1,408, embedding size is 768, and feed-forward intermediate size is 22,528. Training used Muon for the recursive block and AdamW for the embedding-related parameters, with a token batch size of 32,768 and sequence length 512.
19
 
20
  ## Usage
21
 
@@ -44,7 +44,6 @@ Final-checkpoint results before leaderboard submission:
44
  | COMPS | 60.55 |
45
  | GlobalPIQA | 43.15 |
46
  | (Super)GLUE | 71.84 |
47
- | NLP Average | 57.63 |
48
 
49
  Intermediate Strict checkpoints are published as Hub revisions named `chck_1M` through `chck_1000M` using the official BabyLM checkpoint schedule.
50
 
 
11
  - custom_code
12
  ---
13
 
14
+ # BabyLM Challenge 2026 submission | RecGPT-100M
15
 
16
+ RecGPT-100M is a 124.03M-parameter recursive causal language model trained for the BabyLM 2026 Strict track. It was trained for 10 epochs on a custom 100M-word English corpus using a 32,768-token BPE vocabulary.
17
 
18
+ The model applies a shared Transformer block recursively for 24 iterations. Its hidden size is 1,408, embedding size is 768, and feed-forward intermediate size is 22,528. Training used Aurora for the recursive block and AdamW for the embedding-related parameters, with a token batch size of 32,768 and sequence length 512.
19
 
20
  ## Usage
21
 
 
44
  | COMPS | 60.55 |
45
  | GlobalPIQA | 43.15 |
46
  | (Super)GLUE | 71.84 |
 
47
 
48
  Intermediate Strict checkpoints are published as Hub revisions named `chck_1M` through `chck_1000M` using the official BabyLM checkpoint schedule.
49