Per-step record (every 500 steps, 40 ckpts) of the 0.6B Qwen3 softmax-attention baseline trained AdamW + WSD on 80B tokens.
Yifei Zuo
YifeiZuo
AI & ML interests
None yet
Recent Activity
authored a paper about 10 hours ago
Parallax: Parameterized Local Linear Attention for Language Modeling upvoted a paper about 15 hours ago
Parallax: Parameterized Local Linear Attention for Language Modeling submitted a paper about 16 hours ago
Parallax: Parameterized Local Linear Attention for Language ModelingOrganizations
None yet
models 40
YifeiZuo/attention-0.6b-adamw-wsd-step20000
0.6B • Updated • 29
YifeiZuo/attention-0.6b-adamw-wsd-step19500
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step19000
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step18500
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step18000
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step17500
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step17000
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step16500
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step16000
0.6B • Updated
YifeiZuo/attention-0.6b-adamw-wsd-step15500
0.6B • Updated
datasets 0
None public yet