Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Harley-ml 
posted an update 8 days ago
Post
90
Zero-v1.0-144M is currently training. So far, it has completed 26% of its training run, with about 98 billion tokens to go.

Current Val PPL: 13.705

Zero-v1.0 uses a custom architecture consisting of RMSNorm, RoPe, SwiGLU, Engram Conditional Memory, mHC, and XSA GQA.
In this post