YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
A 2L, width 736 SoLU model trained on 15B tokens of the Pile. Bugs: the layernorm just before the unembed is an RMS norm, and the width is not a multiple of 64, so d_head=64 and n_heads=11, and n_heads * d_head != d_model :(
- Downloads last month
- 195
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("NeelNanda/SoLU_2L_v10_old", dtype="auto")