EleutherAI/pile
Updated • 3.09k • 495
How to use hazyresearch/attn-360M-30B with Transformers:
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("hazyresearch/attn-360M-30B", dtype="auto")Configuration Parsing Warning:In config.json: "architectures" must be an array
This model is an Attention (Llama architecture) model pretrained on 30Bn tokens of the Pile corpus.
The model implementation and training code that produced the model are provided here: https://github.com/HazyResearch/based