Atto

atto

Atto is a family of compact contextual models designed for parameter-efficient language representation learning.

These models can be loaded using Hugging Face Transformers. Since Atto uses a custom architecture, trust_remote_code=True is required.

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "neuml/atto-50k-768d",
    trust_remote_code=True
)

Training Parameters

Parameter Value
learning_rate 0.001
gradient_accumulation_steps 1
fp16 True

More Information

See the Introducing Atto article for benchmark results, architecture details, and training methodology.

Downloads last month
26
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NeuML/atto-50k-768d

Finetunes
1 model

Collection including NeuML/atto-50k-768d

Article mentioning NeuML/atto-50k-768d