Hugging Face's logo

Zoha4
/
CortX-340M

Safetensors
gla
Model card Files
xet
CortX-340M / README.md
Zoha4's picture
Zoha4
Update README.md
4767f0c verified
preview code
|
Raw
History Blame
1.01 kB
metadata 
license: apache-2.0

CortX-340M-1.3B-2k

A 340M parameter GLA (Gated Linear Attention) model.

Model Details

  • Architecture: GLA
  • Parameters: 340M
  • Tokens: 1.3B
  • Dataset: Fineweb-Edu
  • Context length: 2k
  • Training steps: 20480

Usage 

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Zoha4/cortx-340M-1.3B-2k")
tokenizer = AutoTokenizer.from_pretrained("Zoha4/cortx-340M-1.3B-2k")

References

  1. Yang et al. Gated Linear Attention Transformers with Hardware-Efficient Training. ICML, 2024. PDF

  2. Zellers et al. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL, 2019. PDF

  3. Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, 2024. Blog · Fineweb-Edu