| --- |
| license: apache-2.0 |
| --- |
| |
| # CortX-340M-1.3B-2k |
|
|
| A 340M parameter GLA (Gated Linear Attention) model. |
|
|
| ## Model Details |
| - Architecture: GLA |
| - Parameters: 340M |
| - Tokens: 1.3B |
| - Dataset: Fineweb-Edu |
| - Context length: 2k |
| - Training steps: 20480 |
|
|
| ## Usage |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("Zoha4/cortx-340M-1.3B-2k") |
| tokenizer = AutoTokenizer.from_pretrained("Zoha4/cortx-340M-1.3B-2k") |
| ``` |
|
|
|
|
| ## References |
|
|
| 1. Yang et al. *Gated Linear Attention Transformers with Hardware-Efficient Training*. ICML, 2024. [PDF](https://arxiv.org/abs/2312.06635) |
|
|
| 2. Zellers et al. *HellaSwag: Can a Machine Really Finish Your Sentence?* In ACL, 2019. [PDF](https://arxiv.org/abs/1905.07830) |
|
|
| 3. Penedo et al. *The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale*, 2024. [Blog](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1) · [Fineweb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) |
