| language: | |
| - en | |
| license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: | |
| - wiola | |
| - decoder-only | |
| - causal-language-model | |
| - pytorch | |
| - transformers | |
| - research | |
| # Wiola-360M | |
| Wiola-360M is a 360 million parameter decoder-only language model developed by OSCOWL AI. | |
| ## Architecture | |
| Wiola introduces several architectural improvements over a standard Transformer decoder: | |
| - Grouped Query Attention (GQA) | |
| - Spiral Rotary Positional Encoding (SRPE) | |
| - Adaptive Token Merging (ATM) | |
| - Gated Cross Layer Attention (GCLA) | |
| - Dual Stream Feed Forward (DSFF) | |
| ## Model | |
| - Parameters: 360M | |
| - Context Length: 2048 | |
| - Vocabulary Size: 32000 | |
| - Attention Heads: 16 | |
| - Key/Value Heads: 4 | |
| ## Usage | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| tokenizer = AutoTokenizer.from_pretrained( | |
| "oscowlai/Wiola360M", | |
| trust_remote_code=True, | |
| ) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| "oscowlai/Wiola360M", | |
| trust_remote_code=True, | |
| ) | |
| ``` | |
| ## License | |
| Apache-2.0 |