Wiola-120M
🦉 Wiola-120M
A Novel Small Language Model (SLM) introducing Spiral Rotary Positional Encoding (SRPE), Gated Cross-Layer Attention (GCLA), Adaptive Token Merging (ATM), Dual-Stream Feed-Forward (DSFF), and WiolaRMSNorm.
Developed by OSCOWL-AI
Overview
Wiola-120M is an experimental decoder-only Small Language Model developed from first principles as part of the Wiola research project.
Unlike conventional transformer architectures, Wiola introduces several novel architectural components designed to improve parameter efficiency, contextual reasoning, and computational efficiency while maintaining compatibility with the Hugging Face ecosystem.
This repository contains the trained model weights.
The implementation of the architecture is provided separately through the Wiola Python package.
Architecture
Wiola introduces five core innovations:
Spiral Rotary Positional Encoding (SRPE)
A rotary positional encoding variant that augments angular rotations with spiral radial modulation for richer positional representations.
Gated Cross-Layer Attention (GCLA)
Allows each decoder layer to selectively attend to compressed contextual representations from previous layers using learnable gates.
Adaptive Token Merging (ATM)
During training, semantically redundant neighbouring tokens are dynamically merged to reduce computation while preserving information.
ATM is automatically disabled during inference.
Dual-Stream Feed Forward Network (DSFF)
Each decoder block contains two parallel feed-forward streams:
- Narrow stream
- Wide stream
whose outputs are fused to improve representation learning.
WiolaRMSNorm
A lightweight RMSNorm implementation used throughout the architecture.
Model Specifications
| Property | Value |
|---|---|
| Parameters | ~122 Million |
| Hidden Size | 768 |
| Decoder Layers | 12 |
| Attention Heads | 12 |
| Key/Value Heads | 4 |
| Vocabulary Size | 32,000 |
| Context Length | 2048 Tokens |
| Architecture | Decoder-only Transformer |
| Precision | BF16 / FP16 / FP32 |
Training
This checkpoint was trained using the official Wiola training pipeline.
Dataset
- WikiText-103
Optimizer
- AdamW
Learning Rate Scheduler
- Cosine Decay
- Linear Warmup
Mixed Precision
- BF16 / FP16 supported
Evaluation
Final validation metrics obtained during training:
| Metric | Value |
|---|---|
| Validation Loss | 4.1532 |
| Validation Perplexity | 63.64 |
These values correspond to the released checkpoint and should be considered baseline research results.
Installation
Install the Wiola framework first.
pip install wiola
or install the latest development version:
pip install git+https://github.com/Wiola-OSCOWL-ai/wiola.git
Usage
from transformers import AutoTokenizer
from wiola import WiolaForCausalLM
repo = "oscowlai/wiola-120m"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = WiolaForCausalLM.from_pretrained(repo)
prompt = "Artificial Intelligence is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=100,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Repository
GitHub
https://github.com/Wiola-OSCOWL-ai/wiola
Intended Uses
Wiola is intended for:
- Language model research
- Architecture experimentation
- Fine-tuning research
- Educational purposes
- Small Language Model benchmarking
Limitations
This checkpoint is released primarily as a research model.
Current limitations include:
- Trained on a relatively small corpus compared to production LLMs.
- Not instruction tuned.
- No RLHF or preference optimization.
- May generate inaccurate or fabricated information.
- Should not be used for high-risk or safety-critical applications.
Citation
@software{wiola2026,
title={WiolaSLM A Novel Small Language Model with Spiral Rotary Positional Encoding, Gated Cross-Layer Attention, Adaptive Token Merging and Dual-Stream Feed Forward Networks},
author={OSCOWL-AI},
year={2026},
url={https://github.com/Wiola-OSCOWL-ai/wiola}
}
License
Apache License 2.0
Acknowledgements
Wiola is an independent research effort by OSCOWL-AI focused on advancing efficient Small Language Models through novel transformer architectures.
Contributions, issues, and feedback are welcome through the official GitHub repository.
- Downloads last month
- -