README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- language-model
- transformer
- causal-language-model
- decoder-only
- pytorch
- research
- custom-architecture
- wiola
---

![Black Gradient Minimalist Corporate Business Personal Profile New LinkedIn Banner (1)](https://cdn-uploads.huggingface.co/production/uploads/676a8e90491172b1dad8e390/rGr2WE8JHNN09ESas_tO1.png)

# Wiola

Wiola is a novel decoder-only Small Language Model (SLM) developed by **OSCOWL-AI**. It introduces several architectural improvements over conventional Transformer decoder models, focusing on improving contextual reasoning, computational efficiency, and parameter utilization while remaining compatible with the Hugging Face ecosystem.

> **Note**
>
> Wiola uses a **custom architecture**. Before loading the model, install the Wiola package:
>
> ```bash
> pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git
> ```
>
> or (once available)
>
> ```bash
> pip install wiola
> ```

---

# Model Overview

Wiola is designed as a research-focused decoder-only language model that explores new methods for improving attention, positional encoding, and feed-forward computation.

The architecture introduces five core innovations:

- **Spiral Rotary Positional Encoding (SRPE)** for enhanced positional representation.
- **Gated Cross-Layer Attention (GCLA)** for improved information flow across decoder layers.
- **Adaptive Token Merging (ATM)** for reducing redundant token computation during training.
- **Dual-Stream Feed Forward Network (DSFF)** for richer feature learning.
- **WiolaRMSNorm**, a lightweight normalization technique designed specifically for Wiola.

These components are integrated into a standard autoregressive language modeling framework and are compatible with Hugging Face Transformers.

---

# Installation

Install directly from GitHub:

```bash
pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git
```

or after the PyPI release:

```bash
pip install wiola
```

---

# Quick Start

Create a new Python file (for example, `test.py`) and paste the following code into it:

```python
from wiola13m import WiolaForCausalLM
from transformers import AutoTokenizer

model_path = "oscowlai/Wiola13M"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = WiolaForCausalLM.from_pretrained(model_path)

prompt = "Once upon a time"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    return_token_type_ids=False,
)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_p=0.95
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

Run the script from your terminal:

```bash
python test.py
```

The first time you run the script, the model and tokenizer will be downloaded from Hugging Face and stored in the local cache. This may take a few moments depending on your internet connection. Subsequent runs will use the cached files and start much faster.

## Using your own prompts

To generate text for a different prompt, replace:

```python
prompt = "Once upon a time"
```

with any text you would like the model to continue.

For example:

```python
prompt = "Artificial Intelligence will"
```

```python
prompt = "Write a short story about a robot."
```

```python
prompt = "Explain gravity in simple terms."
```

## Example output

Running the script prints the generated text to your terminal.

Example:

```text
Once upon a time, a simple idea turned into a powerful innovation. It all started with coding ...
```

Since text generation uses sampling (`do_sample=True`), the generated output will be different each time you run the script.

---

# Inputs

The model accepts tokenized text.

Input shape:

```text
(batch_size, sequence_length)
```

Example:

```python
inputs = tokenizer(
    "Hello, how are you?",
    return_tensors="pt"
)
```

---

# Outputs

The model returns:

- Hidden representations
- Next-token logits
- Optional cached key/value states for autoregressive generation

For text generation, use:

```python
outputs = model.generate(...)
```

The generated token IDs can be converted back into readable text using:

```python
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

# Architecture

Wiola is a decoder-only Transformer architecture composed of:

- Token Embedding Layer
- Multiple Wiola Decoder Layers
- Final WiolaRMSNorm
- Language Modeling Head

Each decoder layer consists of:

- Gated Cross-Layer Attention
- Spiral Rotary Positional Encoding
- Adaptive Token Merging (training only)
- Dual-Stream Feed Forward Network
- Residual Connections
- WiolaRMSNorm

---

# Model Characteristics

## Initialization

The released checkpoints are trained from scratch using the Wiola architecture.

## Architecture Type

- Decoder-only Transformer
- Autoregressive Language Model

## Framework

- PyTorch
- Hugging Face Transformers

---

# Training Data

The architecture supports training on standard causal language modeling corpora such as:

- OpenWebText
- BooksCorpus
- FineWeb
- Wikipedia
- Common Crawl derived datasets

The released checkpoints may use different datasets depending on the model version.

Training data is cleaned using standard preprocessing:

- UTF-8 normalization
- Deduplication
- Document concatenation
- Tokenization

---

# Evaluation

The model is designed to be evaluated using common language modeling benchmarks including:

- WikiText
- HellaSwag
- ARC
- PIQA
- MMLU
- GSM8K
- HumanEval (code models)

Benchmark results will be released alongside future checkpoints.

---

# Hardware Requirements

Training:

- NVIDIA GPUs
- CUDA-enabled PyTorch

Inference:

CPU:

- Supported
- Suitable for small checkpoints

GPU:

- Recommended
- Significantly faster generation

---

# Known Limitations

Wiola is a research model and has several limitations:

- May generate incorrect or fabricated information.
- Does not possess factual understanding.
- Performance depends heavily on training data.
- Not intended for safety-critical or medical decision making.
- Should always be used with human verification.

---

# Intended Use

Suitable for:

- Language model research
- NLP experimentation
- Education
- Fine-tuning
- Benchmarking novel Transformer architectures

Not recommended for:

- Medical diagnosis
- Legal advice
- Financial decision making
- Autonomous high-risk systems

---

# Ethical Considerations

Like other large language models, Wiola may inherit biases from training data.

Developers should:

- Review generated outputs.
- Validate factual information.
- Monitor for harmful or biased generations.
- Apply appropriate safety filters in downstream applications.

---
# Novel architecture
```text
                    Input Tokens
                         │
                         ▼
                 Token Embeddings
                         │
                         ▼
        ┌─────────────────────────────────────┐
        │      Spiral Rotary Encoding      │
        │   Gated Cross-Layer Attention    │
        │     Adaptive Token Merging       │
        │        Dual-Stream FFN           │
        │          WiolaRMSNorm            │
        └─────────────────────────────────────┘
                         │
                         ▼
                 Repeated N Layers
                         │
                         ▼
                   Final RMSNorm
                         │
                         ▼
                       LM Head
                         │
                         ▼
                 Generated Tokens
```

---

# Citation

If you use Wiola in your research, please cite:

```bibtex
@software{wiola2026,
  title={Wiola: A Novel Decoder-Only Small Language Model},
  author={OSCOWL-AI},
  year={2026},
  url={https://github.com/Wiola-OSCOWL-ai/wiola}
}
```

---

# License

Apache License 2.0

---

# Links

- GitHub: https://github.com/Wiola-OSCOWL-ai/Wiola13M
- PyPI: https://pypi.org/project/wiola/ *(coming soon)*
- Hugging Face: https://huggingface.co/OSCOWL-AI

---

# Acknowledgements

Wiola is an open research project developed by **OSCOWL-AI** to explore efficient and scalable Small Language Model architectures. Contributions, feedback, and research collaborations are welcome.