Wiola13M / README.md

Update README.md

f1cf3ba verified 4 days ago

8.49 kB

license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - language-model
  - transformer
  - causal-language-model
  - decoder-only
  - pytorch
  - research
  - custom-architecture
  - wiola

Wiola

Wiola is a novel decoder-only Small Language Model (SLM) developed by OSCOWL-AI. It introduces several architectural improvements over conventional Transformer decoder models, focusing on improving contextual reasoning, computational efficiency, and parameter utilization while remaining compatible with the Hugging Face ecosystem.

Note

Wiola uses a custom architecture. Before loading the model, install the Wiola package:
pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git
or (once available)
pip install wiola

Model Overview

Wiola is designed as a research-focused decoder-only language model that explores new methods for improving attention, positional encoding, and feed-forward computation.

The architecture introduces five core innovations:

Spiral Rotary Positional Encoding (SRPE) for enhanced positional representation.
Gated Cross-Layer Attention (GCLA) for improved information flow across decoder layers.
Adaptive Token Merging (ATM) for reducing redundant token computation during training.
Dual-Stream Feed Forward Network (DSFF) for richer feature learning.
WiolaRMSNorm, a lightweight normalization technique designed specifically for Wiola.

These components are integrated into a standard autoregressive language modeling framework and are compatible with Hugging Face Transformers.

Installation

Install directly from GitHub:

pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git

or after the PyPI release:

pip install wiola

Quick Start

Create a new Python file (for example, test.py) and paste the following code into it:

from wiola13m import WiolaForCausalLM
from transformers import AutoTokenizer

model_path = "oscowlai/Wiola13M"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = WiolaForCausalLM.from_pretrained(model_path)

prompt = "Once upon a time"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    return_token_type_ids=False,
)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_p=0.95
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Run the script from your terminal:

python test.py

The first time you run the script, the model and tokenizer will be downloaded from Hugging Face and stored in the local cache. This may take a few moments depending on your internet connection. Subsequent runs will use the cached files and start much faster.

Using your own prompts

To generate text for a different prompt, replace:

prompt = "Once upon a time"

with any text you would like the model to continue.

For example:

prompt = "Artificial Intelligence will"

prompt = "Write a short story about a robot."

prompt = "Explain gravity in simple terms."

Example output

Running the script prints the generated text to your terminal.

Example:

Once upon a time, a simple idea turned into a powerful innovation. It all started with coding ...

Since text generation uses sampling (do_sample=True), the generated output will be different each time you run the script.

Inputs

The model accepts tokenized text.

Input shape:

(batch_size, sequence_length)

Example:

inputs = tokenizer(
    "Hello, how are you?",
    return_tensors="pt"
)

Outputs

The model returns:

Hidden representations
Next-token logits
Optional cached key/value states for autoregressive generation

For text generation, use:

outputs = model.generate(...)

The generated token IDs can be converted back into readable text using:

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Architecture

Wiola is a decoder-only Transformer architecture composed of:

Token Embedding Layer
Multiple Wiola Decoder Layers
Final WiolaRMSNorm
Language Modeling Head

Each decoder layer consists of:

Gated Cross-Layer Attention
Spiral Rotary Positional Encoding
Adaptive Token Merging (training only)
Dual-Stream Feed Forward Network
Residual Connections
WiolaRMSNorm

Model Characteristics

Initialization

The released checkpoints are trained from scratch using the Wiola architecture.

Architecture Type

Decoder-only Transformer
Autoregressive Language Model

Framework

PyTorch
Hugging Face Transformers

Training Data

The architecture supports training on standard causal language modeling corpora such as:

OpenWebText
BooksCorpus
FineWeb
Wikipedia
Common Crawl derived datasets

The released checkpoints may use different datasets depending on the model version.

Training data is cleaned using standard preprocessing:

UTF-8 normalization
Deduplication
Document concatenation
Tokenization

Evaluation

The model is designed to be evaluated using common language modeling benchmarks including:

WikiText
HellaSwag
ARC
PIQA
MMLU
GSM8K
HumanEval (code models)

Benchmark results will be released alongside future checkpoints.

Hardware Requirements

Training:

NVIDIA GPUs
CUDA-enabled PyTorch

Inference:

CPU:

Supported
Suitable for small checkpoints

GPU:

Recommended
Significantly faster generation

Known Limitations

Wiola is a research model and has several limitations:

May generate incorrect or fabricated information.
Does not possess factual understanding.
Performance depends heavily on training data.
Not intended for safety-critical or medical decision making.
Should always be used with human verification.

Intended Use

Suitable for:

Language model research
NLP experimentation
Education
Fine-tuning
Benchmarking novel Transformer architectures

Not recommended for:

Medical diagnosis
Legal advice
Financial decision making
Autonomous high-risk systems

Ethical Considerations

Like other large language models, Wiola may inherit biases from training data.

Developers should:

Review generated outputs.
Validate factual information.
Monitor for harmful or biased generations.
Apply appropriate safety filters in downstream applications.

Novel architecture

                    Input Tokens
                         │
                         ▼
                 Token Embeddings
                         │
                         ▼
        ┌─────────────────────────────────────┐
        │      Spiral Rotary Encoding      │
        │   Gated Cross-Layer Attention    │
        │     Adaptive Token Merging       │
        │        Dual-Stream FFN           │
        │          WiolaRMSNorm            │
        └─────────────────────────────────────┘
                         │
                         ▼
                 Repeated N Layers
                         │
                         ▼
                   Final RMSNorm
                         │
                         ▼
                       LM Head
                         │
                         ▼
                 Generated Tokens

Citation

If you use Wiola in your research, please cite:

@software{wiola2026,
  title={Wiola: A Novel Decoder-Only Small Language Model},
  author={OSCOWL-AI},
  year={2026},
  url={https://github.com/Wiola-OSCOWL-ai/wiola}
}

License

Apache License 2.0

Acknowledgements

Wiola is an open research project developed by OSCOWL-AI to explore efficient and scalable Small Language Model architectures. Contributions, feedback, and research collaborations are welcome.

oscowlai
/

Wiola13M