README.md

---
license: apache-2.0
language:
- en
- it
tags:
- custom-transformer
- from-scratch
- small-language-model
- gqa
- swiglu
- rope
- pytorch
- streamlit
library_name: custom
---

# Nexus SmAll v1

A **89.8M parameter** causal language model built entirely from scratch using a custom transformer architecture, trained on WikiText-103 + synthetic instruction data.

## Try it Online

Test the model directly in your browser (free, no login required):

[![Try Chat](https://img.shields.io/badge/Try%20Live-Streamlit-FF4B4B?logo=streamlit&logoColor=white)](https://try-nexus-ai.streamlit.app)
[![GitHub](https://img.shields.io/badge/View%20Source-181717?logo=github)](https://github.com/JustScriptzz/nexus-smAll-web)

## Architecture

| Parameter | Value |
|-----------|-------|
| Parameters | 89.8M |
| Layers | 10 |
| Hidden dim | 768 |
| Attention heads | 12 |
| KV heads (GQA) | 4 |
| Max sequence length | 512 |
| Vocab size | 50,304 |
| FFN activation | SwiGLU |
| Position encoding | RoPE (θ=500000) |
| Norm | RMSNorm |
| Training | Float32, full precision |

Key design choices:
- **Grouped Query Attention (GQA)**: 12 query heads, 4 KV heads for efficient inference
- **SwiGLU FFN**: Gated activation for better training dynamics
- **RoPE**: Rotary Position Embeddings for length extrapolation
- **Tied embeddings**: Input/output embeddings share weights

## Installation

Requirements: Python 3.8+ and pip.

```bash
# 1. Clone the repository
git clone https://huggingface.co/JustScriptzz/nexus-smAll-v1
cd nexus-smAll-v1

# 2. Install dependencies
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install tokenizers
```

> **GPU users**: Replace `--index-url https://download.pytorch.org/whl/cpu` with the appropriate CUDA version, e.g. `--index-url https://download.pytorch.org/whl/cu124` for CUDA 12.4.

## Usage

### Command line (quick start)

```bash
python chat.py --weights weights/nexus_instruct.pt
```

### Python API

```python
from src.model import Nexus
from src.config import NexusConfig
from tokenizers import Tokenizer
import torch

config = NexusConfig()
model = Nexus(config)

checkpoint = torch.load("weights/nexus_instruct.pt", map_location="cpu", weights_only=False)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

tokenizer = Tokenizer.from_file("data/tokenizer.json")

prompt = "User: What is Python?\nAssistant:"
encoded = tokenizer.encode(prompt)
tokens = torch.tensor([encoded.ids])

output, _ = model.generate(tokens, max_new_tokens=64, temperature=0.2, top_k=40, top_p=0.9)
reply = tokenizer.decode(output)
print(reply)
```

## Training

- **Phase 1**: 100k steps on WikiText-103 (next-token prediction, ~212k sequences)
- **Phase 2**: 5k steps on instruction data (Dolly-15k + synthetic QA, ~1500 examples)

## Limitations

- 90M parameters is very small by modern standards — outputs may be incoherent
- Not instruction-tuned on diverse enough data
- Best used as a learning experiment, not production

## Nexus Plus v2

Looking for a larger, more capable model? Check out **[Nexus Plus v2](https://huggingface.co/JustScriptzz/nexus-plus-v2)** — a Qwen3-4B fine-tune with QLoRA, trained on the same instruction dataset.

[![Plus v2](https://img.shields.io/badge/Plus%20v2-HuggingFace-FFD21E?logo=huggingface)](https://huggingface.co/JustScriptzz/nexus-plus-v2)

## License

Apache 2.0