README.md

---
language:
- en
base_model:
- Smilyai-labs/CodVa-1-Small
pipeline_tag: text-generation
library_name: transformers
license: other
license_name: codva-community-license
license_link: https://huggingface.co/Smilyai-labs/CodVa-1-Small-IT/blob/main/LICENSE.txt
tags:
- IT
- chat
- LLM
- code
- coding
- math
- moe
- instruction-tuned
- gqa
- rope
- custom-architecture
---


# CodVa-1-Small-IT Model Card

---

## Model Overview

**CodVa-1-Small-IT** is an instruction-tuned Large Language Model developed by **Smilyai-labs** — a team of high-school students passionate about AI research and development. This model is the instruction-tuned (IT) variant of our base model, [CodVa-1-Small](https://huggingface.co/Smilyai-labs/CodVa-1-Small), fine-tuned to follow instructions, answer questions, and assist with coding and mathematical reasoning tasks.

---

## Key Features

- 🧠 **Custom Architecture** — Built on our in-house architecture designed from the ground up to maximise performance on code and math benchmarks
- ⚡ **Mixture of Experts (MoE) FFN** — Sparse MoE feed-forward layers with shared experts, giving a large parameter count while keeping per-token compute low
- 🔄 **Grouped Query Attention (GQA)** — Reduces KV-cache memory footprint significantly compared to standard multi-head attention, enabling longer context at inference time
- 📐 **RoPE with Extended Theta** — Rotary positional embeddings with a high theta value (5,000,000) for improved length generalisation beyond the training context window
- 🔧 **QK Norm** — Per-head RMS normalisation on queries and keys for more stable attention across long sequences
- 💬 **Instruction Tuned** — Fine-tuned on a curated dataset of code, math, and general instruction-following examples to produce a helpful, chat-capable assistant

---

## Architecture Details

| Component | Configuration |
|---|---|
| Architecture | Custom Decoder-Only Transformer |
| Hidden Dimension | 1536 |
| Layers | 28 |
| Attention Heads | 24 (Query) / 6 (KV) |
| Attention Type | Grouped Query Attention (GQA) |
| Positional Encoding | RoPE (θ = 5,000,000) |
| FFN Type | Dense SwiGLU + Sparse MoE |
| MoE Experts | 16 routed + 2 shared |
| MoE Top-K | 2 routed experts per token |
| MoE Hidden Dim | 1024 |
| MoE Frequency | Every 2 layers |
| Context Length | 4096 tokens |
| Vocabulary Size | 32,000 (+ special tokens) |
| Normalisation | RMSNorm throughout |
| QK Norm | ✅ Enabled |
| Structural Bias | ✅ Enabled (4 relation types) |
| Precision | BFloat16 |

---

## Training Details

### Pre-Training (CodVa-1-Small Base)

The base model was pre-trained from scratch on a large corpus of code and mathematical text, optimised for strong reasoning and programming capabilities.

### Instruction Fine-Tuning (This Model)

**CodVa-1-Small-IT** was produced by supervised fine-tuning (SFT) of the base model on a curated instruction dataset.

| Setting | Value |
|---|---|
| Fine-tuning Method | Supervised Fine-Tuning (SFT) |
| Dataset | [Bc-AI/codva-it-data-v5.1](https://huggingface.co/datasets/Bc-AI/codva-it-data-v5.1) |
| Sequence Length | 4096 tokens |
| Optimizer | AdamW (8-bit where available) |
| Learning Rate | 2e-5 |
| LR Schedule | Cosine decay with warmup |
| Weight Decay | 0.01 |
| Gradient Clipping | 1.0 |
| Precision | BFloat16 |

---

## Intended Use

**CodVa-1-Small-IT** is intended for:

- 💻 **Code generation and completion** — Writing functions, classes, scripts, and full programs across a range of programming languages
- 🐛 **Code explanation and debugging** — Understanding existing code, identifying bugs, and suggesting fixes
- 📐 **Mathematical reasoning** — Step-by-step problem solving, symbolic reasoning, and numerical computation
- 🤖 **General instruction following** — Answering questions, summarising text, and following natural language instructions
- 🧪 **Research and experimentation** — Exploring small-scale LLM behaviour, fine-tuning further, or using as a base for downstream tasks

---

## Limitations

As with all language models — particularly smaller ones — there are important limitations to be aware of:

- **Factual accuracy** — The model may confidently produce incorrect information. Always verify outputs for any critical use case
- **Context length** — Performance may degrade on inputs approaching the 4096-token context limit
- **Mathematical reliability** — While the architecture is optimised for math, the model can still make arithmetic and reasoning errors, especially on complex multi-step problems
- **Safety** — This model has not undergone extensive red-teaming or safety fine-tuning. It should not be deployed in production applications without additional safety measures
- **Scale** — As a small model, it will be outperformed by larger models on complex tasks. It is best suited for lightweight deployment, experimentation, and research

---

## About Smilyai-labs

**Smilyai-labs** is a team of high-school students who are passionate about AI research. We design and train our own model architectures from scratch, rather than fine-tuning existing open-source models, with the goal of learning every part of the deep learning stack — from architecture design and custom CUDA kernels to dataset curation and training infrastructure.

CodVa is our flagship model series, focused on code and mathematical reasoning.

> We are students building real models. Feedback, collaboration offers, and questions are very welcome.

---

## Citation

If you use CodVa-1-Small-IT in your research or projects, please consider citing or crediting the Smilyai-labs team:

```
@misc{codva1small,
  author       = {Smilyai-labs},
  title        = {CodVa-1-Small-IT: An Instruction-Tuned Code and Math Language Model},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/Smilyai-labs/CodVa-1-Small-IT}},
}
```

---

## License

Please refer to the repository's license file for terms of use. If you intend to use this model for commercial purposes, please contact the Smilyai-labs team directly.