--- language: - en base_model: - Smilyai-labs/CodVa-1-Small pipeline_tag: text-generation library_name: transformers license: other license_name: codva-community-license license_link: https://huggingface.co/Smilyai-labs/CodVa-1-Small-IT/blob/main/LICENSE.txt tags: - IT - chat - LLM - code - coding - math - moe - instruction-tuned - gqa - rope - custom-architecture --- # CodVa-1-Small-IT Model Card --- ## Model Overview **CodVa-1-Small-IT** is an instruction-tuned Large Language Model developed by **Smilyai-labs** โ€” a team of high-school students passionate about AI research and development. This model is the instruction-tuned (IT) variant of our base model, [CodVa-1-Small](https://huggingface.co/Smilyai-labs/CodVa-1-Small), fine-tuned to follow instructions, answer questions, and assist with coding and mathematical reasoning tasks. --- ## Key Features - ๐Ÿง  **Custom Architecture** โ€” Built on our in-house architecture designed from the ground up to maximise performance on code and math benchmarks - โšก **Mixture of Experts (MoE) FFN** โ€” Sparse MoE feed-forward layers with shared experts, giving a large parameter count while keeping per-token compute low - ๐Ÿ”„ **Grouped Query Attention (GQA)** โ€” Reduces KV-cache memory footprint significantly compared to standard multi-head attention, enabling longer context at inference time - ๐Ÿ“ **RoPE with Extended Theta** โ€” Rotary positional embeddings with a high theta value (5,000,000) for improved length generalisation beyond the training context window - ๐Ÿ”ง **QK Norm** โ€” Per-head RMS normalisation on queries and keys for more stable attention across long sequences - ๐Ÿ’ฌ **Instruction Tuned** โ€” Fine-tuned on a curated dataset of code, math, and general instruction-following examples to produce a helpful, chat-capable assistant --- ## Architecture Details | Component | Configuration | |---|---| | Architecture | Custom Decoder-Only Transformer | | Hidden Dimension | 1536 | | Layers | 28 | | Attention Heads | 24 (Query) / 6 (KV) | | Attention Type | Grouped Query Attention (GQA) | | Positional Encoding | RoPE (ฮธ = 5,000,000) | | FFN Type | Dense SwiGLU + Sparse MoE | | MoE Experts | 16 routed + 2 shared | | MoE Top-K | 2 routed experts per token | | MoE Hidden Dim | 1024 | | MoE Frequency | Every 2 layers | | Context Length | 4096 tokens | | Vocabulary Size | 32,000 (+ special tokens) | | Normalisation | RMSNorm throughout | | QK Norm | โœ… Enabled | | Structural Bias | โœ… Enabled (4 relation types) | | Precision | BFloat16 | --- ## Training Details ### Pre-Training (CodVa-1-Small Base) The base model was pre-trained from scratch on a large corpus of code and mathematical text, optimised for strong reasoning and programming capabilities. ### Instruction Fine-Tuning (This Model) **CodVa-1-Small-IT** was produced by supervised fine-tuning (SFT) of the base model on a curated instruction dataset. | Setting | Value | |---|---| | Fine-tuning Method | Supervised Fine-Tuning (SFT) | | Dataset | [Bc-AI/codva-it-data-v5.1](https://huggingface.co/datasets/Bc-AI/codva-it-data-v5.1) | | Sequence Length | 4096 tokens | | Optimizer | AdamW (8-bit where available) | | Learning Rate | 2e-5 | | LR Schedule | Cosine decay with warmup | | Weight Decay | 0.01 | | Gradient Clipping | 1.0 | | Precision | BFloat16 | --- ## Intended Use **CodVa-1-Small-IT** is intended for: - ๐Ÿ’ป **Code generation and completion** โ€” Writing functions, classes, scripts, and full programs across a range of programming languages - ๐Ÿ› **Code explanation and debugging** โ€” Understanding existing code, identifying bugs, and suggesting fixes - ๐Ÿ“ **Mathematical reasoning** โ€” Step-by-step problem solving, symbolic reasoning, and numerical computation - ๐Ÿค– **General instruction following** โ€” Answering questions, summarising text, and following natural language instructions - ๐Ÿงช **Research and experimentation** โ€” Exploring small-scale LLM behaviour, fine-tuning further, or using as a base for downstream tasks --- ## Limitations As with all language models โ€” particularly smaller ones โ€” there are important limitations to be aware of: - **Factual accuracy** โ€” The model may confidently produce incorrect information. Always verify outputs for any critical use case - **Context length** โ€” Performance may degrade on inputs approaching the 4096-token context limit - **Mathematical reliability** โ€” While the architecture is optimised for math, the model can still make arithmetic and reasoning errors, especially on complex multi-step problems - **Safety** โ€” This model has not undergone extensive red-teaming or safety fine-tuning. It should not be deployed in production applications without additional safety measures - **Scale** โ€” As a small model, it will be outperformed by larger models on complex tasks. It is best suited for lightweight deployment, experimentation, and research --- ## About Smilyai-labs **Smilyai-labs** is a team of high-school students who are passionate about AI research. We design and train our own model architectures from scratch, rather than fine-tuning existing open-source models, with the goal of learning every part of the deep learning stack โ€” from architecture design and custom CUDA kernels to dataset curation and training infrastructure. CodVa is our flagship model series, focused on code and mathematical reasoning. > We are students building real models. Feedback, collaboration offers, and questions are very welcome. --- ## Citation If you use CodVa-1-Small-IT in your research or projects, please consider citing or crediting the Smilyai-labs team: ``` @misc{codva1small, author = {Smilyai-labs}, title = {CodVa-1-Small-IT: An Instruction-Tuned Code and Math Language Model}, year = {2025}, howpublished = {\url{https://huggingface.co/Smilyai-labs/CodVa-1-Small-IT}}, } ``` --- ## License Please refer to the repository's license file for terms of use. If you intend to use this model for commercial purposes, please contact the Smilyai-labs team directly.