| --- |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| license: apache-2.0 |
| tags: |
| - novi |
| - novi-micro |
| - instruction-tuning |
| - full-finetuning |
| - smol-smoltalk |
| - causal-lm |
| - bananaall |
| - project-kairo |
| - small-language-model |
| --- |
| |
| # Novi-Micro-Instruct |
|
|
|  |
|
|
| **Novi-Micro-Instruct** is a tiny instruction-tuned causal language model developed by **Novi-AI** as part of **Project Kairo**. It builds on the Novi-Micro model line, which explores language modeling with a model of approximately **5.04 million parameters**. |
|
|
| The model is an experimental small language model intended for instruction-following experiments, education, and lightweight local inference. Its small size also places significant limits on its ability to follow complex instructions, reason reliably, or provide accurate factual answers. |
|
|
| ⚡ **Approximately 5.04M parameters · 2,048-token context · 502.45M tokens processed in the reported training run** |
|
|
| ## Model Details |
|
|
| ### Architecture |
|
|
| Novi-Micro-Instruct uses the Novi-Micro decoder-only Transformer architecture. The architecture described for Novi-Micro includes RMSNorm, Rotary Position Embeddings (RoPE), grouped-query attention, query/key normalization, SwiGLU feed-forward layers, and tied input/output embeddings. |
|
|
| | Property | Value | |
| | --- | --- | |
| | Model type | Causal language model / instruction-tuned model | |
| | Architecture | BananaMind 2-style decoder architecture | |
| | Parameters | Approximately **5.04M** | |
| | Vocabulary size | **16,384** | |
| | Context length | **2,048 tokens** | |
| | Attention | Grouped-Query Attention (GQA) | |
| | Position encoding | RoPE | |
| | Normalization | RMSNorm | |
| | Feed-forward network | SwiGLU | |
| | Query/key normalization | Enabled | |
| | Input/output embeddings | Tied | |
| | Transformers loading | Custom architecture; `trust_remote_code=True` may be required | |
|
|
| Architecture and tokenizer values above follow the Novi-Micro-Base model configuration and should be checked against the files in this repository before relying on them for downstream tooling. |
|
|
| ## Training |
|
|
| Novi-Micro-Instruct was produced by **full fine-tuning** the existing `Novi-AI/Novi-Micro-Base` checkpoint with BananaAll. It was not trained from scratch. The training script configured **28,772 optimizer steps** and completed all of them. The logger reported **502,453,658 processed input tokens** during the run. |
|
|
| The token counter counts non-padding input positions, including prompt/context tokens that are masked out of the loss. It is not a count of assistant-target tokens alone, unique dataset tokens, or the total size of the dataset. These figures describe the reported run and do not by themselves establish instruction-following quality. |
|
|
| ### Reported Training Results |
|
|
| | Setting / metric | Value | |
| | --- | ---: | |
| | Training environment | BananaAll | |
| | Fine-tuning mode | **Full fine-tuning** | |
| | Base checkpoint | `Novi-AI/Novi-Micro-Base` | |
| | Completed optimizer steps | **28,772 / 28,772** | |
| | Dataset rows traversed | **484,570 unique rows** | |
| | Additional rows from next pass | **11** (repeated from the beginning) | |
| | Total dataset row encounters | **484,581** (484,570 + 11 repeats) | |
| | Trainer-reported epoch | **1.0** (the stream continued 11 rows into a second pass) | |
| | Processed input tokens, according to logger | **502,453,658** | |
| | Aggregate training loss | **1.82655** | |
| | Logged loss at step 28,770 | **1.56224** | |
| | Per-device batch size | **2** | |
| | Gradient accumulation | **8** | |
| | Sequence length | **2,048 tokens** | |
| | Configured learning rate | **2 × 10⁻⁴** | |
| | Scheduler | **Cosine** | |
| | Warmup ratio | **3%** (about 863 warmup steps) | |
| | Weight decay | **0.01** | |
| | Maximum gradient norm | **1.0** | |
| | Precision setting | **Auto** (BF16 when supported; otherwise FP32) | |
| | Torch compilation | **Enabled**, with fallback if compilation fails | |
| | Random seed | **1337** | |
| | Logging interval | **10 steps** | |
| | Checkpoint interval | **100 steps** | |
| | Training runtime | **6,523.67 seconds** (about 1 hour, 48 minutes, 44 seconds) | |
| | Reported steps per second | **4.41** | |
| | Reported samples per second | **70.566** | |
| | Reported total FLOPs | **2.0147 × 10¹⁶** | |
|
|
| The final logged training loss and the aggregate training loss are different metrics: `1.56224` is the loss reported at step 28,770, while `1.82655` is the run's overall `train_loss` value. |
|
|
| The cosine learning-rate schedule reached **0** at the final step. The optimizer is not explicitly selected in the supplied configuration, so this card does not assert a particular optimizer implementation. The configured precision is `auto`; the worker selects BF16 when the GPU supports it and otherwise uses FP32 unless precision is changed. |
|
|
| ## Training Data |
|
|
| The training configuration names **[`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk)**, using the `train` split and the `messages` field. It is the only enabled dataset in the provided configuration, with a mix weight of `1`. The worker streams dataset rows during fine-tuning. |
|
|
| **Dataset coverage:** the `train` split contained **484,570 rows**. The run traversed those rows and then continued for **11 rows from the beginning of the dataset** on the next pass: **484,570 unique rows + 11 repeated row encounters = 484,581 row encounters**. The first 11 rows were repeated because the BananaAll streaming worker recreates the dataset iterator when it reaches the end, allowing training to continue until the configured optimizer-step limit. The 484,581 figure counts row encounters, not unique rows. |
|
|
| For each conversation, the training worker uses the messages before the final assistant message as context and trains the model to predict the final assistant message. The prompt/context labels are masked from the language-model loss; the assistant target and its EOS token contribute to the loss. If the tokenizer provides a chat template, the worker uses it to format the conversation; otherwise it falls back to role-tagged text. Sequences are truncated to the configured maximum of **2,048 tokens**. |
|
|
| The configuration does not set a dataset-specific token limit for this fine-tuning run; training stops at the configured maximum of **28,772 optimizer steps**. The reported **502,453,658 processed input tokens** includes prompt/context positions as well as assistant-target positions, and it is not a count of unique dataset tokens. Before redistributing the model, review the dataset card and license and document any additional preprocessing or filtering performed outside the supplied script. |
|
|
| ## Tokenizer |
|
|
| The fine-tuning script loads the tokenizer from **`Novi-AI/Novi-Micro-Base`**; it does not train a new tokenizer. The base model documentation describes a byte-level BPE tokenizer with a vocabulary of **16,384 tokens** and lists `[PAD]`, `[BOS]`, `[EOS]`, and `[UNK]` as special tokens. Check the tokenizer files shipped with the fine-tuned checkpoint for the exact tokenizer configuration and any chat template. |
|
|
| ## Architecture Notes |
|
|
| ### Attention |
|
|
| The model uses **grouped-query attention (GQA)**, in which multiple query heads share key and value heads. **Rotary Position Embeddings (RoPE)** are applied to the query and key representations. The described BananaMind 2-style architecture also normalizes query and key head dimensions. |
|
|
| ### Feed-Forward Network |
|
|
| Each Transformer block uses a **SwiGLU** feed-forward network: |
|
|
| ```text |
| SwiGLU(x) = SiLU(gate(x)) × up(x) |
| ``` |
|
|
| The result is projected back to the model's hidden dimension. |
|
|
| ### Normalization and Embeddings |
|
|
| The architecture uses **RMSNorm** before the attention and feed-forward sublayers. Input token embeddings and language-model output embeddings are tied, reducing the number of independent parameters. |
|
|
| ## Usage |
|
|
| The checkpoint uses a custom model architecture. Depending on the repository's model code and your Transformers version, loading it may require `trust_remote_code=True`. Only enable remote model code when you have reviewed and trust the code in the repository. |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "Novi-AI/Novi-Micro-Instruct" |
| |
| tokenizer = AutoTokenizer.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| model.eval() |
| |
| # Use the prompt format expected by this checkpoint's training data. |
| prompt = "User: Explain why the sky appears blue.\nAssistant:" |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=100, |
| do_sample=True, |
| temperature=0.8, |
| top_p=0.9, |
| pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id, |
| eos_token_id=tokenizer.eos_token_id, |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| The prompt above is a basic plain-text example, not a claim that `User:` / `Assistant:` is the model's canonical chat format. If the tokenizer or repository defines a chat template, prefer that template for inference. |
|
|
| ## Intended Use |
|
|
| Novi-Micro-Instruct is primarily intended for: |
|
|
| - Research and experimentation with very small language models |
| - Educational demonstrations of instruction-tuned causal language models |
| - Testing training pipelines and small-model architectures |
| - Fine-tuning and inference experiments on constrained hardware |
| - Exploring instruction following at the million-parameter scale |
|
|
| This checkpoint is experimental and should not be treated as a production-ready assistant. |
|
|
| ## Limitations |
|
|
| At approximately **5 million parameters**, Novi-Micro-Instruct is much smaller than general-purpose language models. It may: |
|
|
| - Misunderstand or fail to follow instructions |
| - Generate incoherent, repetitive, or incomplete responses |
| - Produce inaccurate or fabricated information |
| - Struggle with multi-step reasoning and complex tasks |
| - Have limited factual knowledge and weak generalization |
| - Behave inconsistently across prompts or prompt formats |
| - Produce grammatically unusual text or fail to maintain long conversations |
|
|
| A 2,048-token context window does not remove the limitations caused by the model's small parameter count. Training loss is not a substitute for held-out evaluation or human review. Evaluate the model on representative tasks before using its outputs for consequential decisions. |
|
|
| ## Evaluation |
|
|
| No benchmark scores or held-out evaluation results are included in the supplied training log. Do not interpret the training loss as a benchmark score. Independent evaluation is recommended for instruction following, factual accuracy, repetition, robustness, and safety. |
|
|
| ## Project History |
|
|
| Novi-Micro-Instruct is part of **Project Kairo**, the development codename for the Novi model project. The Novi series follows the earlier **AppleMind** experiments and represents Novi-AI's model-development line. |
|
|
| **AppleMind → Novi-Nano → Novi-Micro → Novi-Micro-Instruct** 🚀 |
|
|
| Novi-Micro explores language modeling at the million-parameter scale; Novi-Micro-Instruct is the instruction-oriented checkpoint in that line. |
|
|
| ## Development Status |
|
|
| I'm taking a break from model development for a while after completing Novi-Micro-Instruct. There is currently no announced timeline for a successor model or further model releases. This is a pause, not a declaration that model development is permanently over. |
|
|
| ## Training Infrastructure |
|
|
| The reported run used **BananaAll**. The final log records a completed training run and a saved checkpoint. The original notebook reported the final model archive as: |
|
|
| ```text |
| project-kairo-micro-instruct-model.zip |
| ``` |
|
|
| The training script packages the final model and tokenizer files, while excluding intermediate `checkpoint-*` directories from the ZIP. For full fine-tuning, it also copies the custom Python model code from the loaded base checkpoint when that code is available through Transformers remote modules. |
|
|
| ## Acknowledgements |
|
|
| Novi-Micro-Instruct was built using the open-source machine-learning ecosystem and community model-development tools. Thanks to: |
|
|
| - Hugging Face 🤗 |
| - The open-source Transformers ecosystem |
| - The developers and contributors whose tools support small-model research |
|
|
| ## License |
|
|
| This repository is marked as **Apache-2.0**. Review the licenses of all training data and any included custom code separately; a model's repository license does not automatically establish the license of its training data. |
|
|
| --- |
|
|
| ## 🧠 Novi AI |
|
|
| **Small models. Big experiments.** |
|
|
| Novi-Micro-Instruct is an experimental instruction-tuned model designed to explore what can be learned and tested at the few-million-parameter scale. |
|
|
| *Novi AI 2026 — Project Kairo* |
|
|
