Duplicate from Novi-AI/Novi-Micro-Instruct

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+banner.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,242 @@
+---
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+license: apache-2.0
+tags:
+  - novi
+  - novi-micro
+  - instruction-tuning
+  - full-finetuning
+  - smol-smoltalk
+  - causal-lm
+  - bananaall
+  - project-kairo
+  - small-language-model
+---
+
+# Novi-Micro-Instruct
+
+![Novi-Micro Banner](banner.jpg)
+
+**Novi-Micro-Instruct** is a tiny instruction-tuned causal language model developed by **Novi-AI** as part of **Project Kairo**. It builds on the Novi-Micro model line, which explores language modeling with a model of approximately **5.04 million parameters**.
+
+The model is an experimental small language model intended for instruction-following experiments, education, and lightweight local inference. Its small size also places significant limits on its ability to follow complex instructions, reason reliably, or provide accurate factual answers.
+
+⚡ **Approximately 5.04M parameters · 2,048-token context · 502.45M tokens processed in the reported training run**
+
+## Model Details
+
+### Architecture
+
+Novi-Micro-Instruct uses the Novi-Micro decoder-only Transformer architecture. The architecture described for Novi-Micro includes RMSNorm, Rotary Position Embeddings (RoPE), grouped-query attention, query/key normalization, SwiGLU feed-forward layers, and tied input/output embeddings.
+
+| Property | Value |
+| --- | --- |
+| Model type | Causal language model / instruction-tuned model |
+| Architecture | BananaMind 2-style decoder architecture |
+| Parameters | Approximately **5.04M** |
+| Vocabulary size | **16,384** |
+| Context length | **2,048 tokens** |
+| Attention | Grouped-Query Attention (GQA) |
+| Position encoding | RoPE |
+| Normalization | RMSNorm |
+| Feed-forward network | SwiGLU |
+| Query/key normalization | Enabled |
+| Input/output embeddings | Tied |
+| Transformers loading | Custom architecture; `trust_remote_code=True` may be required |
+
+Architecture and tokenizer values above follow the Novi-Micro-Base model configuration and should be checked against the files in this repository before relying on them for downstream tooling.
+
+## Training
+
+Novi-Micro-Instruct was produced by **full fine-tuning** the existing `Novi-AI/Novi-Micro-Base` checkpoint with BananaAll. It was not trained from scratch. The training script configured **28,772 optimizer steps** and completed all of them. The logger reported **502,453,658 processed input tokens** during the run.
+
+The token counter counts non-padding input positions, including prompt/context tokens that are masked out of the loss. It is not a count of assistant-target tokens alone, unique dataset tokens, or the total size of the dataset. These figures describe the reported run and do not by themselves establish instruction-following quality.
+
+### Reported Training Results
+
+| Setting / metric | Value |
+| --- | ---: |
+| Training environment | BananaAll |
+| Fine-tuning mode | **Full fine-tuning** |
+| Base checkpoint | `Novi-AI/Novi-Micro-Base` |
+| Completed optimizer steps | **28,772 / 28,772** |
+| Dataset rows traversed | **484,570 unique rows** |
+| Additional rows from next pass | **11** (repeated from the beginning) |
+| Total dataset row encounters | **484,581** (484,570 + 11 repeats) |
+| Trainer-reported epoch | **1.0** (the stream continued 11 rows into a second pass) |
+| Processed input tokens, according to logger | **502,453,658** |
+| Aggregate training loss | **1.82655** |
+| Logged loss at step 28,770 | **1.56224** |
+| Per-device batch size | **2** |
+| Gradient accumulation | **8** |
+| Sequence length | **2,048 tokens** |
+| Configured learning rate | **2 × 10⁻⁴** |
+| Scheduler | **Cosine** |
+| Warmup ratio | **3%** (about 863 warmup steps) |
+| Weight decay | **0.01** |
+| Maximum gradient norm | **1.0** |
+| Precision setting | **Auto** (BF16 when supported; otherwise FP32) |
+| Torch compilation | **Enabled**, with fallback if compilation fails |
+| Random seed | **1337** |
+| Logging interval | **10 steps** |
+| Checkpoint interval | **100 steps** |
+| Training runtime | **6,523.67 seconds** (about 1 hour, 48 minutes, 44 seconds) |
+| Reported steps per second | **4.41** |
+| Reported samples per second | **70.566** |
+| Reported total FLOPs | **2.0147 × 10¹⁶** |
+
+The final logged training loss and the aggregate training loss are different metrics: `1.56224` is the loss reported at step 28,770, while `1.82655` is the run's overall `train_loss` value.
+
+The cosine learning-rate schedule reached **0** at the final step. The optimizer is not explicitly selected in the supplied configuration, so this card does not assert a particular optimizer implementation. The configured precision is `auto`; the worker selects BF16 when the GPU supports it and otherwise uses FP32 unless precision is changed.
+
+## Training Data
+
+The training configuration names **[`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk)**, using the `train` split and the `messages` field. It is the only enabled dataset in the provided configuration, with a mix weight of `1`. The worker streams dataset rows during fine-tuning.
+
+**Dataset coverage:** the `train` split contained **484,570 rows**. The run traversed those rows and then continued for **11 rows from the beginning of the dataset** on the next pass: **484,570 unique rows + 11 repeated row encounters = 484,581 row encounters**. The first 11 rows were repeated because the BananaAll streaming worker recreates the dataset iterator when it reaches the end, allowing training to continue until the configured optimizer-step limit. The 484,581 figure counts row encounters, not unique rows.
+
+For each conversation, the training worker uses the messages before the final assistant message as context and trains the model to predict the final assistant message. The prompt/context labels are masked from the language-model loss; the assistant target and its EOS token contribute to the loss. If the tokenizer provides a chat template, the worker uses it to format the conversation; otherwise it falls back to role-tagged text. Sequences are truncated to the configured maximum of **2,048 tokens**.
+
+The configuration does not set a dataset-specific token limit for this fine-tuning run; training stops at the configured maximum of **28,772 optimizer steps**. The reported **502,453,658 processed input tokens** includes prompt/context positions as well as assistant-target positions, and it is not a count of unique dataset tokens. Before redistributing the model, review the dataset card and license and document any additional preprocessing or filtering performed outside the supplied script.
+
+## Tokenizer
+
+The fine-tuning script loads the tokenizer from **`Novi-AI/Novi-Micro-Base`**; it does not train a new tokenizer. The base model documentation describes a byte-level BPE tokenizer with a vocabulary of **16,384 tokens** and lists `[PAD]`, `[BOS]`, `[EOS]`, and `[UNK]` as special tokens. Check the tokenizer files shipped with the fine-tuned checkpoint for the exact tokenizer configuration and any chat template.
+
+## Architecture Notes
+
+### Attention
+
+The model uses **grouped-query attention (GQA)**, in which multiple query heads share key and value heads. **Rotary Position Embeddings (RoPE)** are applied to the query and key representations. The described BananaMind 2-style architecture also normalizes query and key head dimensions.
+
+### Feed-Forward Network
+
+Each Transformer block uses a **SwiGLU** feed-forward network:
+
+```text
+SwiGLU(x) = SiLU(gate(x)) × up(x)
+```
+
+The result is projected back to the model's hidden dimension.
+
+### Normalization and Embeddings
+
+The architecture uses **RMSNorm** before the attention and feed-forward sublayers. Input token embeddings and language-model output embeddings are tied, reducing the number of independent parameters.
+
+## Usage
+
+The checkpoint uses a custom model architecture. Depending on the repository's model code and your Transformers version, loading it may require `trust_remote_code=True`. Only enable remote model code when you have reviewed and trust the code in the repository.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Novi-AI/Novi-Micro-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model.eval()
+
+# Use the prompt format expected by this checkpoint's training data.
+prompt = "User: Explain why the sky appears blue.\nAssistant:"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=100,
+        do_sample=True,
+        temperature=0.8,
+        top_p=0.9,
+        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+The prompt above is a basic plain-text example, not a claim that `User:` / `Assistant:` is the model's canonical chat format. If the tokenizer or repository defines a chat template, prefer that template for inference.
+
+## Intended Use
+
+Novi-Micro-Instruct is primarily intended for:
+
+- Research and experimentation with very small language models
+- Educational demonstrations of instruction-tuned causal language models
+- Testing training pipelines and small-model architectures
+- Fine-tuning and inference experiments on constrained hardware
+- Exploring instruction following at the million-parameter scale
+
+This checkpoint is experimental and should not be treated as a production-ready assistant.
+
+## Limitations
+
+At approximately **5 million parameters**, Novi-Micro-Instruct is much smaller than general-purpose language models. It may:
+
+- Misunderstand or fail to follow instructions
+- Generate incoherent, repetitive, or incomplete responses
+- Produce inaccurate or fabricated information
+- Struggle with multi-step reasoning and complex tasks
+- Have limited factual knowledge and weak generalization
+- Behave inconsistently across prompts or prompt formats
+- Produce grammatically unusual text or fail to maintain long conversations
+
+A 2,048-token context window does not remove the limitations caused by the model's small parameter count. Training loss is not a substitute for held-out evaluation or human review. Evaluate the model on representative tasks before using its outputs for consequential decisions.
+
+## Evaluation
+
+No benchmark scores or held-out evaluation results are included in the supplied training log. Do not interpret the training loss as a benchmark score. Independent evaluation is recommended for instruction following, factual accuracy, repetition, robustness, and safety.
+
+## Project History
+
+Novi-Micro-Instruct is part of **Project Kairo**, the development codename for the Novi model project. The Novi series follows the earlier **AppleMind** experiments and represents Novi-AI's model-development line.
+
+**AppleMind → Novi-Nano → Novi-Micro → Novi-Micro-Instruct** 🚀
+
+Novi-Micro explores language modeling at the million-parameter scale; Novi-Micro-Instruct is the instruction-oriented checkpoint in that line.
+
+## Development Status
+
+I'm taking a break from model development for a while after completing Novi-Micro-Instruct. There is currently no announced timeline for a successor model or further model releases. This is a pause, not a declaration that model development is permanently over.
+
+## Training Infrastructure
+
+The reported run used **BananaAll**. The final log records a completed training run and a saved checkpoint. The original notebook reported the final model archive as:
+
+```text
+project-kairo-micro-instruct-model.zip
+```
+
+The training script packages the final model and tokenizer files, while excluding intermediate `checkpoint-*` directories from the ZIP. For full fine-tuning, it also copies the custom Python model code from the loaded base checkpoint when that code is available through Transformers remote modules.
+
+## Acknowledgements
+
+Novi-Micro-Instruct was built using the open-source machine-learning ecosystem and community model-development tools. Thanks to:
+
+- Hugging Face 🤗
+- The open-source Transformers ecosystem
+- The developers and contributors whose tools support small-model research
+
+## License
+
+This repository is marked as **Apache-2.0**. Review the licenses of all training data and any included custom code separately; a model's repository license does not automatically establish the license of its training data.
+
+---
+
+## 🧠 Novi AI
+
+**Small models. Big experiments.**
+
+Novi-Micro-Instruct is an experimental instruction-tuned model designed to explore what can be learned and tested at the few-million-parameter scale.
+
+*Novi AI 2026 — Project Kairo*

config.json

@@ -0,0 +1,30 @@
+{
+  "architecture_style": "bananamind2",
+  "architectures": [
+    "BananaAllForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_bananaall.BananaAllConfig",
+    "AutoModelForCausalLM": "modeling_bananaall.BananaAllForCausalLM"
+  },
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 40,
+  "hidden_size": 160,
+  "intermediate_size": 864,
+  "lft": false,
+  "max_position_embeddings": 4096,
+  "model_type": "bananaall",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 9,
+  "num_key_value_heads": 2,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 100000.0,
+  "ternary": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 3840
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.14.1",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfff837aa539f56679bc15530ab159fb945abcbc81aeb30052bb6b096cafffe1
+size 22634608

tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "[BOS]",
+  "eos_token": "[EOS]",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 2048,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc3cf2873dbaffaec05d62db799e86d2924fcc9ad67bf29761880a85759cc194
+size 5201