Upload 12 files

LICENSE

@@ -0,0 +1,49 @@
+MIT License
+
+Copyright (c) 2026 StellarAI Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+关于商业使用的说明
+====================
+
+本项目 (StellarAI 模型代码) 在 MIT 许可证下发布，完全可商用。
+
+您可以自由地：
+  ✅ 用于商业产品
+  ✅ 修改源码
+  ✅ 分发再发行
+  ✅ 申请专利
+  ✅ 私有化部署
+
+您需要：
+  📝 在产品中保留本 LICENSE 文件的版权声明
+  📝 如修改了源码，建议在变更记录中注明
+
+模型权重的额外说明：
+- 本仓库提供的是**随机初始化的权重**和**训练代码**，不包含经过数据训练的预训练权重。
+- 如果您使用受版权或许可证限制的数据集训练此模型，请确保您有权使用这些数据，
+  并且训练出的权重文件的许可证符合原始数据集的要求。
+- 对于**您自行训练出的权重**，您完全拥有其所有权，可按任何许可证发布（包括商用）。
+
+第三方数据的合规性：
+- 训练和使用模型时请遵守当地法律法规（如《生成式人工智能服务管理暂行办法》等）。
+- 请勿使用本模型从事违法违规活动。

README.md

@@ -1,3 +1,213 @@
 ---
 license: mit
+language:
+  - zh
+  - en
+tags:
+  - stellarai
+  - multimodal
+  - tiny-llm
+  - causal-lm
+  - text-generation
+  - vision
+  - cpu-friendly
+library_name: transformers
+pipeline_tag: text-generation
+widget:
+  - text: "Artificial intelligence is"
+    example_title: "English Generation"
+  - text: "人工智能是一种"
+    example_title: "Chinese Generation"
 ---
+
+# StellarAI-Tiny
+
+**A lightweight multimodal language model trained from scratch — ~50M parameters (0.05B), runs on CPU with 4GB RAM.**
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-ee4c2c)](https://pytorch.org/)
+[![Model Size](https://img.shields.io/badge/Size-~93MB-blue)](#)
+[![Parameters](https://img.shields.io/badge/Params-50M-green)](#)
+
+---
+
+## Overview
+
+StellarAI-Tiny is a from-scratch, bilingual (Chinese + English) causal language model with multimodal vision support. Designed for educational and prototyping purposes, it requires minimal hardware and ships with a built-in plugin system for tool calling.
+
+| Feature | Description |
+|---------|-------------|
+| Lightweight | 50M parameters, ~93MB weights |
+| CPU-friendly | Runs smoothly on CPU with 4GB RAM |
+| Transformer | 4-layer text encoder + RoPE positional encoding |
+| Multimodal | CNN + ViT hybrid vision encoder + cross-attention fusion |
+| Bilingual | Chinese + English mixed tokenization & generation |
+| License | MIT — fully permissive for commercial use |
+| Plugins | Built-in calculator, knowledge base, translator, text tools, time queries |
+
+---
+
+## Architecture
+
+```
+StellarAI-Tiny (~50M Parameters)
+├── Embedding          vocab(32000) x d_model(384)
+├── Text Transformer (4 layers)
+│   ├── Multi-Head Self-Attention (6 heads, RoPE)
+│   └── FFN (GELU, intermediate=1536)
+├── Vision Encoder (CNN + ViT)
+│   ├── CNN Feature Extractor (4 layers: 24→48→96→192 channels)
+│   └── ViT Transformer (2 layers, 6 heads)
+├── Fusion Block (1 layer)
+│   ├── Self-Attention + Cross-Attention
+│   └── FFN (1536)
+└── LM Head (tied weights)
+```
+
+| Config | Value |
+|--------|-------|
+| `d_model` | 384 |
+| `num_hidden_layers` | 4 |
+| `num_attention_heads` | 6 |
+| `intermediate_size` | 1536 |
+| `vocab_size` | 32000 |
+| `max_position_embeddings` | 1024 |
+| `vision_num_layers` | 2 |
+| `fusion_num_layers` | 1 |
+| Total parameters | ~50M (0.05B) |
+
+---
+
+## Quick Start
+
+### Requirements
+
+```bash
+pip install torch transformers safetensors
+```
+
+### Text Generation
+
+```python
+from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
+import torch
+
+model_name = "amtstudio/stellarai-tiny"  # or your local path
+
+config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    config=config,
+    trust_remote_code=True,
+    torch_dtype=torch.float32,
+)
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+
+# Generate
+prompt = "Artificial intelligence is"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+output_ids = model.generate(
+    **inputs,
+    max_new_tokens=64,
+    temperature=0.7,
+    top_k=40,
+    do_sample=True,
+    pad_token_id=tokenizer.pad_token_id,
+    eos_token_id=tokenizer.eos_token_id,
+)
+
+print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
+```
+
+### Using `generate_text` Convenience Method
+
+```python
+result = model.generate_text(
+    prompt="Artificial intelligence is",
+    tokenizer=tokenizer,
+    max_new_tokens=100,
+    temperature=0.8,
+)
+print(result)
+```
+
+---
+
+## Training Details
+
+| Item | Detail |
+|------|--------|
+| Training steps | 12,000 (5,000 base + 7,000 general training) |
+| Corpus | 19,846 lines of bilingual data (AI, CS, NLP, math, programming, reasoning, dialogue, plugins) |
+| Optimizer | AdamW (lr=3e-4, wd=0.01) |
+| LR Schedule | Cosine annealing + Warmup (100 steps) |
+| Batch size | 4 |
+| Sequence length | 128 |
+| Gradient clipping | 1.0 |
+| Final loss | 4.06 (ppl ≈ 58) |
+| Vocabulary size | 11,030 |
+| Device | CPU |
+| Training time | ~3.2 hours |
+
+The training corpus was built from a mix of hand-crafted bilingual data, synthetic instruction-tuning data, and Chinese NLP datasets across 10+ domains. The model was trained with a next-token-prediction objective using the custom `SimpleTokenizer` (BPE).
+
+---
+
+## File Structure
+
+```
+├── config.json                    # HF model configuration
+├── configuration_stellarai.py     # Custom PretrainedConfig class
+├── modeling_stellarai.py          # Custom PreTrainedModel class
+├── tokenization_stellarai.py      # Custom PreTrainedTokenizer class
+├── model.safetensors              # Safetensors weights (93.6 MB)
+├── pytorch_model.bin              # PyTorch weights (93.6 MB)
+├── tokenizer_config.json          # Tokenizer configuration
+├── special_tokens_map.json        # Special token mappings
+├── tokenizer.json                 # HF tokenizer definition (BPE)
+├── backend_tokenizer.json         # Original backend tokenizer
+├── vocab.json                     # BPE vocabulary (11,030 tokens)
+├── merges.txt                     # BPE merge rules
+├── README.md                      # This file
+└── LICENSE                        # MIT License
+```
+
+---
+
+## Limitations
+
+> **Important**: This is a lightweight educational / prototyping model.
+
+1. **Limited knowledge**: Trained on ~19K lines of curated data. Knowledge coverage is narrow.
+2. **Factual accuracy**: May produce inaccurate, nonsensical, or hallucinated content.
+3. **Generation quality**: Suitable for demonstrating basic language modeling — not production-level dialogue.
+4. **Vision capability**: The vision encoder is pre-trained on text-only data. VQA requires additional fine-tuning with image-text pairs.
+5. **Plugin calling**: The model learned the `[TOOL:xxx]` format but calling accuracy needs improvement.
+6. **Not suitable for**: Production environments, medical/legal/financial domains.
+
+### Suggested Improvements
+
+- [ ] Expand corpus to 50K+ lines or use public datasets (WikiText, C4, Oscar)
+- [ ] Increase training to 50K+ steps
+- [ ] Add real dialogue data (ShareGPT, Alpaca format) for SFT
+- [ ] Collect image-text pairs (e.g., COCO captions) to fine-tune multimodal capability
+- [ ] Try larger config: 6 layers / 512d / 8 heads (~0.1B)
+
+---
+
+## License
+
+[MIT License](LICENSE) — fully permissive for personal and commercial use.
+
+---
+
+## Acknowledgements
+
+- Architecture inspired by GPT-2, LLaMA, ViT, and BLIP-2
+- Built with Hugging Face `transformers`
+- RoPE: *RoFormer: Enhanced Transformer with Rotary Position Embedding*
+
+---
+
+**StellarAI** — Exploring AI, one star at a time.

config.json

@@ -0,0 +1,53 @@
+{
+  "architectures": ["StellarAIForCausalLM"],
+  "model_type": "stellarai",
+  "auto_map": {
+    "AutoConfig": "configuration_stellarai.StellarAIConfig",
+    "AutoModelForCausalLM": "modeling_stellarai.StellarAIForCausalLM",
+    "AutoTokenizer": "tokenization_stellarai.StellarAITokenizer"
+  },
+
+  "d_model": 384,
+  "num_hidden_layers": 4,
+  "num_attention_heads": 6,
+  "intermediate_size": 1536,
+  "hidden_act": "gelu",
+  "max_position_embeddings": 1024,
+  "vocab_size": 32000,
+  "dropout": 0.1,
+  "layer_norm_eps": 1e-6,
+
+  "rope_theta": 10000.0,
+  "use_cache": true,
+  "pad_token_id": 0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "unk_token_id": 3,
+  "sep_token_id": 6,
+  "vision_token_id": 7,
+  "boi_token_id": 8,
+  "eoi_token_id": 9,
+
+  "vision_cfg": {
+    "image_size": 224,
+    "patch_size": 16,
+    "num_channels": 3,
+    "cnn_channels": [24, 48, 96, 192],
+    "vision_num_layers": 2,
+    "vision_num_heads": 6,
+    "vision_ff_dim": 768,
+    "vision_num_patches": 196
+  },
+
+  "mm_fusion_cfg": {
+    "fusion_num_layers": 1,
+    "fusion_num_heads": 6,
+    "fusion_ff_dim": 1536
+  },
+
+  "lm_head_bias": true,
+  "use_weight_tying": true,
+
+  "torch_dtype": "float32",
+  "transformers_version": "4.30.0"
+}

configuration_stellarai.py

@@ -0,0 +1,99 @@
+# coding=utf-8
+""" StellarAI model configuration """
+
+from transformers import PretrainedConfig
+from typing import Dict, Any
+
+
+class StellarAIConfig(PretrainedConfig):
+    r"""
+    This is the configuration class for StellarAI.
+    """
+    model_type = "stellarai"
+    keys_to_ignore_at_inference = ["past_key_values"]
+    attribute_map = {
+        "hidden_size": "d_model",
+    }
+
+    def __init__(
+        self,
+        # Generic transformer
+        d_model: int = 384,
+        num_hidden_layers: int = 4,
+        num_attention_heads: int = 6,
+        intermediate_size: int = 1536,
+        hidden_act: str = "gelu",
+        max_position_embeddings: int = 1024,
+        vocab_size: int = 32000,
+        dropout: float = 0.1,
+        layer_norm_eps: float = 1e-6,
+        # RoPE
+        rope_theta: float = 10000.0,
+        use_cache: bool = True,
+        # Special tokens
+        pad_token_id: int = 0,
+        bos_token_id: int = 1,
+        eos_token_id: int = 2,
+        unk_token_id: int = 3,
+        sep_token_id: int = 6,
+        vision_token_id: int = 7,
+        boi_token_id: int = 8,
+        eoi_token_id: int = 9,
+        # Vision encoder (private nested dict to avoid HF fusion_mapping scanning)
+        vision_cfg: Dict[str, Any] = None,
+        # Multimodal fusion (private nested dict)
+        mm_fusion_cfg: Dict[str, Any] = None,
+        # LM head
+        lm_head_bias: bool = True,
+        use_weight_tying: bool = True,
+        **kwargs,
+    ):
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            return_dict=True,
+            **kwargs,
+        )
+        self.d_model = d_model
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.intermediate_size = intermediate_size
+        self.hidden_act = hidden_act
+        self.max_position_embeddings = max_position_embeddings
+        self.vocab_size = vocab_size
+        self.dropout = dropout
+        self.layer_norm_eps = layer_norm_eps
+        self.rope_theta = rope_theta
+        self.use_cache = use_cache
+
+        self.sep_token_id = sep_token_id
+        self.vision_token_id = vision_token_id
+        self.boi_token_id = boi_token_id
+        self.eoi_token_id = eoi_token_id
+
+        # Store vision parameters
+        if vision_cfg is None:
+            vision_cfg = {
+                "image_size": 224,
+                "patch_size": 16,
+                "num_channels": 3,
+                "cnn_channels": [24, 48, 96, 192],
+                "vision_num_layers": 2,
+                "vision_num_heads": 6,
+                "vision_ff_dim": 768,
+                "vision_num_patches": 196,
+            }
+        self.vision_cfg = vision_cfg
+
+        # Store fusion parameters
+        if mm_fusion_cfg is None:
+            mm_fusion_cfg = {
+                "fusion_num_layers": 1,
+                "fusion_num_heads": 6,
+                "fusion_ff_dim": 1536,
+            }
+        self.mm_fusion_cfg = mm_fusion_cfg
+
+        self.lm_head_bias = lm_head_bias
+        self.use_weight_tying = use_weight_tying

merges.txt

@@ -0,0 +1,2019 @@
+#version: 0.2
+t h
+i n
+o n</w>
+t e
+a n
+t i
+p y
+py th
+pyth on</w>
+a r
+t o
+r e
+to o
+o r
+too l</w>
+e n
+c o
+a t
+l o
+t r
+d e
+l e
+a i</w>
+r o
+a s
+l i
+in g</w>
+e r
+m e</w>
+n o
+u r
+s e
+a l
+a c
+a l</w>
+th e</w>
+i n</w>
+n e
+l l
+0 0</w>
+e r</w>
+u n
+s te
+o n
+s t
+g e</w>
+an d</w>
+i c
+m a
+ti me</w>
+i s</w>
+d a
+e s</w>
+o r</w>
+an s
+tr ans
+ste ll
+stell ar
+stellar ai</w>
+in t</w>
+ti on</w>
+1 0</w>
+t o</w>
+l e</w>
+at e</w>
+p ro
+i s
+f or
+x t</w>
+p o
+i t</w>
+1 9
+lo g</w>
+se l
+sel f</w>
+a b
+m p
+f or</w>
+s q
+c l
+g e
+o f</w>
+p r
+de f</w>
+i t
+2 0</w>
+c h
+s t</w>
+m o
+c e
+d i
+1 00</w>
+a p
+t ur
+c al
+n a
+as s</w>
+k no
+w or
+te r</w>
+ar r</w>
+kno w
+s o
+c e</w>
+d n
+co n
+know le
+knowle d
+knowled ge</w>
+e x
+h t
+i g
+ti on
+cal c</w>
+pr int</w>
+d o
+a s</w>
+te r
+q u
+t s</w>
+m e
+2 5</w>
+d e</w>
+l ate</w>
+or t</w>
+na me</w>
+cl ass</w>
+2 0
+trans late</w>
+f i
+u s
+l a
+re tur
+retur n</w>
+h e
+k e
+d at
+1 5</w>
+trans for
+1 0
+te xt</w>
+p e</w>
+ge t</w>
+1 2</w>
+ht t
+m er</w>
+dn a</w>
+r a
+i f</w>
+r an
+re a
+s tr
+s s</w>
+v er
+at h</w>
+m s</w>
+p e
+dat a</w>
+l d</w>
+2 4</w>
+d p</w>
+v e</w>
+co mp
+s u
+i p</w>
+tion s</w>
+da te
+date time</w>
+f ro
+transfor mer</w>
+m i
+5 0</w>
+3 0</w>
+d x</w>
+s y
+s on</w>
+w e
+lo g
+fro m</w>
+a n</w>
+r i
+b o
+j son</w>
+d u
+a tion</w>
+u t</w>
+p re
+s s
+m m
+l s</w>
+l y</w>
+g en
+n ing</w>
+4 5</w>
+p t</w>
+u t
+a t</w>
+s e</w>
+ig h
+u e</w>
+p er
+c i
+1 2
+htt p</w>
+h i
+ar e</w>
+w a
+v o
+wor ld</w>
+m en
+sq l</w>
+t ar
+c ur
+te d</w>
+u l
+as y
+asy n
+6 0</w>
+le ar
+v e
+r t</w>
+is i
+ab le</w>
+s ort</w>
+w it
+l li
+b y</w>
+sq rt</w>
+wit h</w>
+s i
+t h</w>
+da y</w>
+a v
+a d
+tar get</w>
+no w</w>
+y s</w>
+2 1</w>
+b e
+he ll
+hell o</w>
+f u
+m ath</w>
+c t</w>
+f s</w>
+4 0</w>
+ap i</w>
+st r</w>
+for ma
+c h</w>
+i mp
+imp ort</w>
+ne xt</w>
+20 24</w>
+p s</w>
+an g
+1 4
+f un
+s h
+s in
+f f
+v isi
+ma x</w>
+f lo
+t y
+li st</w>
+n u
+a ge</w>
+te m</w>
+m b
+s p
+f t</w>
+igh t</w>
+le c
+t y</w>
+co mm
+in e</w>
+lear ning</w>
+m an
+t c
+m y
+p h
+mi d</w>
+f ac
+do m</w>
+r e</w>
+al l</w>
+too ls</w>
+1 8
+g o
+1 5
+p ar
+it y</w>
+d s</w>
+t a</w>
+a b</w>
+p u</w>
+ty pe</w>
+d ic
+ran ge</w>
+in it</w>
+p l
+co un
+o s</w>
+a tions</w>
+d ing</w>
+asyn c</w>
+5 6</w>
+m p</w>
+1 8</w>
+1 9</w>
+m on
+comp u
+ac h
+h u
+ur al</w>
+e d</w>
+b i
+tc p</w>
+t e</w>
+n p</w>
+i e
+j av
+1 6</w>
+g ra
+t u
+e l
+w hi
+whi le</w>
+m er
+ic k
+as s
+th o
+st a
+pro ce
+p a
+po w
+o u
+1 1</w>
+en ti
+in forma
+informa tion</w>
+r u
+o p
+vo l
+men t</w>
+te s</w>
+i z
+ac k</w>
+b as
+s pe
+ti c</w>
+dn s</w>
+h as</w>
+tr i
+9 9</w>
+3 5</w>
+ne t</w>
+ic i
+nu ms</w>
+ap p
+dic t</w>
+el se</w>
+d a</w>
+li f
+ran dom</w>
+p y</w>
+i ter
+9 0</w>
+in te
+te ch
+tech no
+techno log
+technolog y</w>
+un i
+s in</w>
+st an
+er s</w>
+ne t
+net wor
+le s</w>
+th e
+v er</w>
+co de</w>
+c es</w>
+ma n</w>
+in f
+e c
+w s</w>
+pro du
+fu l</w>
+a li
+s c
+ar g
+9 5</w>
+lo w</w>
+de l
+ke t</w>
+2 3</w>
+da ys</w>
+r n
+b pe</w>
+p ath</w>
+a g
+po int</w>
+ad d</w>
+tr y</w>
+ex ce
+ro r</w>
+j o
+i dx</w>
+f in
+ti c
+o b
+de e
+iter tools</w>
+no n
+inte lli
+intelli gen
+1 4</w>
+1 7</w>
+co s
+a x</w>
+la r</w>
+ter s</w>
+in ter
+en g
+in s</w>
+un d</w>
+v en
+ic al</w>
+or g
+org an
+g it</w>
+hu man</w>
+de r
+c d</w>
+del ta</w>
+f a
+0 0
+qu e
+c ket</w>
+d fs</w>
+x 1</w>
+3 6</w>
+2 8
+i d</w>
+d ro
+f t
+te n
+s en
+c pu</w>
+5 g</w>
+a u
+o l</w>
+en d</w>
+se t</w>
+la mb
+lamb da</w>
+i m
+er ror</w>
+p er</w>
+m in</w>
+le ft</w>
+r ight</w>
+fac to
+sy s</w>
+w i
+c r
+ar ti
+m or
+ti me
+2 00</w>
+li m</w>
+m ach
+th at</w>
+k s</w>
+l ang
+lang u
+sy ste
+fun c
+v es</w>
+w h
+en er
+ener g
+energ y</w>
+str u
+re s</w>
+ac ce
+he l
+so ur
+tur e</w>
+vol u
+ti ve</w>
+r ap
+re d
+que ue</w>
+su b
+so cket</w>
+h ig
+hig h</w>
+t ing</w>
+d y</w>
+x 2</w>
+y 1</w>
+2 x</w>
+3 x</w>
+k m</w>
+to day</w>
+d ate</w>
+ro pe</w>
+so ft
+b er
+a da
+m it</w>
+as c
+ri pt</w>
+app end</w>
+i tem</w>
+ma p</w>
+exce pt</w>
+pe n</w>
+t xt</w>
+jo in</w>
+en u
+z ip</w>
+qu ick
+le n</w>
+co l
+col lec
+collec tions</w>
+facto ri
+factori al</w>
+10 00</w>
+tr ue</w>
+s or
+ke y</w>
+ch o
+co py</w>
+c lo
+g lo
+glo b
+is e</w>
+ex it</w>
+asyn ci
+a wa
+awa it</w>
+1 3</w>
+p r</w>
+arti f
+artif ici
+artifici al</w>
+intelligen ce</w>
+14 15
+1415 9</w>
+lif e</w>
+pow er</w>
+mo d</w>
+compu ter</w>
+s ci
+19 45</w>
+π r</w>
+v ing</w>
+n s</w>
+it s</w>
+re qu
+requ i
+fi c</w>
+l in
+v i
+y o
+yo u</w>
+con t
+tr a
+n at
+langu age</w>
+o ne
+men ts</w>
+as e</w>
+re le
+htt ps</w>
+p ut</w>
+de le
+dele te</w>
+red is</w>
+do c
+c i</w>
+4 2</w>
+p i</w>
+fun c</w>
+non e</w>
+ge ti
+geti tem</w>
+c s
+cs v</w>
+u x</w>
+b fs</w>
+no de</w>
+cur r</w>
+visi ted</w>
+ne igh
+neigh b
+neighb or</w>
+5 7</w>
+6 5</w>
+we b</w>
+e t</w>
+to ke
+toke n</w>
+i on</w>
+jav asc
+javasc ript</w>
+8 0</w>
+flo at</w>
+p le</w>
+s g</w>
+fi l
+fil ter</w>
+o pen</w>
+rea d</w>
+ac e</w>
+u p
+p i
+pi vo
+pivo t</w>
+quick sort</w>
+coun ter</w>
+di r</w>
+str f
+strf time</w>
+cho ic
+me tho
+metho d</w>
+pro per
+b u
+v al
+en ter</w>
+i ter</w>
+m ul
+ss ing</w>
+i l</w>
+asynci o</w>
+ar t</w>
+1 50</w>
+12 3</w>
+go o
+4 0
+ou r</w>
+h 2
+h2 o</w>
+en ce</w>
+4 8</w>
+6 9</w>
+6 4</w>
+2 8</w>
+on e</w>
+a c</w>
+a d</w>
+a as</w>
+dee p</w>
+mo de
+ne ural</w>
+inter net</w>
+con ne
+c ts</w>
+on s</w>
+in stan
+comm un
+commun ic
+st s</w>
+e ar
+proce ss</w>
+d is
+k e</w>
+comp ut
+comput ing</w>
+se cur
+in es</w>
+fu ture</w>
+nat ural</w>
+gen er
+ne w</w>
+ti es</w>
+a to
+un der
+al lo
+allo ws</w>
+w are</w>
+v an
+p ip</w>
+re st
+rest ful</w>
+no sql</w>
+doc k
+dock er</w>
+or e</w>
+no t</w>
+arg s</w>
+re mo
+de fa
+defa ul
+re ver
+at t
+inf o</w>
+2 56</w>
+tion al</w>
+ac ti
+se d</w>
+pre v</w>
+a m</w>
+w t</w>
+y 2</w>
+7 2</w>
+2 7</w>
+rn a</w>
+p h</w>
+li ty</w>
+12 3
+123 45</w>
+s ig
+dro po
+dropo ut</w>
+c n
+cn n</w>
+e ff
+r l
+rl h
+rlh f</w>
+w ar
+jav a</w>
+bo ol</w>
+comp le
+comple x</w>
+po p</w>
+e li
+sq u
+g re
+do g</w>
+enu mer
+enumer ate</w>
+t re
+y ie
+yie ld</w>
+u d
+m at
+cl ass
+proper ty</w>
+ar e
+lo c
+glob al</w>
+ra ise</w>
+mul ti
+proce ssing</w>
+s to
+cr is
+cris pr</w>
+3 d</w>
+a r</w>
+5 3</w>
+6 7
+po w</w>
+19 3
+19 7
+2 2</w>
+cos θ</w>
+wa y</w>
+mach ine</w>
+networ ks</w>
+p u
+f o
+en ab
+ch an
+g rea
+te st</w>
+l eng
+l l</w>
+de s</w>
+en t</w>
+ear th</w>
+e ss
+ess enti
+essenti al</w>
+s is</w>
+li g
+h t</w>
+ch e
+che m
+per i
+an al
+anal y
+compu ters</w>
+stru c
+organ iz
+acce ss</w>
+con tr
+o ver</w>
+ap pro
+ac hi
+achi e
+re d</w>
+soft ware</w>
+tic s</w>
+s o</w>
+de c
+po st</w>
+d b</w>
+ss l</w>
+sc ore</w>
+ab c
+time delta</w>
+att r</w>
+ru n</w>
+us ers</w>
+th er</w>
+z one
+zone info</w>
+we a
+so l
+l c
+3 l</w>
+5 l</w>
+1 20</w>
+28 57</w>
+7 9
+re volu
+in e
+networ k</w>
+c rea
+m v
+mv c</w>
+we b
+web socket</w>
+j wt</w>
+x x</w>
+8 7
+4 3
+re s
+p tion</w>
+visi on</w>
+l as
+g pt</w>
+di ff
+m u
+k l</w>
+0 5
+05 b</w>
+b le
+3 2</w>
+al ph
+tu ple</w>
+eli f</w>
+l t</w>
+re pl
+repl ace</w>
+up per</w>
+v al</w>
+s wa
+swa p</w>
+n ²</w>
+de qu
+dequ e</w>
+sor ted</w>
+forma t</w>
+mat ch</w>
+se ar
+sear ch</w>
+cl s</w>
+are a</w>
+ob j</w>
+mo st</w>
+comm on</w>
+ch a
+cha in</w>
+co mb
+fin al
+final ly</w>
+rea ding</w>
+st art</w>
+0 1</w>
+2 9</w>
+18 0</w>
+6 0
+e n</w>
+ch ang
+10 24</w>
+goo d</w>
+we e
+wee k
+week day</w>
+log 10</w>
+19 53</w>
+sci ence</w>
+6 3
+7 1</w>
+8 6</w>
+g dp</w>
+19 8
+ht m
+p an
+4 ac</w>
+2 a</w>
+co s</w>
+b i</w>
+m ing</w>
+wor k</w>
+d ri
+c ar
+at ter
+l ar
+e ts</w>
+lear n</w>
+pre sen
+pro gra
+progra mm
+programm ing</w>
+sy n
+syste m</w>
+bi lli
+communic ation</w>
+chan ge</w>
+co o
+ce d</w>
+in cl
+f ru
+fru its</w>
+pro te
+so lar</w>
+pl an
+ar o
+aro und</w>
+co ver
+the sis</w>
+lig ht</w>
+in to</w>
+chem ical</w>
+g ro
+s er
+v ation</w>
+ut h</w>
+li ke</w>
+di f
+hel p</w>
+de ve
+deve lo
+sour ces</w>
+c y
+iz ed</w>
+cont a
+conta ins</w>
+appro x
+approx i
+approxi ma
+approxima te
+approximate ly</w>
+v ac
+spe ci
+w in
+h o
+it al</w>
+k i
+z e</w>
+gen e
+gene tic</w>
+li ving</w>
+organ is
+organis ms</w>
+bi o
+under st
+underst and</w>
+sour ce</w>
+c y</w>
+b ut</w>
+ou s</w>
+rele van
+relevan t</w>
+ur l</w>
+coun t</w>
+ex p</w>
+p ass</w>
+flo or</w>
+c ap
+is instan
+isinstan ce</w>
+ma in</w>
+ass er
+bas e
+s lo
+slo ts</w>
+func tools</w>
+re du
+par ti
+li b</w>
+log g
+logg ing</w>
+de r</w>
+sq li
+b in
+lin ux</w>
+g a
+d 1</w>
+d 2</w>
+re f</w>
+in de
+rever sed</w>
+m r
+se s</w>
+k mp</w>
+ar y</w>
+k o
+pr e</w>
+m n</w>
+d o</w>
+r h</w>
+n x</w>
+sin x</w>
+u v</w>
+d u</w>
+σ p</w>
+6 8</w>
+sin θ</w>
+5 x</w>
+7 5</w>
+ali ce</w>
+bo b</w>
+10 x</w>
+1 3
+4 5
+45 8</w>
+at p</w>
+e ver
+e ct</w>
+revolu tion
+pe op
+peop le</w>
+e lec
+grea t</w>
+lo ve</w>
+on l
+onl ine</w>
+ac h</w>
+ab i
+abi lity</w>
+sy n</w>
+9 87
+43 21</w>
+14 4</w>
+t an
+eff ici
+rn n</w>
+r u</w>
+v it</w>
+ber t</w>
+fi x</w>
+c li
+us ion</w>
+f 1</w>
+u p</w>
+ag o</w>
+g pu</w>
+ar es</w>
+gre et</w>
+m sg</w>
+squ are</w>
+an im
+anim al</w>
+w r
+sp li
+spli t</w>
+fi b</w>
+e q</w>
+st ud
+str ing</w>
+mp s</w>
+r s</w>
+choic e</w>
+loc al</w>
+val u
+th reading</w>
+g il</w>
+i te
+ite ms</w>
+an y</w>
+v r</w>
+i de</w>
+4 56</w>
+chang ing</w>
+mor ning</w>
+ma k
+time sta
+timesta mp</w>
+uni x</w>
+0 7</w>
+5 0
+di v
+div mod</w>
+m a</w>
+18 69</w>
+htm l</w>
+b x</w>
+b ²</w>
+δ x</w>
+b c</w>
+a 1</w>
+ic e</w>
+or i
+c an</w>
+i d
+p atter
+patter ns</w>
+lar ge</w>
+hu man
+m is
+mode ls</w>
+us e</w>
+presen t
+mp le</w>
+e co
+me ters</w>
+v ic
+vic es</w>
+l ing</w>
+g u
+im pro
+impro ves</w>
+re n
+y our</w>
+u s</w>
+incl u
+ge t
+con si
+su n</w>
+tic al</w>
+ab o
+abo ut</w>
+s ur
+ph o
+pho to
+photo syn
+photosyn thesis</w>
+con ver
+sci enti
+vol ves</w>
+co ver</w>
+qu an
+quan tu
+quantu m</w>
+tu res</w>
+syste ms</w>
+tr ack</w>
+co ll
+coll ab
+collab or
+clo u
+clou d</w>
+pro vi
+secur ity</w>
+b ra
+bra in</w>
+ne u
+neu ro
+neuro ns</w>
+i mm
+speci fic</w>
+di se
+dise as
+diseas es</w>
+inf ec
+ne w
+le e
+so li
+c ul
+man a
+mana ge
+manage ment</w>
+ob j
+c le
+ss ion</w>
+li ste
+se con
+secon d</w>
+u m</w>
+peri o
+t able</w>
+e le
+ato mi
+atomi c</w>
+in sp
+insp i
+enab les</w>
+mach ines</w>
+en co
+at ing</w>
+in no
+c re
+v ity</w>
+ac e
+hel ps</w>
+dec isi
+decisi ons</w>
+cont in
+u dp</w>
+s m
+re st</w>
+r s
+ac id</w>
+my sql</w>
+mon go
+mongo db</w>
+t ls</w>
+s 1</w>
+s 2</w>
+gra de</w>
+w n</w>
+to y</w>
+fi r
+lo ad</w>
+g cd</w>
+s a
+r a</w>
+defaul t
+default dict</w>
+d d</w>
+produ ct</w>
+lo w
+low er</w>
+asser t</w>
+h as
+redu ce</w>
+o ut
+h lib</w>
+en code</w>
+base 64</w>
+6 4
+cur s
+curs or</w>
+p ick
+pick le</w>
+ur lli
+urlli b</w>
+stru ct</w>
+ac l
+acl ass</w>
+ven v</w>
+my en
+myen v</w>
+requi re
+require ments</w>
+c ase</w>
+val ue</w>
+i a</w>
+g rap
+sub class</w>
+wea k
+weak ref</w>
+spe c</w>
+b y
+ro und</w>
+tr un
+trun c</w>
+inde x</w>
+leng th</w>
+mr o</w>
+bas es</w>
+c us
+cus to
+bu ff
+buff er</w>
+mer ge
+merge sort</w>
+te mp</w>
+b st</w>
+av l</w>
+tri e</w>
+di j
+dij k
+dijk str
+dijkstr a</w>
+k a
+lc s</w>
+b ac
+choic es</w>
+k ar
+kar p</w>
+ic k</w>
+m in
+bin ary</w>
+tar j
+tarj an</w>
+en ding</w>
+l z
+7 7</w>
+x 1
+2 ab</w>
+b a</w>
+cos x</w>
+x dx</w>
+7 18
+718 28</w>
+d t</w>
+9 7</w>
+2 l</w>
+10 a</w>
+10 b</w>
+8 4</w>
+10 00
+14 2857</w>
+4 7</w>
+2 k</w>
+19 4
+12 7
+2 9
+29 9
+299 79
+29979 2
+299792 458</w>
+0 00</w>
+3 65</w>
+in d
+ind us
+w as</w>
+revolution ized</w>
+c ation</w>
+elec tri
+electri c</w>
+di g
+dig ital</w>
+contr o
+ver s
+b us
+bus ine
+busine ss</w>
+wh er
+wher e</w>
+me as
+ce ll</w>
+ma g
+mag ne
+magne tic</w>
+h a
+be co
+o p</w>
+en cr
+encr y
+secur e</w>
+or m</w>
+o a
+oa uth</w>
+c d
+cd n</w>
+mode l</w>
+v ie
+vie w</w>
+l le
+40 4</w>
+5 00</w>
+6 5
+3 1</w>
+6 7</w>
+20 25</w>
+cal en
+calen da
+calenda r</w>
+mon day</w>
+re l
+rel u</w>
+sig mo
+sigmo id</w>
+tan h</w>
+res net</w>
+effici en
+efficien t
+se q</w>
+soft max</w>
+f las
+d p
+ie ce</w>
+pre fix</w>
+t un
+tun ing</w>
+l 2</w>
+ada m
+adam w</w>
+ble u</w>
+ro u
+f p
+r m
+ada m</w>
+m b</w>
+alph ago</w>
+u se
+squ ares</w>
+c at</w>
+re su
+resu lt</w>
+z e
+ze ro
+zero di
+zerodi visi
+zerodivisi on
+zerodivision error</w>
+con ten
+conten t</w>
+str ip</w>
+lif o</w>
+fi f
+fif o</w>
+ti mer</w>
+r b</w>
+du mps</w>
+lo ad
+load s</w>
+li st
+list dir</w>
+ma ke
+make di
+makedi rs</w>
+ran d
+rand int</w>
+fin d
+find all</w>
+dee p
+deep copy</w>
+ab b
+comb in
+combin ations</w>
+per m
+perm ut
+permut ations</w>
+s ing</w>
+bu i
+non local</w>
+exce ption</w>
+i o</w>
+multi processing</w>
+ste p</w>
+ke ys</w>
+valu es</w>
+f al
+fal se</w>
+3 50</w>
+3 4</w>
+g b
+gb ps</w>
+5 60
+560 8
+5608 8</w>
+z h</w>
+7 8</w>
+mak es</w>
+be t
+bet ter</w>
+50 50</w>
+b on
+to mor
+tomor ro
+tomorro w</w>
+ra di
+mon th</w>
+193 9</w>
+197 8</w>
+8 8
+88 48</w>
+h c
+hc l</w>
+na o
+nao h</w>
+n ac
+nac l</w>
+co 2</w>
+o 2</w>
+63 00</w>
+5 4
+54 64</w>
+m c
+19 28</w>
+20 0
+200 3</w>
+198 9</w>
+19 56</w>
+2 πr</w>
+t an</w>
+x i</w>
+s n</w>
+6 18</w>
+s ss</w>
+s as</w>
+as a</w>
+σ c</w>
+transfor ming</w>
+al g
+alg ori
+algori th
+algorith ms</w>
+dat as
+datas ets</w>
+multi ple</w>
+la y
+po pu
+popu lar</w>
+ag es</w>
+ic h</w>
+func tion</w>
+ke y
+par a
+conne cts</w>
+billi ons</w>
+de vices</w>
+wor l
+worl d
+world wi
+worldwi de</w>
+enab ling</w>
+sh ar
+cli m
+clim ate</w>
+ch al
+chal leng
+challeng es</w>
+fac ing</w>
+human ity</w>
+coo per
+cooper ation</w>
+o v
+u lar</w>
+he al
+heal th</w>
+en s</w>
+bo o
+t al</w>
+ex pan
+expan ds</w>
+con c
+en tr
+inclu des</w>
+ti m
+e ight</w>
+pa th
+wa ter</w>
+sur fac
+surfac e</w>
+for ms</w>
+scienti fic</w>
+in volves</w>
+ser vation</w>
+h y
+ex peri
+dis cover</w>
+sta tes</w>
+o us
+ous ly</w>
+po si
+ra ys</w>
+mo dif
+contr ol</w>
+provi des</w>
+de m
+dem and</w>
+re sources</w>
+cy ber
+cyber security</w>
+prote cts</w>
+au th
+billi on</w>
+vac c
+tra in</w>
+imm un
+immun e</w>
+re new
+renew able</w>
+win d</w>
+ci al</w>
+su sta
+susta in
+sustain able</w>
+the or
+theor y</w>
+e volu
+evolu tion</w>
+se lec
+selec tion</w>
+ex p
+exp la
+expla ins</w>
+ho w</w>
+man y</w>
+gener ations</w>
+s lee
+slee p</w>
+ve n</w>
+da tion</w>
+ph y
+phy s
+diff e
+diffe ren
+differen t</w>
+cul tures</w>
+uni ties</w>
+ll s</w>
+pr i
+achie ve</w>
+mor e</w>
+le ss</w>
+th in
+k ing</w>
+obj ec
+ve ly</w>
+eff ec
+effec tive</w>
+requi res</w>
+bo th</w>
+cle ar</w>
+liste ning</w>
+o th
+oth ers</w>
+o ce
+oce an</w>
+ge st</w>
+cover ing</w>
+th re
+thre e</w>
+for m</w>
+fo un
+class ical</w>
+ic s</w>
+tho us
+thous and</w>
+vac u
+vacu um</w>
+perio di
+periodi c</w>
+organiz es</w>
+ele ments</w>
+the i
+thei r</w>
+nu mb
+numb er</w>
+car ri
+carri es</w>
+in struc
+instruc tions</w>
+develo p
+develop ment</w>
+inspi red</w>
+bio log
+biolog ical</w>
+inter pre
+interpre t</w>
+v is
+vis u
+visu al</w>
+re al</w>
+inno vation</w>
+au to
+ma tion</w>
+ro bo
+robo tics</w>
+in cre
+ti vity</w>
+bi g</w>
+analy tics</w>
+organiz ations</w>
+ma ke</w>
+contin u
+continu ous</w>
+rap id
+rapid ly</w>
+comm it</w>
+p us
+pus h</w>
+a es</w>
+rs a</w>
+c ss</w>
+de vo
+devo ps</w>
+no ht
+noht y
+nohty p</w>
+li c
+bas e</w>
+k w
+remo ve</w>
+stud ent</w>
+su m</w>
+enco ding</w>
+clo se</w>
+my error</w>
+su per</w>
+f ul
+ato r</w>
+wr app
+wrapp er</w>
+du mp</w>
+str p
+strp time</w>
+ce il</w>
+na me
+name d
+named tuple</w>
+rever se</w>
+fin d</w>
+t s
+my class</w>
+cal l</w>
+re pr</w>
+my list</w>
+l ru</w>
+c ach
+cach e</w>
+parti al</w>
+pow 2</w>
+par se</w>
+l r</w>
+sub process</w>
+has hlib</w>
+sh a
+sha 256</w>
+b 64
+rea der</w>
+sqli te
+sqlite 3</w>
+sqli te</w>
+con n</w>
+conne ct</w>
+ut e</w>
+p ack</w>
+ab c</w>
+sh ap
+dat aclass</w>
+enu m</w>
+co lo
+colo r</w>
+ty p
+typ ing</w>
+op tional</w>
+uni on</w>
+mo du
+modu le</w>
+acti v
+activ ate</w>
+p ts</w>
+win do
+windo ws</w>
+in st
+inst all</w>
+do c</w>
+py test</w>
+a i
+res p</w>
+c or
+remo ve
+as ia</w>
+sh ang
+shang h
+shangh ai</w>
+de l</w>
+by tes</w>
+my int</w>
+h int</w>
+ab s</w>
+di v</w>
+at tri
+w rap
+pe d</w>
+defaul ts</w>
+al s</w>
+sol u
+ch ec
+chec k</w>
+d ct</w>
+hi ft</w>
+rele ase</w>
+a enter</w>
+a exit</w>
+a iter</w>
+an e
+ane xt</w>
+parti tion</w>
+mer ge</w>
+flo y
+floy d</w>
+grap h</w>
+for d</w>
+ka h
+kah n</w>
+m st</w>
+k r
+kr us
+krus k
+krusk al</w>
+li s</w>
+bac k
+back track</w>
+f en
+tre e</w>
+min i
+mini max</w>
+alph a</w>
+be ta</w>
+lc a</w>
+sc c</w>
+e d
+i c</w>
+man ach
+manach er</w>
+dif f</w>
+n i
+fa r</w>
+u sed</w>
+m as
+ga p</w>
+tr o
+a a
+a a</w>
+lz 77</w>
+x1 x2</w>
+b n</w>
+θ r</w>
+3 πr</w>
+2 h</w>
+c u</w>
+2 σ</w>
+i j</w>
+de t</w>
+λ x</w>
+k 1</w>
+k 2</w>
+2 p
+h 0</w>
+m ar
+i θ</w>
+n θ</w>
+4 l</w>
+9 4</w>
+4 y</w>
+8 00</w>
+2 b</w>
+2 1
+21 k</w>
+6 x</w>
+3 00</w>
+5 d</w>
+4 8
+1 abc
+1abc de</w>
+abc de
+abcde 1</w>
+1000 00</w>
+4 2857</w>
+28 0</w>
+4 28
+5 v</w>
+19 12</w>
+194 9</w>
+m rna</w>
+3 7</w>
+8 60</w>
+197 4</w>
+im f</w>
+20 15</w>
+g le</w>
+13 8</w>
+0 2</w>
+sin θ
+m r</w>
+5 9</w>
+9 6</w>
+198 7</w>
+20 00</w>
+8 9</w>
+193 7</w>
+19 90</w>
+20 16</w>
+ever y</w>
+pr acti
+practi ce</w>
+per f
+perf ect</w>
+bo dy</w>
+a mo
+be st</w>
+t w
+ne y</w>
+communic ate</w>
+ali z
+la tes</w>
+ne ver</w>
+wh at</w>
+go ing</w>
+e du
+edu cation</w>
+ll ing</w>
+sen si
+sensi tive</w>
+eng ine
+engine er
+engineer ing</w>
+ss i
+me dic
+sp ace</w>
+under stan
+understan ding</w>
+ever y
+rea lity</w>
+crea tes</w>
+o ver
+mo us</w>
+ve h
+veh ic
+vehic les</w>
+wi tho
+witho ut</w>
+produ cts</w>
+s cal
+scal e</w>
+mu ch</w>
+to war
+towar d</w>
+di vers
+divers ity</w>
+e mo
+conne c
+eco no
+econo mi
+economi es</w>
+pl at
+de d</w>
+la tion</w>
+s tic</w>
+produ ced</w>
+ter i
+tr ac
+trac ts</w>
+e ach</w>
+o ther</w>
+conver ting</w>
+tur al</w>
+b le</w>
+c an
+crea ted</w>
+indus tri
+industri al</w>
+transfor me
+transforme d</w>
+pr in
+prin ting</w>
+ha ve</w>
+beco me</w>
+ma de</w>
+remo te</w>
+ad van
+advan ces</w>
+sp ace
+space x</w>
+fi les</w>
+an y
+pa y
+v p
+vp n</w>
+i aas</w>
+pa as</w>
+s aas</w>
+s wi
+swi ft</w>
+ko t
+kot li
+kotli n</w>
+c as
+cas 9</w>
+r y</w>
+b ran
+bran ch</w>
+c ass
+cass an
+cassan d
+cassand ra</w>
+n g
+ng in
+ngin x</w>
+sm art</w>
+meas ur
+measur able</w>
+achie v
+achiev able</w>
+bo und</w>
+present ation</w>
+c a
+ca p</w>
+consi ste
+consiste n
+consisten cy</w>
+contro lle
+controlle r</w>
+soli d</w>
+67 8
+678 90</w>
+987 65
+98765 4321</w>
+4 1</w>
+4 2
+0 7
+6 1</w>
+wea ther</w>
+be au
+beau ti
+beauti ful</w>
+e as
+eas y</w>
+un ti
+unti l</w>
+coo ki
+cooki e</w>
+se ssion</w>
+x ss</w>
+a op</w>
+987 65</w>
+3 5
+9 99</w>
+12 8</w>

modeling_stellarai.py

@@ -0,0 +1,610 @@
+# coding=utf-8
+"""
+StellarAI: Lightweight Multimodal Large Language Model
+Hugging Face compatible implementation
+"""
+
+import math
+from typing import Optional, Tuple, Dict, Any, List
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from transformers import PreTrainedModel, GenerationConfig, GenerationMixin
+from transformers.modeling_outputs import (
+    CausalLMOutputWithPast,
+    BaseModelOutputWithPast,
+)
+from transformers.utils import add_start_docstrings, logging
+
+from configuration_stellarai import StellarAIConfig
+
+logger = logging.get_logger(__name__)
+
+_CONFIG_FOR_DOC = "StellarAIConfig"
+_CHECKPOINT_FOR_DOC = "StellarAI/stellarai-tiny"
+
+STELLARAI_START_DOCSTRING = r"""
+    StellarAI Model: A lightweight multimodal large language model.
+
+    Parameters:
+        config ([`StellarAIConfig`]): Model configuration class with all the parameters.
+"""
+
+
+# ============================================================
+#  RoPE Rotary Position Embedding
+# ============================================================
+class RoPE(nn.Module):
+    """Rotary Position Embedding"""
+
+    def __init__(self, head_dim: int, max_seq_len: int = 2048, base: float = 10000.0):
+        super().__init__()
+        self.head_dim = head_dim
+        self.base = base
+        half = head_dim // 2
+        inv_freq = 1.0 / (base ** (torch.arange(0, head_dim, 2).float() / head_dim))
+        positions = torch.arange(max_seq_len).float()
+        angles = torch.outer(positions, inv_freq)
+        cos = torch.zeros(max_seq_len, head_dim)
+        sin = torch.zeros(max_seq_len, head_dim)
+        cos[:, 0::2] = torch.cos(angles)
+        cos[:, 1::2] = torch.cos(angles)
+        sin[:, 0::2] = torch.sin(angles)
+        sin[:, 1::2] = torch.sin(angles)
+        self.register_buffer("cos", cos, persistent=False)
+        self.register_buffer("sin", sin, persistent=False)
+
+    def forward(self, x: torch.Tensor, offset: int = 0) -> torch.Tensor:
+        *_, S, Dh = x.shape
+        cos = self.cos[offset:offset + S]
+        sin = self.sin[offset:offset + S]
+        x1 = x[..., 0::2]
+        x2 = x[..., 1::2]
+        cos_half = cos[:, 0::2]
+        sin_half = sin[:, 0::2]
+        while cos_half.dim() < x1.dim():
+            cos_half = cos_half.unsqueeze(0)
+            sin_half = sin_half.unsqueeze(0)
+        out1 = x1 * cos_half - x2 * sin_half
+        out2 = x1 * sin_half + x2 * cos_half
+        result = torch.empty_like(x)
+        result[..., 0::2] = out1
+        result[..., 1::2] = out2
+        return result
+
+
+# ============================================================
+#  Multi-Head Self Attention
+# ============================================================
+class MultiHeadAttention(nn.Module):
+    def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1, rope: Optional[RoPE] = None):
+        super().__init__()
+        assert d_model % num_heads == 0
+        self.d_model = d_model
+        self.num_heads = num_heads
+        self.head_dim = d_model // num_heads
+        self.scale = 1.0 / math.sqrt(self.head_dim)
+        self.Wq = nn.Linear(d_model, d_model, bias=True)
+        self.Wk = nn.Linear(d_model, d_model, bias=True)
+        self.Wv = nn.Linear(d_model, d_model, bias=True)
+        self.Wo = nn.Linear(d_model, d_model, bias=True)
+        self.dropout = nn.Dropout(dropout)
+        self.rope = rope
+
+    def forward(
+        self, x: torch.Tensor, mask: Optional[torch.Tensor] = None,
+        kv_cache: Optional[Dict] = None,
+    ) -> Tuple[torch.Tensor, Optional[Dict]]:
+        B, S, D = x.shape
+        Q = self.Wq(x).view(B, S, self.num_heads, self.head_dim).transpose(1, 2)
+        K = self.Wk(x).view(B, S, self.num_heads, self.head_dim).transpose(1, 2)
+        V = self.Wv(x).view(B, S, self.num_heads, self.head_dim).transpose(1, 2)
+
+        offset = 0
+        if kv_cache is not None and "K" in kv_cache:
+            offset = kv_cache["K"].shape[2]
+        if self.rope is not None:
+            Q = self.rope(Q.transpose(1, 2), offset=offset).transpose(1, 2)
+            K = self.rope(K.transpose(1, 2), offset=offset).transpose(1, 2)
+
+        if kv_cache is not None:
+            if "K" in kv_cache:
+                K = torch.cat([kv_cache["K"], K], dim=2)
+                V = torch.cat([kv_cache["V"], V], dim=2)
+            kv_cache["K"] = K
+            kv_cache["V"] = V
+
+        scores = torch.matmul(Q, K.transpose(-2, -1)) * self.scale
+        Sq = Q.shape[2]
+        Sk = K.shape[2]
+        if mask is not None:
+            scores = scores.masked_fill(mask[:Sq, :Sk], float('-inf'))
+        else:
+            causal = torch.triu(torch.ones(Sq, Sk, device=x.device, dtype=torch.bool), diagonal=1)
+            scores = scores.masked_fill(causal, float('-inf'))
+
+        attn = F.softmax(scores, dim=-1)
+        attn = self.dropout(attn)
+        out = torch.matmul(attn, V)
+        out = out.transpose(1, 2).contiguous().view(B, S, D)
+        return self.Wo(out), kv_cache
+
+
+# ============================================================
+#  Cross Attention
+# ============================================================
+class CrossAttention(nn.Module):
+    def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1):
+        super().__init__()
+        assert d_model % num_heads == 0
+        self.d_model = d_model
+        self.num_heads = num_heads
+        self.head_dim = d_model // num_heads
+        self.scale = 1.0 / math.sqrt(self.head_dim)
+        self.Wq = nn.Linear(d_model, d_model, bias=True)
+        self.Wk = nn.Linear(d_model, d_model, bias=True)
+        self.Wv = nn.Linear(d_model, d_model, bias=True)
+        self.Wo = nn.Linear(d_model, d_model, bias=True)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, x: torch.Tensor, ctx: torch.Tensor) -> torch.Tensor:
+        B, Sx, D = x.shape
+        Sc = ctx.shape[1]
+        Q = self.Wq(x).view(B, Sx, self.num_heads, self.head_dim).transpose(1, 2)
+        K = self.Wk(ctx).view(B, Sc, self.num_heads, self.head_dim).transpose(1, 2)
+        V = self.Wv(ctx).view(B, Sc, self.num_heads, self.head_dim).transpose(1, 2)
+        scores = torch.matmul(Q, K.transpose(-2, -1)) * self.scale
+        attn = F.softmax(scores, dim=-1)
+        attn = self.dropout(attn)
+        out = torch.matmul(attn, V)
+        out = out.transpose(1, 2).contiguous().view(B, Sx, D)
+        return self.Wo(out)
+
+
+# ============================================================
+#  Transformer Block (Pre-LN)
+# ============================================================
+class TransformerBlock(nn.Module):
+    def __init__(self, d_model: int, num_heads: int, ff_dim: int,
+                 dropout: float = 0.1, rope: Optional[RoPE] = None, eps: float = 1e-6):
+        super().__init__()
+        self.attn = MultiHeadAttention(d_model, num_heads, dropout, rope)
+        self.ffn = nn.Sequential(
+            nn.Linear(d_model, ff_dim),
+            nn.GELU(),
+            nn.Linear(ff_dim, d_model),
+        )
+        self.ln1 = nn.LayerNorm(d_model, eps=eps)
+        self.ln2 = nn.LayerNorm(d_model, eps=eps)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(
+        self, x: torch.Tensor, mask: Optional[torch.Tensor] = None,
+        kv_cache: Optional[Dict] = None,
+    ) -> Tuple[torch.Tensor, Optional[Dict]]:
+        attn_out, kv_cache = self.attn(self.ln1(x), mask, kv_cache)
+        x = x + self.dropout(attn_out)
+        x = x + self.dropout(self.ffn(self.ln2(x)))
+        return x, kv_cache
+
+
+# ============================================================
+#  Fusion Block (Self + Cross Attention)
+# ============================================================
+class FusionBlock(nn.Module):
+    def __init__(self, d_model: int, num_heads: int, ff_dim: int,
+                 dropout: float = 0.1, rope: Optional[RoPE] = None, eps: float = 1e-6):
+        super().__init__()
+        self.self_attn = MultiHeadAttention(d_model, num_heads, dropout, rope)
+        self.cross_attn = CrossAttention(d_model, num_heads, dropout)
+        self.ffn = nn.Sequential(
+            nn.Linear(d_model, ff_dim),
+            nn.GELU(),
+            nn.Linear(ff_dim, d_model),
+        )
+        self.ln1 = nn.LayerNorm(d_model, eps=eps)
+        self.ln2 = nn.LayerNorm(d_model, eps=eps)
+        self.ln3 = nn.LayerNorm(d_model, eps=eps)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(
+        self, x: torch.Tensor, ctx: Optional[torch.Tensor],
+        mask: Optional[torch.Tensor] = None, kv_cache: Optional[Dict] = None,
+    ) -> Tuple[torch.Tensor, Optional[Dict]]:
+        attn_out, kv_cache = self.self_attn(self.ln1(x), mask, kv_cache)
+        x = x + self.dropout(attn_out)
+        if ctx is not None:
+            x = x + self.dropout(self.cross_attn(self.ln2(x), ctx))
+        x = x + self.dropout(self.ffn(self.ln3(x)))
+        return x, kv_cache
+
+
+# ============================================================
+#  Vision Encoder (CNN + ViT Hybrid)
+# ============================================================
+class VisionEncoder(nn.Module):
+    def __init__(self, cfg: StellarAIConfig):
+        super().__init__()
+        self.cfg = cfg
+        vc = cfg.vision_cfg
+        D = cfg.d_model
+
+        layers = []
+        c_in = vc["num_channels"]
+        for c_out in vc["cnn_channels"]:
+            layers.append(nn.Conv2d(c_in, c_out, 3, stride=2, padding=1))
+            layers.append(nn.BatchNorm2d(c_out))
+            layers.append(nn.SiLU())
+            c_in = c_out
+        self.cnn = nn.Sequential(*layers)
+        self.cnn_proj = nn.Linear(c_in, D)
+
+        self.pos_emb = nn.Parameter(torch.randn(vc["vision_num_patches"], D) * 0.02)
+
+        self.blocks = nn.ModuleList([
+            TransformerBlock(D, vc["vision_num_heads"], vc["vision_ff_dim"],
+                             cfg.dropout, rope=None, eps=cfg.layer_norm_eps)
+            for _ in range(vc["vision_num_layers"])
+        ])
+        self.final_ln = nn.LayerNorm(D, eps=cfg.layer_norm_eps)
+
+    def forward(self, images: torch.Tensor) -> torch.Tensor:
+        x = self.cnn(images)
+        B, C, Hp, Wp = x.shape
+        target = int(math.sqrt(self.cfg.vision_cfg["vision_num_patches"]))
+        if Hp != target:
+            x = F.adaptive_avg_pool2d(x, (target, target))
+        x = x.flatten(2).transpose(1, 2)
+        x = self.cnn_proj(x)
+        x = x + self.pos_emb.unsqueeze(0)
+        for blk in self.blocks:
+            x, _ = blk(x, mask=None)
+        return self.final_ln(x)
+
+
+# ============================================================
+#  StellarAI Core Model
+# ============================================================
+@add_start_docstrings(
+    "The bare StellarAI Model outputting raw hidden-states.",
+    STELLARAI_START_DOCSTRING,
+)
+class StellarAIModel(PreTrainedModel):
+    config_class = StellarAIConfig
+    base_model_prefix = "model"
+    _no_split_modules = ["TransformerBlock", "FusionBlock"]
+
+    def __init__(self, config: StellarAIConfig):
+        super().__init__(config)
+        self.config = config
+        D = config.d_model
+        fc = config.mm_fusion_cfg
+
+        self.wte = nn.Embedding(config.vocab_size, D)
+        self.text_rope = RoPE(D // config.num_attention_heads,
+                              config.max_position_embeddings, config.rope_theta)
+
+        self.text_blocks = nn.ModuleList([
+            TransformerBlock(D, config.num_attention_heads, config.intermediate_size,
+                             config.dropout, rope=self.text_rope, eps=config.layer_norm_eps)
+            for _ in range(config.num_hidden_layers)
+        ])
+        self.text_final_ln = nn.LayerNorm(D, eps=config.layer_norm_eps)
+
+        self.vision_encoder = VisionEncoder(config)
+
+        self.fusion_blocks = nn.ModuleList([
+            FusionBlock(D, fc["fusion_num_heads"], fc["fusion_ff_dim"],
+                        config.dropout, rope=self.text_rope, eps=config.layer_norm_eps)
+            for _ in range(fc["fusion_num_layers"])
+        ])
+        self.fusion_final_ln = nn.LayerNorm(D, eps=config.layer_norm_eps)
+
+        self.gradient_checkpointing = False
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.wte
+
+    def set_input_embeddings(self, value):
+        self.wte = value
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        images: Optional[torch.FloatTensor] = None,
+        image_positions: Optional[List] = None,
+        use_cache: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> BaseModelOutputWithPast:
+        output_hidden_states = output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("Cannot specify both input_ids and inputs_embeds")
+        if input_ids is None and inputs_embeds is None:
+            raise ValueError("Must specify either input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.wte(input_ids)
+
+        B, S, D = inputs_embeds.shape
+        device = inputs_embeds.device
+
+        # Attention mask -> causal mask
+        if attention_mask is not None and attention_mask.dim() == 2:
+            mask = ~(attention_mask[:, None, None, :].bool())
+        else:
+            mask = None
+
+        # Vision encoding
+        vision_features = None
+        if images is not None:
+            vision_features = self.vision_encoder(images)
+            if image_positions is not None:
+                for b in range(B):
+                    if b < len(image_positions):
+                        s, e = image_positions[b]
+                        n = min(e - s, vision_features.shape[1])
+                        inputs_embeds[b, s:s + n] = vision_features[b, :n]
+
+        # Text Transformer
+        # HF may pass DynamicCache (not subscriptable). Our layers use dict-per-layer format.
+        # Safest: if use_cache is requested but past_key_values is not a plain sequence of
+        # layer-level cache dicts, fall back to no-cache by treating past as all-None and
+        # still producing present_kv (the dict-style) so the next iteration can consume it.
+        if past_key_values is not None:
+            # Accept both tuple[dict, ...] and list[dict, ...]; reject DynamicCache & friends.
+            if not isinstance(past_key_values, (tuple, list)) or not all(
+                (isinstance(x, dict) or x is None) for x in past_key_values
+            ):
+                past_key_values = None
+
+        x = inputs_embeds
+        all_hidden_states = () if output_hidden_states else None
+        present_kv = [] if use_cache else None
+
+        past = past_key_values if past_key_values is not None else [None] * (
+            self.config.num_hidden_layers + self.config.mm_fusion_cfg["fusion_num_layers"]
+        )
+
+        for i, blk in enumerate(self.text_blocks):
+            if output_hidden_states:
+                all_hidden_states += (x,)
+            kv = past[i] if use_cache else None
+            if use_cache and kv is None:
+                kv = {}
+            x, kv_out = blk(x, mask, kv_cache=kv)
+            if use_cache:
+                present_kv.append(kv_out)
+
+        x = self.text_final_ln(x)
+
+        # Fusion layers
+        offset = self.config.num_hidden_layers
+        ctx = vision_features
+        for i, fblk in enumerate(self.fusion_blocks):
+            if output_hidden_states:
+                all_hidden_states += (x,)
+            kv = past[offset + i] if use_cache else None
+            if use_cache and kv is None:
+                kv = {}
+            x, kv_out = fblk(x, ctx, mask, kv_cache=kv)
+            if use_cache:
+                present_kv.append(kv_out)
+
+        x = self.fusion_final_ln(x)
+
+        if output_hidden_states:
+            all_hidden_states += (x,)
+
+        if not return_dict:
+            return tuple(v for v in [x, tuple(present_kv) if use_cache else None, all_hidden_states, None] if v is not None)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=x,
+            past_key_values=tuple(present_kv) if use_cache else None,
+            hidden_states=all_hidden_states,
+            attentions=None,
+        )
+
+
+# ============================================================
+#  StellarAI for Causal Language Modeling
+# ============================================================
+class StellarAIForCausalLM(PreTrainedModel, GenerationMixin):
+    r"""
+    StellarAI Model with a language modeling head on top (for causal LM).
+    """
+    config_class = StellarAIConfig
+    base_model_prefix = "model"
+    _no_split_modules = ["TransformerBlock", "FusionBlock"]
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config: StellarAIConfig):
+        super().__init__(config)
+        self.model = StellarAIModel(config)
+        D = config.d_model
+        self.lm_bias = nn.Parameter(torch.zeros(config.vocab_size)) if config.lm_head_bias else None
+        self.post_init()
+
+    def get_output_embeddings(self):
+        # Weight tying: output shares embedding weights
+        return self.model.wte
+
+    def set_output_embeddings(self, new_embeddings):
+        self.model.wte = new_embeddings
+
+    def get_input_embeddings(self):
+        return self.model.wte
+
+    def set_input_embeddings(self, value):
+        self.model.wte = value
+
+    def tie_weights(self, recompute_mapping: bool = False, missing_keys=None, **kwargs):
+        """Weight tying: LM head shares embedding weights"""
+        if self.config.use_weight_tying:
+            # weight is implicitly shared since we use F.linear(model.wte.weight)
+            pass
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        images: Optional[torch.FloatTensor] = None,
+        image_positions: Optional[List] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> CausalLMOutputWithPast:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            images=images,
+            image_positions=image_positions,
+            use_cache=use_cache,
+            output_hidden_states=output_hidden_states,
+            output_attentions=output_attentions,
+            return_dict=return_dict,
+        )
+
+        hidden_states = outputs[0]
+        logits = F.linear(hidden_states, self.model.wte.weight)
+        if self.lm_bias is not None:
+            logits = logits + self.lm_bias
+
+        loss = None
+        if labels is not None:
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            loss = F.cross_entropy(
+                shift_logits.view(-1, self.config.vocab_size),
+                shift_labels.view(-1),
+                ignore_index=-100,
+            )
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    @torch.no_grad()
+    def generate_text(
+        self,
+        prompt: str,
+        tokenizer=None,
+        max_new_tokens: int = 100,
+        temperature: float = 0.7,
+        top_k: int = 40,
+        device: torch.device = None,
+    ) -> str:
+        """Convenience method for text generation."""
+        if tokenizer is None:
+            raise ValueError("tokenizer is required for generate_text()")
+        if device is None:
+            device = next(self.parameters()).device
+        self.eval()
+
+        ids = tokenizer.encode(prompt)
+        # Ensure [BOS] at start if using SimpleTokenizer
+        if hasattr(tokenizer, 'bos_id') and ids[0] != tokenizer.bos_id:
+            ids = [tokenizer.bos_id] + ids
+        input_ids = torch.tensor([ids], dtype=torch.long, device=device)
+        eos_id = tokenizer.eos_id if hasattr(tokenizer, 'eos_id') else 2
+
+        for _ in range(max_new_tokens):
+            if input_ids.shape[1] >= self.config.max_position_embeddings:
+                break
+            out = self.forward(input_ids)
+            logits = out.logits[:, -1, :self.config.vocab_size]
+            if temperature <= 0:
+                next_id = logits.argmax(dim=-1, keepdim=True)
+            else:
+                logits = logits / temperature
+                v, _ = torch.topk(logits, min(top_k, logits.shape[-1]), dim=-1)
+                logits[logits < v[:, [-1]]] = float('-inf')
+                probs = F.softmax(logits, dim=-1)
+                next_id = torch.multinomial(probs, num_samples=1)
+            input_ids = torch.cat([input_ids, next_id], dim=1)
+            if next_id.item() == eos_id:
+                break
+
+        result_ids = input_ids[0].tolist()
+        return tokenizer.decode(result_ids, skip_special=True) if hasattr(tokenizer, 'decode') else str(result_ids)
+
+    # ===== HuggingFace GenerationMixin required hooks =====
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        images=None,
+        image_positions=None,
+        **kwargs,
+    ):
+        """Transform generation-time inputs into forward() arguments. Handles KV cache truncation."""
+        if past_key_values is not None:
+            # Only pass the last token as input; KV holds earlier context
+            past_length = 0
+            try:
+                # past_key_values is tuple of kv_cache dicts, each has K tensor shape (B, H, S, Dh)
+                first_cache = past_key_values[0]
+                if isinstance(first_cache, dict) and "K" in first_cache:
+                    past_length = first_cache["K"].shape[2]
+            except Exception:
+                past_length = 0
+            input_ids = input_ids[:, past_length:]
+
+        return {
+            "input_ids": input_ids if inputs_embeds is None else None,
+            "inputs_embeds": inputs_embeds,
+            "past_key_values": past_key_values,
+            "attention_mask": attention_mask,
+            "images": images,
+            "image_positions": image_positions,
+            "use_cache": False,
+        }
+
+    @staticmethod
+    def _reorder_cache(past_key_values, beam_idx):
+        """Reorder KV cache entries for beam search (no-op for greedy / sampling single-beam)."""
+        if past_key_values is None:
+            return None
+        reordered = []
+        for layer_cache in past_key_values:
+            if isinstance(layer_cache, dict):
+                new_cache = {}
+                for k, v in layer_cache.items():
+                    if isinstance(v, torch.Tensor):
+                        new_cache[k] = v.index_select(0, beam_idx.to(v.device))
+                    else:
+                        new_cache[k] = v
+                reordered.append(new_cache)
+            else:
+                reordered.append(layer_cache)
+        return tuple(reordered)

special_tokens_map.json

@@ -0,0 +1,14 @@
+{
+  "bos_token": "[BOS]",
+  "eos_token": "[EOS]",
+  "unk_token": "[UNK]",
+  "sep_token": "[SEP]",
+  "pad_token": "[PAD]",
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "additional_special_tokens": [
+    "[IMG]",
+    "[BOI]",
+    "[EOI]"
+  ]
+}

tokenization_stellarai.py

@@ -0,0 +1,268 @@
+# coding=utf-8
+"""
+StellarAI Tokenizer - Hugging Face compatible
+Wraps SimpleTokenizer to match PreTrainedTokenizer interface.
+"""
+import os
+import json
+from typing import List, Optional, Dict, Tuple, Union, Any
+
+from transformers import PreTrainedTokenizer
+from transformers.tokenization_utils import AddedToken
+
+# Import SimpleTokenizer from sibling stellarai package
+import sys
+_CURDIR = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(_CURDIR, ".."))
+from stellarai.tokenizer import SimpleTokenizer as _StellarTokenizer
+sys.path.pop(0)
+
+VOCAB_FILES_NAMES = {
+    "vocab_file": "backend_tokenizer.json",
+}
+
+PRETRAINED_VOCAB_FILES_MAP = {}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
+    "stellarai-tiny": 1024,
+}
+PRETRAINED_INIT_CONFIGURATION = {}
+
+
+def _find_backend_vocab(search_path: Optional[str]) -> Optional[str]:
+    """Locate the SimpleTokenizer format JSON file (backend_tokenizer.json)."""
+    candidates = []
+    if search_path is not None:
+        if os.path.isfile(search_path):
+            return search_path
+        if os.path.isdir(search_path):
+            candidates.append(os.path.join(search_path, "backend_tokenizer.json"))
+            candidates.append(os.path.join(search_path, "tokenizer.json"))
+    # default: same directory as this file
+    candidates.append(os.path.join(_CURDIR, "backend_tokenizer.json"))
+    candidates.append(os.path.join(_CURDIR, "..", "outputs", "stellar_pt", "tokenizer.json"))
+    for c in candidates:
+        if c and os.path.isfile(c):
+            # Make sure it's the SimpleTokenizer format (has "token_to_id")
+            try:
+                with open(c, "r", encoding="utf-8") as f:
+                    head = f.read(256)
+                    if '"token_to_id"' in head or "'token_to_id'" in head:
+                        return c
+            except Exception:
+                continue
+    return None
+
+
+class StellarAITokenizer(PreTrainedTokenizer):
+    """
+    Hugging Face compatible tokenizer for StellarAI.
+    Wraps the original SimpleTokenizer for 100% training-consistent tokenization.
+    """
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["input_ids", "attention_mask"]
+
+    def __init__(
+        self,
+        vocab_file=None,
+        unk_token="[UNK]",
+        bos_token="[BOS]",
+        eos_token="[EOS]",
+        sep_token="[SEP]",
+        pad_token="[PAD]",
+        cls_token="[CLS]",
+        mask_token="[MASK]",
+        additional_special_tokens=None,
+        model_max_length=1024,
+        do_lower_case=False,
+        **kwargs,
+    ):
+        if additional_special_tokens is None:
+            additional_special_tokens = ["[IMG]", "[BOI]", "[EOI]"]
+        # Wrap mask_token to AddedToken for HF compatibility
+        mask_token = AddedToken(mask_token, lstrip=False, rstrip=False) if isinstance(mask_token, str) else mask_token
+
+        # === IMPORTANT: create backend BEFORE super().__init__() ===
+        resolved = _find_backend_vocab(vocab_file)
+        self._tok = _StellarTokenizer(vocab_size=32000)
+        if resolved is not None:
+            try:
+                self._tok.load(resolved)
+            except Exception:
+                # Fall back to base charset without pre-trained merges
+                pass
+        self.do_lower_case = do_lower_case
+
+        super().__init__(
+            unk_token=unk_token,
+            bos_token=bos_token,
+            eos_token=eos_token,
+            sep_token=sep_token,
+            pad_token=pad_token,
+            cls_token=cls_token,
+            mask_token=mask_token,
+            additional_special_tokens=additional_special_tokens,
+            model_max_length=model_max_length,
+            do_lower_case=do_lower_case,
+            **kwargs,
+        )
+
+    @property
+    def vocab_size(self) -> int:
+        return len(self._tok.token_to_id)
+
+    def get_vocab(self) -> Dict[str, int]:
+        return dict(self._tok.token_to_id)
+
+    def _tokenize(self, text: str, **kwargs) -> List[str]:
+        """Tokenize a string into BPE token strings (used by encode/decode pipeline)."""
+        if self.do_lower_case:
+            text = text.lower()
+        ids = self._tok.encode(text, add_bos=False, add_eos=False)
+        return [self._tok.id_to_token.get(i, self.unk_token) for i in ids]
+
+    def _convert_token_to_id(self, token: str) -> int:
+        return self._tok.token_to_id.get(token, self._tok.token_to_id.get(self.unk_token, 3))
+
+    def _convert_id_to_token(self, index: int) -> str:
+        return self._tok.id_to_token.get(index, self.unk_token)
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        ids = [self._convert_token_to_id(t) for t in tokens]
+        return self._tok.decode(ids, skip_special=False)
+
+    # --- direct encode / decode overrides ---
+    def _encode_plus(
+        self,
+        text,
+        text_pair=None,
+        add_special_tokens=True,
+        padding_strategy="do_not_pad",
+        truncation_strategy="longest_first",
+        max_length=None,
+        stride=0,
+        is_split_into_words=False,
+        pad_to_multiple_of=None,
+        return_tensors=None,
+        return_token_type_ids=None,
+        return_attention_mask=None,
+        return_overflowing_tokens=False,
+        return_special_tokens_mask=False,
+        return_offsets_mapping=False,
+        return_length=False,
+        verbose=True,
+        **kwargs,
+    ):
+        if is_split_into_words:
+            text = "".join(text) if isinstance(text, list) else text
+        if self.do_lower_case:
+            text = text.lower()
+            if text_pair is not None:
+                text_pair = text_pair.lower() if not isinstance(text_pair, list) else "".join(text_pair)
+
+        ids = list(self._tok.encode(text, add_bos=False, add_eos=False))
+        if text_pair is not None:
+            pair_ids = list(self._tok.encode(text_pair, add_bos=False, add_eos=False))
+        else:
+            pair_ids = None
+
+        if add_special_tokens:
+            bos_id = self._tok.SPECIAL_TOKENS.get("[BOS]", 1)
+            eos_id = self._tok.SPECIAL_TOKENS.get("[EOS]", 2)
+            sep_id = self._tok.SPECIAL_TOKENS.get("[SEP]", 6)
+            if pair_ids is None:
+                ids = [bos_id] + ids + [eos_id]
+            else:
+                ids = [bos_id] + ids + [sep_id] + pair_ids + [eos_id]
+
+        # Truncation
+        if max_length is not None and len(ids) > max_length:
+            if truncation_strategy == "longest_first":
+                ids = ids[:max_length]
+
+        input_ids = ids
+        attention_mask = [1] * len(ids)
+
+        # Padding
+        if padding_strategy != "do_not_pad" and max_length is not None and len(ids) < max_length:
+            pad_id = self._tok.SPECIAL_TOKENS.get("[PAD]", 0)
+            pad_len = max_length - len(ids)
+            input_ids = input_ids + [pad_id] * pad_len
+            attention_mask = attention_mask + [0] * pad_len
+
+        encoding = {"input_ids": input_ids, "attention_mask": attention_mask}
+        if return_token_type_ids:
+            tti = [0] * len(input_ids)
+            if pair_ids is not None and add_special_tokens:
+                sep_id = self._tok.SPECIAL_TOKENS.get("[SEP]", 6)
+                sep_idx = None
+                for i, _id in enumerate(input_ids):
+                    if _id == sep_id and sep_idx is None:
+                        sep_idx = i
+                if sep_idx is not None:
+                    for j in range(sep_idx + 1, len(tti)):
+                        tti[j] = 1
+            encoding["token_type_ids"] = tti
+        if return_length:
+            encoding["length"] = len(input_ids)
+
+        if return_tensors is not None:
+            import torch
+            for k, v in list(encoding.items()):
+                if isinstance(v, list) and all(isinstance(x, int) for x in v):
+                    encoding[k] = torch.tensor([v], dtype=torch.long)
+                elif isinstance(v, int):
+                    encoding[k] = torch.tensor([v], dtype=torch.long)
+
+        return encoding
+
+    def decode(
+        self,
+        token_ids: Union[int, List[int], Any],
+        skip_special_tokens: bool = False,
+        clean_up_tokenization_spaces: bool = None,
+        **kwargs,
+    ) -> str:
+        if hasattr(token_ids, "tolist"):
+            token_ids = token_ids.tolist()
+        if isinstance(token_ids, int):
+            token_ids = [token_ids]
+        if (
+            isinstance(token_ids, list)
+            and len(token_ids) == 1
+            and isinstance(token_ids[0], list)
+        ):
+            token_ids = token_ids[0]
+        if not isinstance(token_ids, list):
+            token_ids = list(token_ids)
+        int_ids = [int(x) for x in token_ids]
+        return self._tok.decode(int_ids, skip_special=skip_special_tokens)
+
+    def batch_decode(self, sequences, **kwargs):
+        return [self.decode(seq, **kwargs) for seq in sequences]
+
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str, ...]:
+        if not os.path.isdir(save_directory):
+            raise ValueError(f"Vocabulary path ({save_directory}) should be a directory")
+        fname = (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
+        out_path = os.path.join(save_directory, fname)
+        self._tok.save(out_path)
+        return (out_path,)
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs):
+        # Ensure vocab_file points to a resolved backend tokenizer JSON (SimpleTokenizer format)
+        if "vocab_file" not in kwargs or kwargs["vocab_file"] is None:
+            search = pretrained_model_name_or_path
+            if isinstance(search, str) and os.path.isdir(search):
+                candidate = os.path.join(search, "backend_tokenizer.json")
+                if os.path.isfile(candidate):
+                    kwargs["vocab_file"] = candidate
+                else:
+                    # fallback to outputs dir for local development
+                    alt = os.path.join(_CURDIR, "backend_tokenizer.json")
+                    if os.path.isfile(alt):
+                        kwargs["vocab_file"] = alt
+        return super().from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)

tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "name_or_path": "stellarai-tiny",
+  "do_lower_case": false,
+  "model_max_length": 1024,
+  "padding_side": "right",
+  "truncation_side": "right",
+  "bos_token": "[BOS]",
+  "eos_token": "[EOS]",
+  "unk_token": "[UNK]",
+  "sep_token": "[SEP]",
+  "pad_token": "[PAD]",
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "additional_special_tokens": ["[IMG]", "[BOI]", "[EOI]"],
+  "strip_accents": false,
+  "clean_up_tokenization_spaces": true,
+  "comment_tokens": [],
+  "added_tokens_decoder": {
+    "0":  {"content": "[PAD]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "1":  {"content": "[BOS]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "2":  {"content": "[EOS]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "3":  {"content": "[UNK]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "4":  {"content": "[MASK]", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "5":  {"content": "[CLS]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "6":  {"content": "[SEP]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "7":  {"content": "[IMG]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "8":  {"content": "[BOI]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "9":  {"content": "[EOI]",  "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true}
+  }
+}