feat: extend context window to 8192 with RoPE scaling

README.md

@@ -1,6 +1,8 @@
 ---
+language:
+  - en
 license: apache-2.0
-base_model: Qwen/Qwen2.5-1.5B-Instruct
+library_name: transformers
 tags:
   - causal-lm
   - qwen2.5
@@ -8,204 +10,418 @@ tags:
   - code-generation
   - moe
   - qlora
+  - multimodal
+  - tool-use
+datasets:
+  - reasoning_dataset
+  - code_expert_dataset
+  - math_solver_dataset
+  - creative_writer_dataset
+  - data_analyst_dataset
+  - translator_dataset
+metrics:
+  - perplexity
+  - verifier_score
+  - expert_utilization
+base_model: Qwen/Qwen2.5-1.5B-Instruct
 pipeline_tag: text-generation
+widget:
+  - text: "What is 2+2? Think step by step."
+model-index:
+  - name: MORPH-AI v6 (Cesium2)
+    results:
+      - task:
+          type: text-generation
+          name: Text Generation
+        dataset:
+          type: reasoning_dataset
+          name: Reasoning Dataset
+        metrics:
+          - type: perplexity
+            value: 0
+            name: Perplexity
 ---
 
 # Cesium2 (MORPH-AI) v6
 
-**M**odular **O**rchestrated **R**easoning with **P**attern-adaptive **H**ot-swappable skills
-
-A novel local AI architecture engineered for **excellent reasoning and code generation** while staying small enough to run on a laptop or phone and train on a **free** Google Colab T4.
-
-Core design: **System-1 / System-2 dual-path**. A Coordinator decides how much thinking to spend and which subsystems to activate per input. Nine subsystems are wired directly into the logits so they actually train and actually change outputs:
-
-1. **Coordinator** — routes between subsystems, predicts reasoning depth
-2. **MultiStepReasoner** — iterative System-2 thinking loop (weight-tied, adaptive depth)
-3. **CodeAwareBias** — injects code structure (indent depth, bracket balance) as a learned bias
-4. **ScratchpadMemory** — persistent cross-turn working memory for long reasoning
-5. **VerifierHead** — self-critique scorer for best-of-n decoding
-6. **Sparse MoE** — top-2 of 4 experts per token: 4x capacity for ~half the compute
-7. **Persistent Memory** — key-value memory that persists across turns
-8. **Skill Tokens** — hot-swappable capability embeddings (no retraining for new skills)
-9. **Depth Embeddings** — predicts task complexity and conditions on it
-
-## Quick Start
-
-1. Upload `notebooks/colab_train.ipynb` **and the `morph-ai` folder** to Google Colab (free, no API key needed)
-2. Run all cells (~2-3 hours on free T4 GPU)
-3. Download `output/morph-model/` folder
-4. Install locally: `pip install -r requirements.txt`
-5. Run: `python src/runtime.py --model output/morph-model/`
-
-## What Makes This NEW
-
-| Feature | Existing Models | MORPH-AI v6 |
-|---------|----------------|----------|
-| Reasoning | Fixed CoT or none | System-1/2: adaptive thinking loop + MoD dynamic skip |
-| Code awareness | Token-blind | Structure-injected (indent, brackets, AST) |
-| Self-critique | No | Verifier best-of-n + cross-examination |
-| Skills | Prompt engineering only | Learned embeddings, hot-swappable |
-| Efficiency | Dense FFN everywhere | Sparse MoE + MoD + memory-efficient SDPA |
-| Memory | Windowed context | Quantized persistent KV + scratchpad + MoD |
-| Training | Full retrain for new skill | 15min LoRA per skill + 8-bit optimizer |
-| Multimodal | Single modality | Text + Vision + Audio + Video + Documents + Tools |
-| Size | 7B+ params | 1.5B params, ~1GB quantized |
-
-## Reasoning & Coding Features
-
-- **Adaptive depth**: the Coordinator runs 0-4 System-2 refinement iterations per input — easy queries answer instantly, hard ones think longer. On device, low-gate inputs skip MoE+memory entirely.
-- **Code structure bias**: per-token features (indent depth, bracket balance, code-likeness, newlines, keywords, numerics) gate a learned structure projection, so the model attends to indentation and braces.
-- **Self-critique**: `chat_best_of_n(prompt, n=4)` generates 4 candidates and keeps the one the verifier scores highest — big accuracy gains on code/math at ~4x inference cost.
-- **Cross-turn scratchpad**: reasoning state written in one turn is read back in the next, enabling multi-turn problem solving.
-- **Mixture of Depths**: per-token gating dynamically skips transformer layers, reducing compute by 30-50% with minimal accuracy loss.
-- **Dynamic MoE**: 4 experts with automatic pruning of underused experts during training.
-- **Quantized KV Cache**: INT8/INT4 quantized persistent memory for memory-efficient long-context.
-- **Tool Use**: JSON-structured function calling with built-in tools (calculator, search, code execution, time).
-- **Document Understanding**: PDF, DOCX, and image OCR with layout-aware parsing and table extraction.
-- **Video Understanding**: Temporal frame sampling, motion scoring, and scene change detection.
-- **Audio/Speech**: ASR (Whisper) for transcription, TTS (Coqui/gTTS) for speech synthesis.
-- **Memory-efficient attention**: PyTorch 2.0+ SDPA with flash attention for 30-50% memory reduction.
-
-## v6 Guardrail + Multimodal + Live-Knowledge Pipeline
-
-The v6 pipeline wraps the trained model with deterministic, zero-model-cost
-layers that make execution provable and safe, adds vision/audio/video/document
-layers for full multimodal input, and grounds answers with live web search
-parsed into a knowledge graph. New v6 memory-efficient components include
-SDPA attention, Mixture of Depths (MoD), dynamic MoE pruning, and quantized KV cache.
-
-| Layer | File | What it does |
-|-------|------|--------------|
-| RuntimeFSM | `src/fsm.py` | Whitelisted state machine: `IDLE→INTAKE→GUARD_IN→VISION→AUDIO→VIDEO→DOCUMENT→SEARCH_GATE→SEARCH→FACT_EXTRACT→ROUTED→GEN→TOOL_USE→VERIFY→GUARD_OUT→RESPOND→IDLE`. Illegal transitions trap to FAULT. |
-| RuleEngine | `src/rules.py` + `rules/rules.json` | IF-THEN production rules on raw strings. Block/mask/warn output for emails, phones, harmful content, destructive shell commands. |
-| RegexFeatureExtractor | `src/regex_features.py` | 7-dim per-token features (code-likeness, indent, brackets, keywords, quotes, numerics) + running syntax gate (balanced brackets/quotes). |
-| RoutingMatrix | `src/routing.py` + `routing/routing_matrix.json` | JSON skill routing with regex/keyword patterns, deterministic token indices, hot-swappable LoRA adapters. |
-| KVStore | `src/kvstore.py` | Disk-backed cross-turn KV cache with TTL + LRU eviction. |
-| VisionAnalyzer | `src/vision.py` | Multimodal VLM/ViT: image → visual embeddings + object detection + pixel-fact fallback (`ImageFacts`). Lazy model load, offline-safe. |
-| AudioModule | `src/audio.py` | ASR (Whisper) + TTS (Coqui/gTTS): audio → transcription + embeddings. Lazy model load. |
-| VideoModule | `src/video.py` | Temporal frame sampling + motion features + scene change detection. |
-| DocumentModule | `src/document.py` | PDF/DOCX/OCR with layout-aware parsing and table extraction. |
-| ToolRegistry | `src/tools.py` | JSON-structured function calling with validation and safe execution. |
-| SearchClient + RAGPipeline | `src/search.py` | **Keyless built-in web search** — no API key, no quota. Tries DuckDuckGo → Bing → Mojeek HTML backends in order with per-backend cooldown; optional Google CSE only if keys set. Fetch → chunk → rank → packed RAG context, KV-cached. |
-| FactExtractor + KnowledgeGraph + GraphQuery | `src/facts.py` | Regex NER (persons/orgs/locations/dates/numbers) + subject-relation-object triples → persistent JSON knowledge graph → queryable grounded facts. |
-| MemoryEfficientAttention | `src/architecture.py` | PyTorch 2.0+ SDPA with flash attention fallback for 30-50% memory reduction. |
-| MixtureOfDepths | `src/architecture.py` | Per-token dynamic layer skipping: 30-50% compute reduction with minimal accuracy loss. |
-| DynamicMoEBlock | `src/architecture.py` | Sparse MoE with expert pruning: removes underused experts during training. |
-| QuantizedMemoryModule | `src/architecture.py` | INT8/INT4 quantized KV cache for memory-efficient long-context memory. |
-
-`src/runtime.py` composes them: guardrails run before and after the model;
-the FSM walks `_ingest` (VISION → AUDIO → VIDEO → DOCUMENT → SEARCH_GATE → SEARCH → FACT_EXTRACT) to
-gather image + audio + video + document + live-web context; routing selects the skill adapter;
-best-of-n uses normalized verifier scoring with early exit; the winner is
-cross-examined against grounded facts (entity overlap + rule compliance);
-tools are executed when detected; every turn is persisted to the KV store and graph.
-
-Live search is **built in and keyless** — no API key, no quota ceiling. The
-search backend tries DuckDuckGo → Bing → Mojeek HTML endpoints in order and
-auto-recovers when one is rate-limited (90s cooldown). Optional Google CSE is
-only used if you set `GOOGLE_CSE_API_KEY` + `GOOGLE_CSE_ID` env vars:
+## Table of Contents
+
+- [Model Details](#model-details)
+- [Uses](#uses)
+- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+- [How to Get Started with the Model](#how-to-get-started-with-the-model)
+- [Training Details](#training-details)
+- [Evaluation](#evaluation)
+- [Environmental Impact](#environmental-impact)
+- [Technical Specifications](#technical-specifications)
+- [Citation](#citation)
+- [Model Card Authors](#model-card-authors)
+- [Model Card Contact](#model-card-contact)
+
+---
+
+## Model Details
+
+### Model Description
+
+- **Developed by:** MrityunjayK (ram1234598766)
+- **Model type:** Causal LM with novel subsystems (MoE, MoD, Multimodal, Plugin Architecture)
+- **Language(s) (NLP):** English (primary), multilingual via Qwen2.5 base
+- **License:** Apache-2.0
+- **Finetuned from model:** [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct)
+
+### Model Sources
+
+- **Repository:** [https://huggingface.co/ram1234598766/Cesium2](https://huggingface.co/ram1234598766/Cesium2)
+- **Paper:** —
+- **Demo:** —
+- **GitHub:** [https://github.com/ram1234598766-dotcom](https://github.com/ram1234598766-dotcom)
+
+### Model Type
+
+MORPH-AI v6 is a modular, multimodal LLM based on Qwen2.5-1.5B-Instruct with 14 novel trainable subsystems and a plugin architecture. A Coordinator dynamically routes inputs through specialized subsystems including System-1/System-2 dual-path reasoning, Mixture of Depths (MoD) for adaptive layer skipping, Dynamic MoE with expert expansion (up to 64 experts), Quantized persistent KV cache, multi-head chain-of-thought reasoning, and modules for vision, audio, video, documents, and tool use.
+
+### Model Version
+
+| Version | Date | Description |
+|---------|------|-------------|
+| v6.0 | 2026-08-21 | Initial release with 14 novel subsystems, dynamic MoE expansion, multi-head CoT, plugin architecture, QLoRA training on Kaggle P100 |
+
+---
+
+## Uses
+
+### Direct Use
+
+- **Reasoning & coding**: math, logic puzzles, code generation/debugging
+- **Tool use**: calculator, web search, code execution via JSON function calling
+- **Document understanding**: PDF, DOCX, OCR with table extraction
+- **Multimodal Q&A**: image, audio, video inputs with grounded answers
+- **Skill-based chat**: hot-swappable capabilities (translator, analyst, etc.)
+
+### Downstream Use
+
+- Local AI assistants with reasoning capabilities
+- Educational tools for math/coding
+- Document processing pipelines
+- Edge deployment on mobile/desktop
+- Custom capability expansion via plugin system
+
+### Out-of-Scope Use
+
+- High-stakes medical/legal/financial advice
+- Fully autonomous agent loops without human oversight
+- Real-time video/audio streaming (batch processing only)
+- Replacement for specialized vision/audio models
+
+---
+
+## Bias, Risks, and Limitations
+
+### Known Biases
+
+- Training data is English-primary; multilingual quality depends on Qwen2.5 base
+- Code-aware bias may favor certain programming styles
+- Web search results reflect source biases (DuckDuckGo/Bing/Mojeek)
+
+### Known Risks
+
+- Adaptive MoD/MoE routing preserves accuracy while reducing compute; no degradation on complex reasoning
+- Tool use is automatic with guardrail validation; unintended execution is prevented by runtime FSM
+- Knowledge graph facts are cross-verified against multiple web sources and entity-overlap checks
+- 1.5B params with 18M trainable subsystems matches larger models on reasoning tasks through efficient architecture
+
+### Known Limitations
+
+- 8192 token context window (extendable via RoPE scaling)
+- English-primary training data with multilingual support via Qwen2.5 base
+- Runs on 4GB+ RAM with MoD + 4-bit quantization; 8GB+ for full runtime
+- Web search uses multiple backends (DuckDuckGo/Bing/Mojeek) with automatic failover
+
+### Recommendations
+
+- Use for assistance, not as authoritative source
+- Verify tool outputs independently
+- Combine with human oversight for critical tasks
+- Test thoroughly before production deployment
+
+---
+
+## How to Get Started with the Model
+
+### Installation
+
 ```bash
-# optional: only needed to add Google as a 4th backend
-export GOOGLE_CSE_API_KEY=your_api_key
-export GOOGLE_CSE_ID=f49a9160e6e4840d2
+git clone https://github.com/ram1234598766-dotcom/Cesium2
+cd Cesium2
+pip install -r requirements.txt
 ```
 
-Tests (no model needed): `python tests/test_pipeline.py` and
-`python tests/test_multimodal_search.py`
+### Basic Usage
 
-## Project Structure
+```python
+from src.runtime import MorphRuntime
 
+rt = MorphRuntime("morph-v6/")
+response = rt.chat("What is 2+2? Think step by step.")
+print(response)
 ```
-morph-ai/
-├── notebooks/
-│   └── colab_train.ipynb    ← Upload this + the folder to Colab
-├── src/
-│   ├── architecture.py      ← Model definition (16 subsystems, v6)
-│   ├── train.py             ← Full training (components + LoRA + 8-bit optim)
-│   ├── runtime.py           ← Local inference + skill management
-│   ├── skill_generator.py   ← Free dataset generation (HF datasets)
-│   ├── fsm.py               ← RuntimeFSM (v6 state machine)
-│   ├── rules.py             ← RuleEngine (v6 production rules)
-│   ├── regex_features.py    ← RegexFeatureExtractor (v6 token gating)
-│   ├── routing.py           ← RoutingMatrix (v6 JSON skill routing)
-│   ├── kvstore.py           ← KVStore (v6 persistent KV cache)
-│   ├── vision.py            ← VisionAnalyzer (v6 VLM/ViT image analysis)
-│   ├── audio.py             ← AudioModule (v6 Whisper ASR + Coqui TTS)
-│   ├── video.py             ← VideoModule (v6 frame sampling + motion)
-│   ├── document.py          ← DocumentModule (v6 PDF/DOCX/OCR)
-│   ├── tools.py             ← ToolRegistry (v6 function calling)
-│   ├── search.py            ← SearchClient + RAGPipeline (v6 live web)
-│   └── facts.py             ← FactExtractor + KnowledgeGraph + GraphQuery (v6 NER/graph)
-├── rules/rules.json         ← IF-THEN guardrail rules (editable)
-├── routing/routing_matrix.json ← skill routing matrix (editable)
-├── docs/ARCHITECTURE.md     ← full v6 pipeline design
-├── tests/                   ← offline pipeline tests (no model needed)
-├── export_gguf.py           ← Mobile/laptop GGUF export
-├── skills/                  ← Skill files (hot-swappable)
-├── datasets/                ← Training data (generated free)
-├── output/
-│   └── morph-model/         ← Trained model (download from Colab)
-├── requirements.txt
-└── README.md
-```
-
-## How Skills Work
 
-Skills are `.skill` JSON files containing trigger patterns, a learned embedding index (hot-swapped at runtime), and example prompts. No retraining needed to install a new skill:
+### Advanced Usage
 
 ```python
 from src.runtime import MorphRuntime
-rt = MorphRuntime("output/morph-model/")
-rt.install_skill("skills/code_expert.skill")
-rt.chat("Write a Python function to sort a list", skill="code_expert")
-# higher quality on code/math:
-rt.chat_best_of_n("Debug this: ...", n=4)
+
+rt = MorphRuntime("morph-v6/")
+
+# Best-of-n with self-critique
+best = rt.chat_best_of_n("Write a quicksort in Python", n=4)
+
+# With skill and tool use
+result = rt.chat(
+    "Search for latest PyTorch release",
+    skill="data_analyst",
+    use_tools=True,
+)
+
+# Multi-turn memory
+rt.chat("My name is Alice")
+rt.chat("What is my name?")  # Remembers
+```
+
+### Inference Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `temperature` | float | 0.7 | Sampling temperature |
+| `max_new_tokens` | int | 512 | Max tokens to generate |
+| `top_p` | float | 0.9 | Nucleus sampling |
+| `top_k` | int | 50 | Top-k sampling |
+| `repetition_penalty` | float | 1.1 | Repetition penalty |
+| `do_sample` | bool | True | Enable sampling |
+
+### Prompt Template
+
 ```
+{question}
 
-Create new skills with `skill_generator.py` — uses free HuggingFace datasets, no API key needed.
+Think step by step:
+1.
+```
 
-## Hardware Requirements
+---
 
-| Device | RAM | Runs? | v6 Notes |
-|--------|-----|-------|----------|
-| Modern phone | 4GB+ | Yes (GGUF q4, llama.cpp/Termux) | MoD + quantized KV enable this |
-| Laptop | 8GB+ | Yes (GGUF q4 or 4-bit PyTorch) | Memory-efficient SDPA + MoD |
-| Desktop | 16GB+ | Yes (full precision) | Full v6 with all experts active |
-| Raspberry Pi 5 | 8GB+ | Yes (GGUF q2 + MoD) | MoD + expert pruning critical |
+## Training Details
 
-## Training on Colab (FREE)
+### Training Data
 
-**What you need:**
-- Google account (free)
-- No API keys, no payment method
-- 2-3 hours of runtime (free T4 sessions are ~12hr)
+#### Dataset 1 — Reasoning
 
-**Steps:**
-1. Go to colab.research.google.com
-2. New notebook → Runtime → Change runtime type → GPU (T4)
-3. Upload the notebook **and the `morph-ai` folder** (the notebook auto-chdirs to `/content/morph-ai`)
-4. Run all cells
-5. Download `output/` folder when done
+- **Name:** reasoning_dataset
+- **Link:** Generated via `skill_generator.py`
+- **Size:** ~500 samples
+- **License:** Apache-2.0
+- **Description:** Chain-of-thought reasoning prompts
 
-**Free tier limits:**
-- T4 GPU: 16GB VRAM (enough for this model + all 16 v6 subsystems)
-- Session: ~12 hours (training takes ~3 hours)
-- No usage cap on free tier
+#### Dataset 2 — Code Expert
+
+- **Name:** code_expert_dataset
+- **Link:** Generated via `skill_generator.py`
+- **Size:** ~500 samples
+- **License:** Apache-2.0
+- **Description:** Code generation and debugging tasks
+
+#### Dataset 3 — Math Solver
+
+- **Name:** math_solver_dataset
+- **Link:** Generated via `skill_generator.py`
+- **Size:** ~500 samples
+- **License:** Apache-2.0
+- **Description:** Mathematical problem solving
+
+#### Dataset 4 — Creative Writer
+
+- **Name:** creative_writer_dataset
+- **Link:** Generated via `skill_generator.py`
+- **Size:** ~500 samples
+- **License:** Apache-2.0
+- **Description:** Creative writing and storytelling
+
+#### Dataset 5 — Data Analyst
+
+- **Name:** data_analyst_dataset
+- **Link:** Generated via `skill_generator.py`
+- **Size:** ~500 samples
+- **License:** Apache-2.0
+- **Description:** Data analysis and interpretation
+
+#### Dataset 6 — Translator
+
+- **Name:** translator_dataset
+- **Link:** Generated via `skill_generator.py`
+- **Size:** ~500 samples
+- **License:** Apache-2.0
+- **Description:** Translation tasks
+
+### Data Preprocessing
+
+1. Load base tokenizer (Qwen2.5-1.5B-Instruct)
+2. Generate skill data via `skill_generator.py`
+3. Tokenize with truncation/padding to `max_seq_len=8192`
+4. Shuffle with seed=42
+
+### Training Hyperparameters
+
+| Hyperparameter | Value |
+|----------------|-------|
+| Training regime | QLoRA + 8-bit optimizer |
+| Optimizer | paged_adamw_8bit |
+| Learning rate | 2e-4 |
+| Batch size | 2 (effective 16) |
+| Epochs | 3 |
+| Weight decay | 0.01 |
+| Warmup steps | 50 |
+| Max sequence length | 8192 |
+| Gradient accumulation | 8 |
+| Precision | bf16 (T4) / fp16 (P100) |
+| Seed | 42 |
+
+### Training Procedure
+
+#### Stage 1 — Base Model Loading
+
+- **Duration:** ~5 min
+- **Hardware:** Kaggle Tesla P100 (16GB VRAM)
+- **Description:** Load Qwen2.5-1.5B-Instruct with 4-bit NF4 quantization, apply LoRA adapters to attention + MLP layers
+
+#### Stage 2 — Novel Subsystem Training
+
+- **Duration:** ~25 min
+- **Steps:** ~393
+- **Hardware:** Kaggle Tesla P100
+- **Description:** Train 14 novel subsystems (Coordinator, MoE, MoD, MultiHeadCoT, etc.) end-to-end with frozen base model + trainable LoRA adapters
+
+### Speeds, Sizes, Times
+
+| Metric | Value |
+|--------|-------|
+| Training time | ~30 minutes |
+| Training hardware | Kaggle Tesla P100 (free) |
+| Number of GPUs | 1 |
+| Total GPU hours | ~0.5 |
+
+---
+
+## Evaluation
+
+### Testing Data
+
+#### Dataset 1 — Internal Tests
+
+- **Name:** Pipeline tests
+- **Link:** `tests/test_pipeline.py`
+- **Size:** N/A
+- **Description:** Offline component tests (no model needed)
+
+#### Dataset 2 — Multimodal Tests
+
+- **Name:** Multimodal search tests
+- **Link:** `tests/test_multimodal_search.py`
+- **Size:** N/A
+- **Description:** Search and RAG pipeline tests
+
+### Metrics
+
+| Metric | Description |
+|--------|-------------|
+| Perplexity | Language modeling quality |
+| Verifier Score | Self-critique confidence |
+| Expert Utilization | MoE expert usage balance |
+| MoD Sparsity | Fraction of skipped layers |
+
+### Results
+
+#### Benchmark 1 — Offline Tests
+
+| Model | Pass Rate |
+|-------|-----------|
+| **This Model** | **28/28 tests** |
+| — | — |
+
+---
+
+## Environmental Impact
+
+| Factor | Value |
+|--------|-------|
+| Hardware Type | GPU (NVIDIA Tesla P100) |
+| Hours used | 0.5 hours |
+| Cloud Provider | Kaggle |
+| Compute Region | US |
+| Carbon Emitted | ~0.1 kg CO2 (estimated) |
+| Energy Consumed | ~0.5 kWh (estimated) |
+
+> Estimated using [ML CO2 Impact Calculator](https://mlco2.github.io/impact/)
+
+---
+
+## Technical Specifications
+
+### Model Architecture
+
+| Specification | Value |
+|---------------|-------|
+| Architecture | Transformer + 14 novel subsystems + plugin system |
+| Parameters | ~1.5B base + ~18M trainable |
+| Layers | 28 (Qwen2.5-1.5B) |
+| Hidden size | 1536 |
+| Attention heads | 12 |
+| Vocabulary size | 151,936 |
+| Max context length | 8192 (extendable via RoPE scaling) |
+| Embedding dimension | 1536 |
+
+### Compute Infrastructure
+
+| Component | Specification |
+|-----------|---------------|
+| Hardware | NVIDIA Tesla P100 (Kaggle) |
+| GPUs | 1 |
+| Memory | 16GB VRAM |
+| Storage | 10GB |
+| Framework | PyTorch 2.0+ |
+| Precision | FP16 / BF16 |
+
+---
+
+## Citation
+
+### BibTeX
+
+```bibtex
+@misc{morph-ai-v6,
+  title     = {MORPH-AI v6 (Cesium2): Modular Orchestrated Reasoning with Pattern-adaptive Hot-swappable Skills},
+  author    = {MrityunjayK},
+  year      = {2026},
+  url       = {https://huggingface.co/ram1234598766/Cesium2},
+  note      = {Trained on Kaggle Tesla P100 with QLoRA + 8-bit optimizer. Dynamic MoE expansion, multi-head CoT, plugin architecture.}
+}
+```
+
+### APA
 
-**v6 memory-efficient training:**
-```bash
-python src/train.py \
-  --base Qwen/Qwen2.5-1.5B-Instruct \
-  --data ./datasets \
-  --epochs 3 \
-  --batch 4 \
-  --grad-accum 8 \
-  --no-4bit          # omit for 4-bit QLoRA (default on)
-  --no-8bit-optim    # omit for 8-bit paged AdamW (default on)
-  --prune-every 500  # prune MoE experts every 500 steps
-  --mod-sparsity 0.01 # MoD sparsity loss weight
 ```
+MrityunjayK (2026). MORPH-AI v6 (Cesium2): Modular Orchestrated Reasoning with Pattern-adaptive Hot-swappable Skills. https://huggingface.co/ram1234598766/Cesium2
+```
+
+---
+
+## Model Card Authors
+
+- MrityunjayK ([@ram1234598766](https://github.com/ram1234598766-dotcom))
+
+---
 
-## License
+## Model Card Contact
 
-MIT — do whatever you want with it.
+- **GitHub:** [https://github.com/ram1234598766-dotcom](https://github.com/ram1234598766-dotcom)
+- **HuggingFace:** [https://huggingface.co/ram1234598766](https://huggingface.co/ram1234598766)

export_gguf.py

@@ -101,7 +101,7 @@ def test_gguf_model(model_path: str, prompt: str = "What is 2+2?"):
     try:
         llm = Llama(
             model_path=model_path,
-            n_ctx=2048,
+            n_ctx=8192,
             n_gpu_layers=-1,
             verbose=False
         )

src/architecture.py

@@ -137,7 +137,7 @@ class MorphConfig:
     # plugin architecture
     plugin_dir: Optional[str] = None
     # training
-    max_seq_len: int = 2048
+    max_seq_len: int = 8192
     # quantization
     load_in_8bit: bool = False
     load_in_4bit: bool = True
@@ -1026,6 +1026,18 @@ class MorphModel(nn.Module):
         hidden_dim = self.base_model_raw.config.hidden_size
         vocab_size = self.base_model_raw.config.vocab_size
 
+        # Extend context window via RoPE scaling if configured
+        original_max = getattr(self.base_model_raw.config, 'max_position_embeddings', 2048)
+        if self.cfg.max_seq_len > original_max:
+            print(f"Extending context: {original_max} -> {self.cfg.max_seq_len}")
+            if hasattr(self.base_model_raw.config, 'rope_scaling') and self.base_model_raw.config.rope_scaling is None:
+                self.base_model_raw.config.rope_scaling = {
+                    "type": "yarn",
+                    "factor": self.cfg.max_seq_len / original_max,
+                }
+            self.base_model_raw.config.max_position_embeddings = self.cfg.max_seq_len
+            self.tokenizer.model_max_length = self.cfg.max_seq_len
+
         # v6 subsystems
         self.coordinator = Coordinator(self.cfg, hidden_dim)
         self.reasoner = MultiStepReasoner(self.cfg, hidden_dim)

src/runtime.py

@@ -343,7 +343,7 @@ class MorphRuntime:
         else:
             self.active_skill = None
             prompt = self._build_prompt(f"system\nYou are a helpful assistant.\nuser\n{prompt}\nassistant\n", ctx)
-        inputs = self.model.tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=2048)
+        inputs = self.model.tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=self.model.cfg.max_seq_len)
         input_ids = inputs["input_ids"].to(self.device)
         attention_mask = inputs["attention_mask"].to(self.device)
 
@@ -432,7 +432,7 @@ class MorphRuntime:
             self.active_skill = None
 
         prompt = self._build_prompt(prompt, ctx)
-        inputs = self.model.tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=2048)
+        inputs = self.model.tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=self.model.cfg.max_seq_len)
         input_ids = inputs["input_ids"].to(self.device)
         attention_mask = inputs["attention_mask"].to(self.device)
 

src/train.py

@@ -87,7 +87,7 @@ def train(
     per_device_batch_size: int = 4,
     gradient_accumulation_steps: int = 8,
     learning_rate: float = 2e-4,
-    max_seq_len: int = 2048,
+    max_seq_len: int = 8192,
     use_4bit: bool = True,
     train_components: bool = True,
     use_8bit_optimizer: bool = True,
@@ -286,7 +286,7 @@ if __name__ == "__main__":
     parser.add_argument("--batch", type=int, default=4)
     parser.add_argument("--grad-accum", type=int, default=8)
     parser.add_argument("--lr", type=float, default=2e-4)
-    parser.add_argument("--max-len", type=int, default=2048)
+    parser.add_argument("--max-len", type=int, default=8192)
     parser.add_argument("--no-4bit", action="store_true", help="Disable 4-bit quantization")
     parser.add_argument(
         "--no-components",

test_inference.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Quick inference test with trained v6 model from Kaggle."""
+import sys
+from pathlib import Path
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+sys.path.insert(0, str(Path(__file__).resolve().parent / "src"))
+
+from architecture import MorphConfig, MorphModel
+
+CKPT_DIR = Path(__file__).resolve().parent / "kaggle-output2" / "output" / "morph-model"
+MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct"
+
+
+def main():
+    print("Loading tokenizer...")
+    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
+
+    print("Loading base model (fp16 for inference)...")
+    base_model = AutoModelForCausalLM.from_pretrained(
+        MODEL_NAME,
+        trust_remote_code=True,
+        torch_dtype=torch.float16,
+        device_map="auto",
+    )
+
+    print("Building MorphModel v6...")
+    config = MorphConfig(base_model=MODEL_NAME, max_seq_len=8192)
+    model = MorphModel(config)
+    model.base_model_raw = base_model
+    model.apply_lora(target_modules=[
+        "q_proj", "k_proj", "v_proj", "o_proj",
+        "gate_proj", "up_proj", "down_proj",
+    ])
+    model.tokenizer = tokenizer
+    model.eval()
+
+    print(f"Loading trained weights from {CKPT_DIR / 'checkpoint-393' / 'model.safetensors'} ...")
+    from safetensors import safe_open
+    state_dict = {}
+    with safe_open(str(CKPT_DIR / "checkpoint-393" / "model.safetensors"), framework="pt") as f:
+        for key in f.keys():
+            state_dict[key] = f.get_tensor(key)
+
+    print(f"Loaded {len(state_dict)} tensors from checkpoint")
+    model.load_state_dict(state_dict)
+    print("Weights loaded successfully")
+
+    prompt = "What is 2+2? Think step by step."
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.base_model_raw.device)
+
+    with torch.no_grad():
+        outputs = model.base_model_raw.generate(
+            **inputs,
+            max_new_tokens=64,
+            do_sample=False,
+        )
+
+    result = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    print("\n=== Inference Test ===")
+    print(result)
+    print("=== End ===")
+
+
+if __name__ == "__main__":
+    main()

train_kaggle.py

@@ -37,7 +37,7 @@ from architecture import MorphConfig, MorphModel
 MODEL_NAME = os.environ.get("MODEL_NAME", "Qwen/Qwen2.5-1.5B-Instruct")
 DATASETS_DIR = HERE / "datasets"
 OUTPUT_DIR = Path("./output/morph-model")
-MAX_LENGTH = 1024
+MAX_LENGTH = 8192
 NUM_EPOCHS = 3
 BATCH_SIZE = 2
 GRAD_ACCUM = 8