README.md

---
license: openrail
pipeline_tag: text-generation
---
# PiCo-7B

PiCo-7B is a **7 billion parameter (7B)** large language model featuring a novel **Adaptive Hierarchical Mixture of Experts (AHMoE)** architecture. This model aims to significantly reduce inference costs while maintaining powerful expressive capabilities through dynamic routing and expert compression techniques.

---

## 🚀 Model Highlights

- **High-Performance AHMoE Architecture**: With approximately 6.95B total parameters, the model activates only about **2.63B** parameters per token, achieving a 
"large capacity, lightweight" inference experience.
- **Adaptive Hierarchical Routing**: Introduces a two-level routing mechanism to optimize expert selection efficiency.
- **Context-Aware Meta-Router**: Dynamically analyzes input complexity and context to adjust routing decisions.
- **Ultra-Long Context Support**: Natively supports context lengths up to **131,072 (128K)** tokens.
- **OzmosToken Tokenizer**: Custom-developed multilingual tokenizer supporting 20+ language families, with specialized thought and reasoning markers.

---

## 🏗️ Architecture Specifications

### Core Parameters

| Parameter | Value |
| :--- | :--- |
| **Total Parameters** | ~6.95B |
| **Non-Embedding Parameters** | ~6.32B |
| **Active Parameters per Token** | ~2.63B (Top-2 Experts + Shared Expert) |
| **Hidden Size** | 2048 |
| **Intermediate Size** | 5888 |
| **Number of Layers** | 30 (15 MoE Layers + 15 Dense Layers) |
| **Attention Heads** | 16 (Q) / 4 (GQA) |
| **Context Window** | 131,072 tokens |
| **Vocabulary Size** | 152,064 |

### AHMoE Technical Details

1.  **Hierarchical Routing**: Experts are grouped, allowing for coarse-grained group selection followed by fine-grained expert selection within groups, reducing routing computational complexity.
2.  **Meta-Router**: Utilizes a context encoder to evaluate token difficulty, dynamically invoking more appropriate expert combinations for complex tasks.
3.  **Shared Knowledge Base**: A shared knowledge query module across all layers reduces redundant information storage among experts.
4.  **Dynamic Expert Pruning**: Automatically prunes low-contributing experts during runtime based on importance scores, improving computational efficiency.
5.  **Expert Compression**: Compresses expert weights (default 50%) to further optimize memory footprint.

---

## 🔤 OzmosToken Tokenizer

PiCo-7B uses the specially optimized **OzmosToken**, supporting over 20 language families including East Asian (Chinese, Japanese, Korean), European, Middle Eastern, and South Asian languages.

### Special Tokens

The model incorporates rich functional tokens to guide it in performing specific tasks:

- `<|thought|>` / `<|reasoning|>`: Triggers Chain-of-Thought (CoT) and logical reasoning.
- `<|summary|>` / `<|translation|>`: Task-oriented markers.
- `<|code|>` / `<|math|>`: For programming and mathematical problem-solving.
- `<|user|>` / `<|assistant|>` / `<|system|>`: Standard conversational role markers.

---

## 🛠️ Installation and Usage

### Environment Setup

```bash
pip install torch accelerate transformers safetensors datasets bitsandbytes peft trl lm-eval
```

### Quick Start (Inference)

```python
import torch
from pico import PiCoConfig, PiCoForCausalLM
from ozmos_tokenizer import OzmosTokenizer

# Load configuration and model
config = PiCoConfig.from_json("pico-7b/configs/pico_7b.json")
model = PiCoForCausalLM(config)
tokenizer = OzmosTokenizer.from_pretrained("pico-7b/ozmos_tokenizer_model")

# Example generation
inputs = tokenizer("<|user|>Please explain what a Mixture of Experts (MoE) model is?<|assistant|><|thought|>", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))
```

---


## 📜 License and Citation

This project is licensed under a **custom license based on OpenRAIL**.

```bibtex
@misc{pico7b,
    title={PiCo-7B: Adaptive Hierarchical Mixture of Experts},
    author={PiCo Team},
    year={2025},
    url={https://github.com/pico-ai/pico-7b}
}
```