| | --- |
| | language: |
| | - en |
| | license: mit |
| | base_model: Qwen/Qwen2.5-Coder-7B |
| | tags: |
| | - zenith |
| | - tenstorrent |
| | - code |
| | - reasoning |
| | - moe |
| | - ring-attention |
| | - eq-adapter |
| | - matrix-corp |
| | pipeline_tag: text-generation |
| | library_name: transformers |
| | model_type: zenith |
| | hardware: |
| | - tenstorrent-blackhole-p300a |
| | --- |
| | |
| | # Zenith-7B V1 |
| |
|
| | Standard GPU-optimized language model with code generation and emotional intelligence capabilities. |
| |
|
| | ## Features |
| |
|
| | - **7B Parameter Model**: Efficient for consumer GPUs (8-16GB VRAM) |
| | - **Code Generation**: Fine-tuned on Qwen2.5-Coder base for exceptional programming abilities |
| | - **Emotional Intelligence**: EQ adapter for recognizing and responding to emotions |
| | - **OpenThoughts Integration**: Trained on high-quality reasoning data |
| | - **LoRA/QLoRA Support**: Efficient fine-tuning with 4-bit quantization |
| | - **Ollama Compatible**: Ready-to-use Modelfile for easy deployment |
| |
|
| | ## Quick Start |
| |
|
| | ### Installation |
| |
|
| | ```bash |
| | # Clone and setup |
| | cd Zenith/V1/7B |
| | pip install -r requirements.txt |
| | ``` |
| |
|
| | ### Training |
| |
|
| | ```bash |
| | # Full fine-tuning |
| | python train.py \ |
| | --base_model Qwen/Qwen2.5-Coder-7B \ |
| | --train_data path/to/train.json \ |
| | --epochs 3 \ |
| | --batch_size 4 \ |
| | --learning_rate 2e-5 |
| | |
| | # LoRA fine-tuning (recommended for most users) |
| | python train.py \ |
| | --base_model Qwen/Qwen2.5-Coder-7B \ |
| | --train_data path/to/train.json \ |
| | --use_lora \ |
| | --lora_r 16 \ |
| | --lora_alpha 32 \ |
| | --epochs 3 \ |
| | --batch_size 8 |
| | ``` |
| |
|
| | ### Inference |
| |
|
| | ```bash |
| | # Interactive mode |
| | python inference.py --checkpoint ./outputs/checkpoint-final |
| | |
| | # Single prompt |
| | python inference.py \ |
| | --checkpoint ./outputs/checkpoint-final \ |
| | --prompt "Write a Python function to reverse a linked list" \ |
| | --max_new_tokens 512 |
| | ``` |
| |
|
| | ### Ollama Deployment |
| |
|
| | ```bash |
| | # Build and run with Ollama |
| | ollama create zenith-7b -f Modelfile |
| | ollama run zenith-7b "Explain quantum computing in simple terms" |
| | ``` |
| |
|
| | ## Project Structure |
| |
|
| | ``` |
| | Zenith/V1/7B/ |
| | ├── configs/ # Configuration files |
| | │ ├── zenith_config.py # Model architecture config |
| | │ ├── data_config.py # Data processing config |
| | │ └── training_config.py # Training hyperparameters |
| | ├── data/ # Data processing modules |
| | │ ├── openthoughts_processor.py |
| | │ ├── quality_filter.py |
| | │ ├── curriculum_sampler.py |
| | │ ├── advanced_tokenizer.py |
| | │ └── preprocessing.py |
| | ├── src/ # Source code |
| | │ ├── models/ |
| | │ │ ├── zenith_model.py |
| | │ │ ├── dense_layer.py |
| | │ │ └── moe_layer.py |
| | │ └── utils/ |
| | ├── scripts/ # Utility scripts |
| | ├── tests/ # Test suite |
| | ├── train.py # Main training script |
| | ├── inference.py # Inference and generation |
| | ├── test_model.py # Model validation tests |
| | ├── finetune_qwen.py # Qwen fine-tuning guide |
| | ├── Modelfile # Ollama configuration |
| | ├── requirements.txt # Python dependencies |
| | └── README.md # This file |
| | ``` |
| |
|
| | ## Configuration |
| |
|
| | The model uses a unified configuration system in `configs/zenith_config.py`: |
| |
|
| | ```python |
| | from configs.zenith_config import get_7b_config |
| | |
| | config = get_7b_config() |
| | # Parameters: |
| | # - hidden_size: 4096 |
| | # - num_layers: 32 |
| | # - num_heads: 32 |
| | # - num_experts: 0 (dense only, set >1 for MoE) |
| | # - use_eq_adapter: True (emotional intelligence) |
| | # - max_seq_len: 8192 |
| | ``` |
| |
|
| | ## Data Processing |
| |
|
| | ### OpenThoughts Integration |
| |
|
| | The data pipeline supports the OpenThoughts-1.2M dataset: |
| |
|
| | ```python |
| | from data.openthoughts_processor import OpenThoughtsProcessor, OpenThoughtsConfig |
| | |
| | config = OpenThoughtsConfig( |
| | dataset_name="open-thoughts/OpenThoughts3-1.2M", |
| | streaming=True, |
| | quality_filtering=True, |
| | curriculum_learning=True, |
| | augmentation=True |
| | ) |
| | processor = OpenThoughtsProcessor(config) |
| | dataset = processor.load_dataset() |
| | ``` |
| |
|
| | ### Quality Filtering |
| |
|
| | Multi-dimensional quality assessment: |
| | - Length appropriateness |
| | - Language detection (English only) |
| | - Repetition detection |
| | - Coherence scoring |
| | - Structure validation |
| | - Thought quality (for CoT data) |
| |
|
| | ### Curriculum Learning |
| |
|
| | Progressive training stages: |
| | 1. **Foundation**: High-quality, well-structured samples |
| | 2. **Reasoning**: Chain-of-thought and problem-solving |
| | 3. **Code**: Programming and technical content |
| | 4. **Full**: Complete dataset with all samples |
| |
|
| | ## Advanced Features |
| |
|
| | ### MoE (Mixture of Experts) |
| |
|
| | Enable sparse activation for better performance: |
| |
|
| | ```bash |
| | python train.py --use_moe --num_experts 8 |
| | ``` |
| |
|
| | - Top-2 routing with load balancing |
| | - 60% of layers use MoE (middle layers) |
| | - Shared router groups for efficiency |
| |
|
| | ### EQ Adapter |
| |
|
| | Emotional intelligence module: |
| |
|
| | ```bash |
| | python train.py --use_eq_adapter --eq_loss_weight 0.1 |
| | ``` |
| |
|
| | - Frustration detection (regression) |
| | - 8-emotion classification |
| | - Fused with attention mechanism |
| |
|
| | ### LoRA/QLoRA |
| |
|
| | Efficient fine-tuning with low-rank adaptation: |
| |
|
| | ```bash |
| | # LoRA |
| | python train.py --use_lora --lora_r 16 --lora_alpha 32 |
| | |
| | # QLoRA (4-bit quantization) |
| | python train.py --use_qlora --use_lora --lora_r 8 |
| | ``` |
| |
|
| | ## Testing |
| |
|
| | Run the test suite: |
| |
|
| | ```bash |
| | python test_model.py |
| | ``` |
| |
|
| | Tests include: |
| | - Model creation and initialization |
| | - Forward pass and gradient flow |
| | - Text generation |
| | - Multi-task outputs (EQ adapter) |
| | - Loss computation |
| |
|
| | ## Requirements |
| |
|
| | See `requirements.txt` for full dependencies. Key packages: |
| |
|
| | - torch>=2.0.0 |
| | - transformers>=4.35.0 |
| | - datasets>=2.14.0 |
| | - accelerate>=0.24.0 |
| | - peft>=0.6.0 (for LoRA) |
| | - bitsandbytes>=0.41.0 (for QLoRA) |
| | - tensorboard>=2.14.0 |
| |
|
| | ## Performance Tips |
| |
|
| | 1. **Mixed Precision**: Use `--mixed_precision bf16` for faster training (Ampere+ GPUs) |
| | 2. **Gradient Checkpointing**: Enabled by default to reduce memory |
| | 3. **Batch Size**: Adjust based on VRAM (4-8 for 7B full, 16-32 for LoRA) |
| | 4. **Sequence Length**: Longer sequences use more memory; adjust `--max_seq_length` |
| |
|
| | ## Troubleshooting |
| |
|
| | ### Out of Memory |
| | - Reduce batch size |
| | - Use gradient accumulation |
| | - Enable LoRA/QLoRA |
| | - Use mixed precision |
| | - Reduce sequence length |
| |
|
| | ### Slow Training |
| | - Increase batch size if possible |
| | - Use more gradient accumulation steps |
| | - Ensure data loading is not the bottleneck |
| | - Use mixed precision |
| |
|
| | ### Poor Quality Outputs |
| | - Train longer (more epochs) |
| | - Use higher quality data |
| | - Adjust learning rate (try 1e-5 to 5e-5) |
| | - Enable curriculum learning |
| | - Use quality filtering |
| |
|
| | ## Citation |
| |
|
| | If you use Zenith-7B in your research, please cite: |
| |
|
| | ```bibtex |
| | @misc{zenith-7b-2025, |
| | title={Zenith-7B: A Hybrid MoE Model for Code and Emotional Intelligence}, |
| | year={2025}, |
| | publisher={Zenith Project} |
| | } |
| | ``` |
| |
|
| | ## License |
| |
|
| | [Specify your license here] |
| |
|
| | ## Contact |
| |
|
| | For issues and questions, please open an issue on the project repository. |
