|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
- es |
|
|
- zh |
|
|
tags: |
|
|
- qwen |
|
|
- qwen3-4b |
|
|
- unsloth |
|
|
- midnight-ai |
|
|
- enosis-labs |
|
|
- text-generation |
|
|
- code-generation |
|
|
- mathematics |
|
|
- reasoning |
|
|
- fine-tuned |
|
|
- MMLU |
|
|
- HumanEval |
|
|
- HellaSwag |
|
|
- Winogrande |
|
|
- LAMBADA |
|
|
- CEVAL |
|
|
pipeline_tag: text-generation |
|
|
model_name: Midnight Mini High Thinking |
|
|
model_id: enosislabs/midnight-mini-high-thinking-exp |
|
|
base_model: Qwen/Qwen3-4B |
|
|
datasets: |
|
|
- enosislabs/math-mini-shareGPT |
|
|
- enosislabs/midnight-mini-think-shareGPT |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# Midnight Mini High Thinking: Efficient Reasoning Architecture |
|
|
|
|
|
**Model ID:** `midnight-mini-high-thinking-05-25` |
|
|
**Developed by:** Enosis Labs AI Research Division |
|
|
**Model Version:** 05-25 (Production Release) |
|
|
**Base Architecture:** Qwen3-4B |
|
|
|
|
|
## Executive Summary |
|
|
|
|
|
Midnight Mini High Thinking is a state-of-the-art causal language model engineered for complex reasoning applications within enterprise environments. This 4-billion parameter architecture delivers sophisticated analytical capabilities through advanced fine-tuning methodologies, demonstrating superior performance in mathematical computation, logical reasoning, and code synthesis tasks while maintaining computational efficiency for production deployment. |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Core Architecture |
|
|
|
|
|
- **Base Model:** Qwen/Qwen3-4B |
|
|
- **Parameter Count:** 4.02 billion trainable parameters |
|
|
- **Model Type:** Autoregressive Transformer (Causal Language Model) |
|
|
- **Fine-tuning Framework:** Unsloth optimization pipeline |
|
|
- **Quantization Support:** Native 16-bit precision, GGUF quantized variants (Q4_K_M, Q5_K_M, Q8_0) |
|
|
- **Maximum Context Length:** 32,768 tokens |
|
|
- **Vocabulary Size:** 151,936 tokens |
|
|
- **Attention Heads:** 32 (Multi-Head Attention) |
|
|
- **Hidden Dimensions:** 2,048 |
|
|
- **Feed-Forward Network Dimensions:** 11,008 |
|
|
|
|
|
### Performance Characteristics |
|
|
|
|
|
The model architecture incorporates several advanced optimizations: |
|
|
|
|
|
- **Enhanced Attention Mechanisms:** Specialized for multi-step reasoning workflows with improved long-range dependency modeling |
|
|
- **Parameter-Efficient Fine-Tuning:** Utilizing LoRA (Low-Rank Adaptation) and QLoRA techniques for optimal training efficiency |
|
|
- **Memory Optimization:** Gradient checkpointing and mixed-precision training for reduced memory footprint during inference |
|
|
- **Inference Optimization:** Native support for key-value cache optimization and dynamic batching |
|
|
|
|
|
### Deployment Formats |
|
|
|
|
|
#### 16-bit Precision Model |
|
|
|
|
|
- **Memory Requirements:** ~8GB VRAM (inference) |
|
|
- **Inference Speed:** ~150-200 tokens/second (RTX 4090) |
|
|
- **Precision:** Full fp16 precision for maximum accuracy |
|
|
|
|
|
#### GGUF Quantized Variants |
|
|
|
|
|
- **Q4_K_M:** 2.6GB, optimal balance of quality and efficiency |
|
|
- **Q5_K_M:** 3.2GB, enhanced quality with moderate compression |
|
|
- **Q8_0:** 4.3GB, near-original quality with minimal compression |
|
|
|
|
|
## Core Capabilities & Design Objectives |
|
|
|
|
|
Midnight Mini High Thinking is specifically engineered for enterprise applications requiring sophisticated analytical capabilities: |
|
|
|
|
|
### Primary Capabilities |
|
|
|
|
|
- **Advanced Multi-Step Reasoning:** Demonstrates exceptional performance in complex logical sequences requiring iterative analysis and synthesis |
|
|
- **Mathematical Computation & Analysis:** Excels in advanced mathematical operations, theorem proving, and quantitative analysis |
|
|
- **Code Generation & Software Engineering:** Proficient in generating, debugging, and optimizing code across multiple programming languages |
|
|
- **Technical Documentation Processing:** Advanced comprehension and generation of technical documentation, research papers, and analytical reports |
|
|
- **Multilingual Intelligence:** Primary optimization for English with demonstrated capabilities in Spanish and Chinese for specialized tasks |
|
|
|
|
|
### Design Principles |
|
|
|
|
|
- **Ethical AI Framework:** Integrated safety mechanisms for responsible AI deployment |
|
|
- **Bias Mitigation:** Advanced training protocols designed to minimize harmful biases and promote equitable outputs |
|
|
- **Computational Efficiency:** Optimized for production environments with resource-conscious design |
|
|
- **Scalability:** Architecture designed for horizontal scaling in enterprise deployments |
|
|
|
|
|
## Enterprise Applications & Use Cases |
|
|
|
|
|
Midnight Mini High Thinking is architected for professional environments requiring sophisticated analytical capabilities: |
|
|
|
|
|
### Primary Application Domains |
|
|
|
|
|
- **Advanced Mathematical Research:** Complex problem solving, theorem verification, mathematical proof assistance, and quantitative analysis |
|
|
- **Software Engineering & Development:** Code generation, debugging assistance, architecture planning, and technical documentation |
|
|
- **Business Intelligence & Analytics:** Data analysis interpretation, report generation, and strategic decision support |
|
|
- **Academic Research Support:** Literature analysis, research methodology assistance, and technical writing enhancement |
|
|
- **Educational Technology:** Advanced tutoring systems, curriculum development, and personalized learning assistance |
|
|
|
|
|
### Implementation Examples |
|
|
|
|
|
#### Mathematical Analysis Implementation |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
import torch |
|
|
|
|
|
# Initialize model with optimized settings |
|
|
model_id = "enosislabs/midnight-mini-high-thinking-05-25" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_id, |
|
|
torch_dtype=torch.float16, |
|
|
device_map="auto" |
|
|
) |
|
|
|
|
|
# Mathematical reasoning example |
|
|
prompt = """Analyze the convergence properties of the Taylor series for e^x around x=0. |
|
|
Provide a rigorous mathematical explanation including convergence radius and error bounds.""" |
|
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
with torch.no_grad(): |
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=400, |
|
|
temperature=0.7, |
|
|
do_sample=True, |
|
|
top_p=0.9 |
|
|
) |
|
|
|
|
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
print(f"Mathematical Analysis:\n{response}") |
|
|
``` |
|
|
|
|
|
#### Code Generation & Technical Documentation |
|
|
|
|
|
```python |
|
|
# Advanced code generation with documentation |
|
|
coding_prompt = """Design a Python class for implementing a thread-safe LRU cache |
|
|
with TTL (time-to-live) functionality. Include comprehensive documentation |
|
|
and error handling.""" |
|
|
|
|
|
inputs = tokenizer(coding_prompt, return_tensors="pt") |
|
|
with torch.no_grad(): |
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=500, |
|
|
temperature=0.3, |
|
|
do_sample=True |
|
|
) |
|
|
|
|
|
code_response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
print(f"Generated Solution:\n{code_response}") |
|
|
``` |
|
|
|
|
|
## Training Methodology & Data Engineering |
|
|
|
|
|
### Training Infrastructure |
|
|
|
|
|
- **Base Model:** Qwen/Qwen3-4B |
|
|
- **Fine-tuning Framework:** Unsloth optimization pipeline with custom extensions |
|
|
- **Hardware Configuration:** Multi-GPU training environment (A100 80GB clusters) |
|
|
- **Training Duration:** 72 hours of optimized training across distributed systems |
|
|
- **Optimization Strategy:** Parameter-efficient fine-tuning with LoRA and gradient accumulation |
|
|
|
|
|
### Dataset Composition & Curation |
|
|
|
|
|
The training regimen incorporates a proprietary, meticulously curated dataset collection designed to enhance analytical capabilities: |
|
|
|
|
|
- **Mathematical Reasoning Corpus:** Advanced mathematical problems, proofs, and analytical reasoning chains |
|
|
- **Code Generation Suite:** Multi-language programming challenges with comprehensive documentation requirements |
|
|
- **Technical Documentation Archive:** Scientific papers, technical specifications, and analytical reports |
|
|
- **Ethical Alignment Dataset:** Carefully curated examples promoting responsible AI behavior and bias mitigation |
|
|
- **Multilingual Reasoning Collection:** Cross-linguistic reasoning tasks with emphasis on knowledge transfer |
|
|
|
|
|
### Training Optimization Techniques |
|
|
|
|
|
- **Gradient Checkpointing:** Memory-efficient training enabling larger effective batch sizes |
|
|
- **Mixed Precision Training:** FP16 optimization for accelerated training without precision loss |
|
|
- **Dynamic Learning Rate Scheduling:** Adaptive learning rate adjustment based on validation performance |
|
|
- **Regularization Strategies:** Dropout, weight decay, and label smoothing for improved generalization |
|
|
|
|
|
## Performance Benchmarks & Evaluation Results |
|
|
|
|
|
Midnight Mini High Thinking has undergone comprehensive evaluation across industry-standard benchmarks, demonstrating exceptional performance characteristics for its parameter class. |
|
|
|
|
|
### Benchmark Results Overview |
|
|
|
|
|
| Benchmark Category | Task Specification | Metric | Score | Standard Error | |
|
|
|:-------------------|:-------------------|:-------|:------|:---------------| |
|
|
| **Code Generation** | | | | | |
|
|
| | HumanEval | `pass@1` | 0.5920 | ±0.0389 | |
|
|
| **Common Sense Reasoning** | | | | | |
|
|
| | HellaSwag | `acc` | 0.5074 | ±0.0050 | |
|
|
| | | `acc_norm` | 0.6782 | ±0.0047 | |
|
|
| | Winogrande | `acc` | 0.6748 | ±0.0132 | |
|
|
| **Language Modeling** | | | | | |
|
|
| | LAMBADA OpenAI (English) | `acc` | 0.6218 | ±0.0068 | |
|
|
| | | `perplexity` | 5.8048 | ±0.1720 | |
|
|
| **Knowledge & Reasoning** | | | | | |
|
|
| | MMLU (English) - General | `acc` | 0.6920 | ±0.0453 | |
|
|
| | MMLU (English) - STEM | `acc` | 0.5870 | ±0.0734 | |
|
|
| | MMLU (Spanish) - General | `acc` | 0.6050 | ±0.0246 | |
|
|
| | MMLU (Spanish) - STEM | `acc` | 0.6304 | ±0.0720 | |
|
|
| **Specialized Knowledge** | | | | | |
|
|
| | CEVAL - Advanced Mathematics | `acc` | 0.5863 | ±0.1177 | |
|
|
|
|
|
### Performance Analysis |
|
|
|
|
|
**Code Generation Excellence:** The 59.2% pass@1 score on HumanEval demonstrates superior code synthesis capabilities, positioning the model among the top performers in its parameter class for software engineering applications. |
|
|
|
|
|
**Knowledge Integration:** MMLU performance of 69.2% (English) indicates strong knowledge retention and application across diverse domains, with particularly notable STEM performance in Spanish (63.04%) suggesting effective cross-linguistic knowledge transfer. |
|
|
|
|
|
**Reasoning Capabilities:** Winogrande accuracy of 67.48% and HellaSwag normalized accuracy of 67.82% demonstrate robust common-sense reasoning and contextual understanding. |
|
|
|
|
|
**Mathematical Proficiency:** CEVAL mathematics performance of 58.63% showcases specialized mathematical reasoning capabilities, particularly valuable for technical and scientific applications. |
|
|
|
|
|
## Model Limitations & Risk Assessment |
|
|
|
|
|
### Technical Constraints |
|
|
|
|
|
- **Knowledge Temporal Boundary:** Training data cutoff limits real-time information access and contemporary knowledge integration |
|
|
- **Computational Resource Requirements:** 4B parameter architecture demands significant computational resources for optimal performance |
|
|
- **Context Window Limitations:** 32,768 token limit may constrain processing of extremely large documents or extended conversations |
|
|
- **Quantization Trade-offs:** GGUF variants exhibit quality degradation proportional to compression level |
|
|
|
|
|
### Performance Limitations |
|
|
|
|
|
- **Hallucination Potential:** Like all large language models, may generate factually incorrect or logically inconsistent outputs |
|
|
- **Domain-Specific Accuracy:** Performance varies across specialized domains; validation recommended for critical applications |
|
|
- **Language Proficiency Variance:** Optimal performance in English with graduated capabilities in Spanish and Chinese |
|
|
- **Reasoning Depth Constraints:** Complex multi-step reasoning may occasionally exhibit logical gaps or incomplete analysis |
|
|
|
|
|
### Bias & Fairness Considerations |
|
|
|
|
|
- **Training Data Bias Inheritance:** May reflect societal biases present in training corpora despite mitigation efforts |
|
|
- **Cultural Context Limitations:** Responses may exhibit Western-centric perspectives due to training data composition |
|
|
- **Demographic Representation:** Potential underrepresentation of certain demographic groups in training examples |
|
|
- **Professional Domain Bias:** May exhibit preferences toward certain professional or academic perspectives |
|
|
|
|
|
## Ethical Framework & Responsible AI Implementation |
|
|
|
|
|
### Safety Mechanisms |
|
|
|
|
|
- **Content Safety Filters:** Integrated mechanisms to identify and refuse harmful content generation |
|
|
- **Bias Detection & Mitigation:** Ongoing monitoring for discriminatory outputs with corrective measures |
|
|
- **Harmful Use Prevention:** Design features to discourage malicious applications and misuse |
|
|
- **Privacy Protection:** No retention of user inputs or personal data during inference |
|
|
|
|
|
### Deployment Guidelines |
|
|
|
|
|
- **Human Oversight Requirement:** Critical decisions should maintain human validation and review |
|
|
- **Domain-Specific Validation:** Professional applications require subject matter expert verification |
|
|
- **Continuous Monitoring:** Regular assessment of outputs for quality and ethical compliance |
|
|
- **User Education:** Clear communication of model capabilities and limitations to end users |
|
|
|
|
|
### Research Ethics Compliance |
|
|
|
|
|
Development adheres to established AI research ethics principles: |
|
|
|
|
|
- **Beneficence:** Designed to augment human capabilities and provide positive societal impact |
|
|
- **Non-maleficence:** Active measures to prevent harmful applications and negative consequences |
|
|
- **Autonomy:** Respects user agency while providing transparent information about model behavior |
|
|
- **Justice:** Efforts to ensure equitable access and fair treatment across user populations |
|
|
|
|
|
## Technical Support & Model Citation |
|
|
|
|
|
### Model Attribution |
|
|
|
|
|
When utilizing Midnight Mini High Thinking in research or production environments, please cite: |
|
|
|
|
|
```bibtex |
|
|
@software{midnight_mini_high_thinking_2025, |
|
|
author = {Enosis Labs AI Research Division}, |
|
|
title = { Midnight Mini High Thinking: Efficient Reasoning Architecture}, |
|
|
version = {05-25}, |
|
|
year = {2025}, |
|
|
publisher = {Enosis Labs}, |
|
|
url = {https://huggingface.co/enosislabs/midnight-mini-high-thinking-exp} |
|
|
} |
|
|
``` |
|
|
|
|
|
### Technical Support Channels |
|
|
|
|
|
For technical inquiries, deployment assistance, or research collaboration: |
|
|
|
|
|
- **Primary Contact:** <contact@enosislabs.com> |
|
|
- **Model Repository:** [Hugging Face Model Hub](https://huggingface.co/enosislabs/midnight-mini-high-thinking-exp) |
|
|
|
|
|
### License & Distribution |
|
|
|
|
|
Licensed under Apache 2.0, permitting commercial use, modification, and distribution with appropriate attribution. |
|
|
|
|
|
--- |
|
|
|
|
|
**Enosis Labs AI Research Division** |
|
|
*Advancing the frontiers of artificial intelligence through responsible innovation* |
