169Pi/Alpie-core

Model Summary

169Pi/Alpie-core is a 32B parameter causal language model.
It is the world’s first large-scale 4-bit LoRA-trained model, optimized over three distinct training phases for reasoning, knowledge integration, and benchmark performance.
The model specializes in mathematics, coding, science, competitive exams, Indian context, and law.

Model Details

Base Model: deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
Architecture: 32B parameter causal LM (chat-optimized)
Quantization: 4-bit NF4 with double quantization enabled
Precision for Inference: 4-bit NF4
Frameworks: PEFT, LoRA, bitsandbytes, PyTorch
Max Context Length: 65k tokens
Deployment Framework: vLLM
License: (to be filled)

Hyperparameters

Epochs per phase: 2
Batch Size: 256
Gradient Accumulation Steps: 4
Learning Rate: 1e-5 (initially 2e-5, reduced to avoid early over-generalization)
Scheduler: Cosine
Optimizer: AdamW (adamw_torch)
LoRA Rank (r): 16
LoRA Alpha: 8
LoRA Dropout: 0.1
Target Modules: q_proj, v_proj

Intended Use

Primary: Educational tutoring, competitive exam preparation, coding assistance, legal reasoning, general knowledge Q&A.
Secondary: Research support, problem-solving in science and mathematics.

Limitations & Warnings

May produce inaccurate or outdated information for highly recent events.
Not suitable for tasks requiring legal or medical advice without expert review.
Performance may vary outside trained domains.

Citation

If you use this model in your research, please cite: