---
license: other
license_name: deepseek-license
license_link: LICENSE
pipeline_tag: text-generation
tags:
- code
- mixture-of-experts
- SarvaCode
- india-stack
language:
- en
base_model:
- deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
---

# SarvaCode-16B-Indigenous

**SarvaCode** is an indigenously customized, open-source Mixture-of-Experts (MoE) code language model. It is built upon the DeepSeek-Coder-V2 architecture but optimized for the **Indian Software Ecosystem**. 

While global models focus on general code, SarvaCode is fine-tuned to understand **Indian English instructions**, local financial protocols (GST, TDS), and the technical frameworks of **India Stack** (UPI, ONDC, Aadhaar/UIDAI).

## 1. Key Improvements
Compared to the base Lite model, **SarvaCode** features:
- **Higher Active Parameters:** Increased from 6 to **8 active experts per token**, boosting reasoning power to **~3.2B active parameters** per message.
- **Indigenous Logic:** Enhanced accuracy for Indian-specific tasks like GST calculation logic, IFSC validation, and regional date/currency formatting.
- **India Stack Awareness:** Pre-loaded context for integrating with NPCI (UPI), ONDC, and DigiLocker APIs.
- **Massive Context:** Maintains a **128K context window** to digest entire Indian government technical gazettes or large codebases in one go.

## 2. Model Specifications

| **Model** | **#Total Params** | **#Active Params** | **Context Length** | **Specialization** |
| :---: | :---: | :---: | :---: | :---: |
| **SarvaCode-16B** | 16B | **3.2B** | 128k | India Stack & Fintech |

## 3. How to Run Locally

### Inference with Transformers
Ensure you use `trust_remote_code=True` to load the specialized MoE configuration.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_path = "./SarvaCode" # Your local directory
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()

# Example: Indian Financial Logic
input_text = "User: Write a Python function to calculate the GST for a service with an 18% slab, ensuring the output separates CGST and SGST.\n\nAssistant:"

inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))