| | --- |
| | license: other |
| | license_name: deepseek-license |
| | license_link: LICENSE |
| | pipeline_tag: text-generation |
| | tags: |
| | - code |
| | - mixture-of-experts |
| | - SarvaCode |
| | - india-stack |
| | language: |
| | - en |
| | base_model: |
| | - deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct |
| | --- |
| | |
| | # SarvaCode-16B-Indigenous |
| |
|
| | **SarvaCode** is an indigenously customized, open-source Mixture-of-Experts (MoE) code language model. It is built upon the DeepSeek-Coder-V2 architecture but optimized for the **Indian Software Ecosystem**. |
| |
|
| | While global models focus on general code, SarvaCode is fine-tuned to understand **Indian English instructions**, local financial protocols (GST, TDS), and the technical frameworks of **India Stack** (UPI, ONDC, Aadhaar/UIDAI). |
| |
|
| | ## 1. Key Improvements |
| | Compared to the base Lite model, **SarvaCode** features: |
| | - **Higher Active Parameters:** Increased from 6 to **8 active experts per token**, boosting reasoning power to **~3.2B active parameters** per message. |
| | - **Indigenous Logic:** Enhanced accuracy for Indian-specific tasks like GST calculation logic, IFSC validation, and regional date/currency formatting. |
| | - **India Stack Awareness:** Pre-loaded context for integrating with NPCI (UPI), ONDC, and DigiLocker APIs. |
| | - **Massive Context:** Maintains a **128K context window** to digest entire Indian government technical gazettes or large codebases in one go. |
| |
|
| | ## 2. Model Specifications |
| |
|
| | | **Model** | **#Total Params** | **#Active Params** | **Context Length** | **Specialization** | |
| | | :---: | :---: | :---: | :---: | :---: | |
| | | **SarvaCode-16B** | 16B | **3.2B** | 128k | India Stack & Fintech | |
| |
|
| | ## 3. How to Run Locally |
| |
|
| | ### Inference with Transformers |
| | Ensure you use `trust_remote_code=True` to load the specialized MoE configuration. |
| |
|
| | ```python |
| | from transformers import AutoTokenizer, AutoModelForCausalLM |
| | import torch |
| | |
| | model_path = "./SarvaCode" # Your local directory |
| | tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) |
| | model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda() |
| | |
| | # Example: Indian Financial Logic |
| | input_text = "User: Write a Python function to calculate the GST for a service with an 18% slab, ensuring the output separates CGST and SGST.\n\nAssistant:" |
| | |
| | inputs = tokenizer(input_text, return_tensors="pt").to(model.device) |
| | outputs = model.generate(**inputs, max_new_tokens=256) |
| | print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
