README.md

---
license: apache-2.0
language:
- en
- es
tags:
- moe
- code
- math
- vllm
- mxfp4
- pruning
- scalai
datasets:
- HuggingFaceH4/CodeAlpaca_20K
base_model: openai/gpt-oss-120b
---

# 🚀 ScaLite-60B-Coder by SCALAI

**ScaLite-60B-Coder** is a highly specialized, bilingual (English/Spanish) Mixture-of-Experts (MoE) large language model optimized for strict logical reasoning, Python programming, and mathematics. 

Engineered by [SCALAI](https://scalai.es), this model was surgically distilled from OpenAI's dense 117B parameter MoE (`gpt-oss-120b`) down to a 60B active parameter footprint. Quantized in MXFP4, **ScaLite-60B-Coder requires only ~30GB of VRAM, making it fully deployable on a single NVIDIA L40S (48GB) GPU** with ample room for large KV-caches in production environments.

## 🧠 Model Details
* **Developer:** SCALAI
* **Model Type:** Pruned Mixture-of-Experts (MoE) Causal Language Model
* **Base Model:** `openai/gpt-oss-120b` (128 experts)
* **Pruned Architecture:** 60B active parameters (64 experts)
* **Quantization:** MXFP4
* **Context Length:** 8,192 tokens (optimized for complex Chain-of-Thought)
* **Languages:** English, Spanish

## 📊 Benchmark Performance
ScaLite-60B-Coder sheds general trivia to hyper-focus on logic and computation. Incredibly, removing the interference of general-knowledge experts allowed this pruned 60B model to **outperform its 120B parent in Computer Science**.

### MMLU-Pro (5-shot) - The Domain Specialist Effect
| Category | Original 120B Dense | **ScaLite-60B-Coder** | Delta |
| :--- | :--- | :--- | :--- |
| **Computer Science** | 39.02% | **49.27%** | 🟢 **+10.25%** |
| Math | 53.29% | 47.52% | 🟡 -5.77% |
| Law | 42.69% | 14.90% | 🔴 -27.79% (Pruned) |
| History | 40.68% | 17.06% | 🔴 -23.62% (Pruned) |

### GSM8K (Math Reasoning)
* **Pre-Healing (Pruned):** 17.59%
* **Post-Healing (Cross-Domain Code):** **61.03%** *(Zero math data leakage)*

## 🎯 Intended Use & Limitations
* **Use Cases:** Asynchronous Python coding, JSON/schema extraction, bilingual logical deduction, and structured step-by-step math problem solving.
* **Limitations:** **Do not use this model for encyclopedic trivia, historical facts, or legal advice.** The experts responsible for memorizing general internet knowledge were intentionally amputated to save 50% of the VRAM footprint. 

## 💻 How to Use (vLLM on a single L40S)
ScaLite-60B-Coder is optimized for the `vLLM` engine. Because the model features an extensive *Chain-of-Thought*, we recommend specific settings to prevent Out-Of-Memory (OOM) errors on 48GB cards.

```python
from vllm import LLM, SamplingParams

model_id = "SCALAI/ScaLite-60B-Coder"

# Load the model optimized for a single 48GB GPU (e.g., L40S)
llm = LLM(
    model=model_id,
    tensor_parallel_size=1,
    dtype="bfloat16",
    max_model_len=4096,
    gpu_memory_utilization=0.90,
    enforce_eager=True, # Disables CUDA graphs to save memory
    max_num_seqs=16,    # Limits pre-allocated KV cache 
    trust_remote_code=True
)

# Allow enough max_tokens for complex reasoning and CoT
sampling_params = SamplingParams(
    temperature=0.6,
    top_p=0.95,
    max_tokens=3500 
)

messages = [
    {"role": "system", "content": "You are a logical, bilingual AI assistant. Think step-by-step."},
    {"role": "user", "content": "Encuentra todos los pares de números enteros (x, y) que satisfacen la siguiente ecuación: x^3 + y^3 = (x+y)^2. Piensa paso a paso y justifica cada deducción lógica."}
]

prompt = llm.llm_engine.tokenizer.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt