Create README.md

README.md

@@ -0,0 +1,92 @@
+---
+license: apache-2.0
+language:
+- en
+- es
+tags:
+- moe
+- code
+- math
+- vllm
+- mxfp4
+- pruning
+- scalai
+datasets:
+- HuggingFaceH4/CodeAlpaca_20K
+base_model: openai/gpt-oss-120b
+---
+
+# 🚀 ScaLite-60B-Coder by SCALAI
+
+**ScaLite-60B-Coder** is a highly specialized, bilingual (English/Spanish) Mixture-of-Experts (MoE) large language model optimized for strict logical reasoning, Python programming, and mathematics. 
+
+Engineered by [SCALAI](https://scalai.es), this model was surgically distilled from OpenAI's dense 117B parameter MoE (`gpt-oss-120b`) down to a 60B active parameter footprint. Quantized in MXFP4, **ScaLite-60B-Coder requires only ~30GB of VRAM, making it fully deployable on a single NVIDIA L40S (48GB) GPU** with ample room for large KV-caches in production environments.
+
+## 🧠 Model Details
+* **Developer:** SCALAI (Vicens Gaitan)
+* **Model Type:** Pruned Mixture-of-Experts (MoE) Causal Language Model
+* **Base Model:** `openai/gpt-oss-120b` (128 experts)
+* **Pruned Architecture:** 60B active parameters (64 experts)
+* **Quantization:** MXFP4
+* **Context Length:** 8,192 tokens (optimized for complex Chain-of-Thought)
+* **Languages:** English, Spanish
+
+## 🔬 The Innovation: Activation-Guided Sparsity & Router Healing
+Standard magnitude-based pruning often lobotomizes minority capabilities like secondary languages or strict syntax. To build ScaLite-60B, we pioneered a behavioral approach:
+
+1. **Activation-Guided Sparsity:** We injected forward hooks into the original 120B model and passed a bilingual calibration dataset of complex math and code. We tracked actual expert utilization and permanently severed the 64 experts that represented the "encyclopedic long tail," preserving only the structural and logical specialists.
+2. **Cross-Domain Router Healing:** Pruning 50% of the experts causes "Router Trauma" (probability misalignment). Instead of retraining on math (which risks data leakage), we froze the surviving experts and fine-tuned the router for 3,000 steps *exclusively* on Python code (`CodeAlpaca_20K`). This taught the router structural discipline, which miraculously generalized to mathematical reasoning.
+
+## 📊 Benchmark Performance
+ScaLite-60B-Coder sheds general trivia to hyper-focus on logic and computation. Incredibly, removing the interference of general-knowledge experts allowed this pruned 60B model to **outperform its 120B parent in Computer Science**.
+
+### MMLU-Pro (5-shot) - The Domain Specialist Effect
+| Category | Original 120B Dense | **ScaLite-60B-Coder** | Delta |
+| :--- | :--- | :--- | :--- |
+| **Computer Science** | 39.02% | **49.27%** | 🟢 **+10.25%** |
+| Math | 53.29% | 47.52% | 🟡 -5.77% |
+| Law | 42.69% | 14.90% | 🔴 -27.79% (Pruned) |
+| History | 40.68% | 17.06% | 🔴 -23.62% (Pruned) |
+
+### GSM8K (Math Reasoning)
+* **Pre-Healing (Traumatized):** 17.59%
+* **Post-Healing (Cross-Domain Code):** **61.03%** *(Zero math data leakage)*
+
+## 🎯 Intended Use & Limitations
+* **Use Cases:** Asynchronous Python coding, JSON/schema extraction, bilingual logical deduction, and structured step-by-step math problem solving.
+* **Limitations:** **Do not use this model for encyclopedic trivia, historical facts, or legal advice.** The experts responsible for memorizing general internet knowledge were intentionally amputated to save 50% of the VRAM footprint. 
+
+## 💻 How to Use (vLLM on a single L40S)
+ScaLite-60B-Coder is optimized for the `vLLM` engine. Because the model features an extensive *Chain-of-Thought*, we recommend specific settings to prevent Out-Of-Memory (OOM) errors on 48GB cards.
+
+```python
+from vllm import LLM, SamplingParams
+
+model_id = "SCALAI/ScaLite-60B-Coder"
+
+# Load the model optimized for a single 48GB GPU (e.g., L40S)
+llm = LLM(
+    model=model_id,
+    tensor_parallel_size=1,
+    dtype="bfloat16",
+    max_model_len=4096,
+    gpu_memory_utilization=0.90,
+    enforce_eager=True, # Disables CUDA graphs to save memory
+    max_num_seqs=16,    # Limits pre-allocated KV cache 
+    trust_remote_code=True
+)
+
+# Allow enough max_tokens for complex reasoning and CoT
+sampling_params = SamplingParams(
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=3500 
+)
+
+messages = [
+    {"role": "system", "content": "You are a logical, bilingual AI assistant. Think step-by-step."},
+    {"role": "user", "content": "Encuentra todos los pares de números enteros (x, y) que satisfacen la siguiente ecuación: x^3 + y^3 = (x+y)^2. Piensa paso a paso y justifica cada deducción lógica."}
+]
+
+prompt = llm.llm_engine.tokenizer.tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt