Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,175 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.5-27B
+tags:
+  - text-to-sql
+  - sql
+  - qwen3.5
+  - fine-tuned
+  - fsdp
+  - nebius
+datasets:
+  - b-mc2/sql-create-context
+  - gretelai/synthetic_text_to_sql
+language:
+  - en
+pipeline_tag: text-generation
+---
+
+# Qwen3.5-27B-Text2SQL
+
+Fine-tuned [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) for **Text-to-SQL** generation. Given a database schema and a natural language question, the model outputs a clean SQL query.
+
+## Key Results
+
+| Metric | Base Model | This Model | Improvement |
+|---|---|---|---|
+| **SQL Execution Accuracy** | 19.5% | **61.0%** | **+41.5%** |
+| **Valid SQL Output** | 41.5% | **90.2%** | +48.7% |
+| **Spider Exact Match** | 0.0% | **22.2%** | +22.2% |
+| **Spider Keyword Score** | 45.5% | **85.4%** | +39.9% |
+| **Clean SQL Format** | 0% | **100%** | +100% |
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "mahernaija/Qwen3.5-27B-Text2SQL",
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True,
+)
+tokenizer = AutoTokenizer.from_pretrained(
+    "mahernaija/Qwen3.5-27B-Text2SQL",
+    trust_remote_code=True,
+)
+
+schema = "CREATE TABLE employees (id INTEGER, name TEXT, department TEXT, salary REAL);"
+question = "Find all employees in Engineering with salary above 90000."
+
+prompt = f"<|im_start|>system\nYou are a SQL expert. Given a database schema and a natural language question, write the correct SQL query.<|im_end|>\n<|im_start|>user\nSchema: {schema}\nQuestion: {question}<|im_end|>\n<|im_start|>assistant\n"
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=256, temperature=0)
+response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+print(response)
+# SELECT * FROM employees WHERE department = 'Engineering' AND salary > 90000
+```
+
+### With vLLM
+
+```bash
+vllm serve mahernaija/Qwen3.5-27B-Text2SQL --tensor-parallel-size 2
+```
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
+
+response = client.chat.completions.create(
+    model="mahernaija/Qwen3.5-27B-Text2SQL",
+    messages=[
+        {"role": "system", "content": "You are a SQL expert. Given a database schema and a natural language question, write the correct SQL query."},
+        {"role": "user", "content": "Schema: CREATE TABLE products (id INT, name TEXT, price REAL);\nQuestion: What are the 3 most expensive products?"},
+    ],
+    max_tokens=256,
+    temperature=0,
+)
+print(response.choices[0].message.content)
+# SELECT name FROM products ORDER BY price DESC LIMIT 3
+```
+
+## Training Details
+
+| Parameter | Value |
+|---|---|
+| **Base model** | Qwen/Qwen3.5-27B (26.9B params) |
+| **Method** | Full fine-tuning with FSDP |
+| **Hardware** | 16× NVIDIA H200 (2 nodes, Nebius AI Cloud) |
+| **Training time** | 3 hours 6 minutes |
+| **Dataset** | sql-create-context (78K) + Gretel synthetic (100K) = 178K samples |
+| **Epochs** | 1 |
+| **Batch size** | 2 per GPU × 8 grad accum × 16 GPUs = 256 global |
+| **Learning rate** | 2e-5 (cosine decay, 5% warmup) |
+| **Sequence length** | 512 (SQL samples P99=373 tokens) |
+| **Precision** | BF16 |
+| **GPU utilization** | 98-100% |
+| **Final train loss** | 0.144 |
+| **Final eval loss** | 0.176 |
+
+### Data Preprocessing
+
+- Schemas stripped of INSERT data (model learns SQL from schema structure, not memorized answers)
+- Manual chat format (bypasses Qwen3.5 `<think>` tag injection)
+- Label masking: loss only on SQL output, prompt tokens masked with -100
+- Deduplication + contamination check between train and eval splits
+
+### Evaluation
+
+**Gretel Execution Accuracy** (gold standard — runs SQL in SQLite, compares results):
+
+| Complexity | Base | This Model |
+|---|---|---|
+| Basic SQL | 23.8% | **71.4%** |
+| Aggregation | 18.2% | **54.5%** |
+| Single JOIN | 25.0% | **75.0%** |
+| Window functions | 0.0% | **33.3%** |
+
+**Spider Benchmark** (1,034 dev questions, public standard):
+- Exact match: 22.2%
+- Keyword score: 85.4%
+
+### Known Limitations
+
+**Catastrophic forgetting**: Full fine-tuning on SQL-only data caused regression in general capabilities. The model tries to answer non-SQL questions with SQL (56% SQL contamination on general prompts). For production use with mixed tasks, consider:
+- LoRA fine-tuning instead of full FT
+- Mixed training data (SQL + general chat)
+- Using this model only for SQL-specific pipelines
+
+### Regression Test
+
+| Category | Base | This Model | SQL Contamination |
+|---|---|---|---|
+| General Knowledge | 84% | 44% | 2/5 |
+| Math | 100% | 40% | 3/5 |
+| Code | 48% | 47% | 5/5 |
+| Language | 90% | 78% | 4/5 |
+| Common Sense | 82% | 93% | 0/5 |
+
+## Architecture
+
+- **Architecture**: Qwen3_5ForConditionalGeneration (VLM with text + vision)
+- **Text backbone**: 64 layers, hidden_size=5120, 24 attention heads, 4 KV heads
+- **Attention**: Hybrid GDN (linear_attention + full_attention)
+- **Context**: 262K tokens
+- **Vision**: Built-in 27-layer ViT (weights from base model, not finetuned)
+
+## Files
+
+| File | Description |
+|---|---|
+| `model-00001-of-00003.safetensors` | Text backbone weights (shard 1/3) |
+| `model-00002-of-00003.safetensors` | Text backbone weights (shard 2/3) |
+| `model-00003-of-00003.safetensors` | Text backbone weights (shard 3/3) |
+| `model-visual.safetensors` | Vision encoder weights (from base model) |
+| `config.json` | Full VLM config (required by vLLM) |
+| `tokenizer.json` | Tokenizer |
+
+## Citation
+
+```bibtex
+@misc{naija2026qwen35text2sql,
+  title={Qwen3.5-27B-Text2SQL: Fine-tuned Qwen 3.5 for Text-to-SQL},
+  author={Maher Naija},
+  year={2026},
+  url={https://huggingface.co/mahernaija/Qwen3.5-27B-Text2SQL}
+}
+```
+
+## Acknowledgments
+
+- Trained on [Nebius AI Cloud](https://nebius.com) using Soperator (Slurm on Kubernetes)
+- Base model: [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) by Alibaba
+- Training data: [sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) + [Gretel synthetic_text_to_sql](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)