Upload folder using huggingface_hub

README.md

@@ -1,84 +1,75 @@
 ---
-language:
-- en
-- pt
-- zh
-license: apache-2.0
 base_model: Qwen/Qwen2.5-0.5B-Instruct
+library_name: peft
+pipeline_tag: text-generation
 tags:
-- knowledge-distillation
+- base_model:adapter:Qwen/Qwen2.5-0.5B-Instruct
 - lora
-- peft
-- qwen2
-- geometry-distillation
+- transformers
+- knowledge-distillation
 - cka
-pipeline_tag: text-generation
+license: mit
 ---
 
 # Deku — One for All Student
 
-Deku is a **Qwen2.5-0.5B-Instruct** fine-tuned with LoRA via gated CKA geometry distillation (Path B). It absorbs representation structure from 5 heterogeneous teacher LLMs simultaneously, without access to teacher logits or shared tokenizers.
+Qwen2.5-0.5B-Instruct fine-tuned via **gated CKA geometry distillation** from 5 heterogeneous teacher LLMs. The student learns to absorb the representation geometry of multiple teachers simultaneously through a learned routing gate.
 
-## Model Details
+## Teachers
 
-- **Base model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
-- **Fine-tuning method:** LoRA (PEFT) + GatingNetwork (linear head over student hidden states)
-- **Distillation strategy:** Geometry-only (CKA) — Path B, tokenizer-agnostic
-- **Language(s):** English, Portuguese, Chinese (inherited from base)
-- **License:** Apache 2.0
-- **Developed by:** build-small-hackathon team
+| Model | Strength |
+|---|---|
+| Qwen2.5-1.5B-Instruct | code, structured reasoning |
+| SmolLM2-1.7B-Instruct | curated quality |
+| Phi-3.5-mini-instruct | instruction following, CoT |
+| gemma-2-2b-it | long context |
+| MiniCPM-2B-sft-bf16 | multilingual, efficiency |
 
-## Teachers
+## Method
+
+**Path B — geometry-only, tokenizer-agnostic distillation.**
 
-| Teacher | Parameters | Hidden dim |
-|---------|-----------|------------|
-| Qwen2.5-1.5B-Instruct | 1.5B | 1536 |
-| SmolLM2-1.7B-Instruct | 1.7B | 2048 |
-| Phi-3.5-mini-instruct | 3.8B | 3072 |
-| gemma-2-2b-it | 2.7B | 2304 |
-| MiniCPM-2B-sft-bf16 | 2.7B | 2304 |
+Each teacher has a different tokenizer and hidden dimension, making token-level KL divergence ill-defined across the ensemble. Instead, the student learns to align its hidden-state geometry with each teacher via **CKA (Centered Kernel Alignment)**, weighted by a learned gating network that routes each input to the most relevant teacher.
 
-## Distillation Approach
+The objective is:
 
-**Path B — geometry-only, tokenizer-agnostic:**
+```
+L = λ1·L_task + λ2·L_KL(Qwen1.5B) + λ3·L_geo(gate)
+```
 
-1. Each teacher processes its own tokenization of the same text. No shared vocabulary required.
-2. Sequence representations are pooled via masked mean (attention mask weighted) to a single vector per model.
-3. Linear projection heads map each teacher's hidden space into the student's hidden space (d=896).
-4. A **GatingNetwork** (linear layer over student pooled state → softmax over 5 teachers) learns which teacher's geometry to prioritize per input.
-5. Loss = task cross-entropy + λ·CKA geometry loss (student vs. gated teacher mixture).
+- `L_task` — next-token cross-entropy on the training mix
+- `L_KL` — KL divergence from Qwen2.5-1.5B (same tokenizer, zero friction)
+- `L_geo` — gated CKA loss: `1 - mean_i gate_i · CKA(H_student, Pi_i · H_teacher_i)`
 
-The CKA geometry loss aligns the *relational structure* of representations (which samples are similar to which) rather than raw activation values, making it robust to dimension mismatch and tokenizer differences.
+Lambdas follow a three-phase curriculum: task-only warmup → KL ramp-in → geometry ramp-in.
+
+## Training
+
+- **Base:** Qwen/Qwen2.5-0.5B-Instruct
+- **Adapter:** LoRA r=64, α=128 on all attention + MLP projections
+- **Data:** OpenHermes-2.5 (70%) + GSM8K (20%) + ARC-Challenge (10%)
+- **Steps:** 5 000 · batch 8 · seq 512
+- **Hardware:** A100-80GB via Modal
+- **Precision:** bfloat16
 
 ## Usage
 
 ```python
-from transformers import AutoTokenizer
-from peft import PeftModel, AutoPeftModelForCausalLM
-
-model = AutoPeftModelForCausalLM.from_pretrained(
-    "build-small-hackathon/deku",
-    torch_dtype="auto",
-    device_map="auto",
-)
-tokenizer = AutoTokenizer.from_pretrained("build-small-hackathon/deku")
-
-inputs = tokenizer("Explain gradient descent in one sentence.", return_tensors="pt").to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=128)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
-```
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
 
-## Additional Artifacts
+base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+model = PeftModel.from_pretrained(base, "build-small-hackathon/deku")
+tok = AutoTokenizer.from_pretrained("build-small-hackathon/deku")
 
-- **GatingNetwork weights:** `gating.pt` — `torch.load("gating.pt")`, `state_dict` for a `nn.Linear(896, 5)`
-- **Projection weights:** `projections.pt` — list of 5 `nn.Linear` state dicts (teacher_i → student space)
-- **Visualization data:** [build-small-hackathon/ofa-viz-data](https://huggingface.co/datasets/build-small-hackathon/ofa-viz-data) — raw projected embeddings for 3D UMAP soul space
-- **Interactive Space:** [build-small-hackathon/one-for-all](https://huggingface.co/spaces/build-small-hackathon/one-for-all)
+inputs = tok("Explain what a hash map is.", return_tensors="pt")
+out = model.generate(**inputs, max_new_tokens=200)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
 
-## Training
+## Demo
 
-- **Steps:** 5000
-- **Batch size:** 4 (gradient accumulation × 4 = effective 16)
-- **Optimizer:** AdamW, lr=2e-4, cosine decay
-- **Hardware:** Modal A10G (24 GB VRAM)
-- **Data:** subset of [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) (all-Pro split)
+Live soul space + probe interface: [build-small-hackathon/one-for-all](https://huggingface.co/spaces/build-small-hackathon/one-for-all)
+
+---
+PEFT 0.19.1

adapter_config.json

@@ -30,13 +30,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "q_proj",
     "down_proj",
+    "v_proj",
     "k_proj",
     "up_proj",
-    "o_proj",
-    "q_proj",
     "gate_proj",
-    "v_proj"
+    "o_proj"
   ],
   "target_parameters": null,
   "task_type": "CAUSAL_LM",

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9163bd31e7daa3eac668e4f6fe3bb43adc89ac6e6433b234909b1c5be34a2148
+oid sha256:4efd053da21f51c9efa349ed66951904f5f7aaaca866de90f22b69df01ab3745
 size 140815952

gating.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7cb71343d2d3ca1ea32a0fe820f09acf28893c9739611b9a4c67bc419c2975f2
+oid sha256:4b0822147ff61b1a76a8df4424596b4c5af5e7a974f652d5841e5ee30580ecb5
 size 19805

projections.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:11f7e53dfb1ed34903ddd1a93f8760adaf36cf09f33d1625a11deafa18e485c9
+oid sha256:d340a923e58d0be9422a08aaa96124e10e7db75e84a0401943bc3cbaf7e05e35
 size 40373189