π Qwen35b-Agent-R2O3 β Agent-R2 + Ornith (Ξ±=0.3)
Built on Qwen35B-Agent-R2 with SVD-extracted Ornith LoRA. The best of both worlds.
--- ## 𧬠How This Model Was Built ``` βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β Qwen35b-Agent-R2O3 Construction β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€ β β β Qwen35B-Agent-R2 (Base) ββββββββββ 70% weights kept β β β β β βββ 7 LoRAs already fused: β β β Opus | Fable | Routing | Tool | Math | Mythos β β β | ToolFmt (all trained via SFT) β β β β β βββ + Ornith LoRA (Ξ±=0.3) β SVD Weight-Diff β β β β Ornith-1.0-35B Qwen-AgentWorld β β β β β β βββββββββ Weight-Diff SVD βββββββββββββββ β β β β β ββββββββ΄βββββββ β β β LoRA r=32 β β Merged at Ξ±=0.3 β β β 422 tensorsβ β β βββββββββββββββ β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ ``` ### Step 1: SVD Weight-Diff Extraction We extract **Ornith's unique knowledge** by computing the weight difference between Ornith-1.0-35B and the shared Huihui-Qwen-AgentWorld base: ```python delta = W_ornith - W_base # What Ornith learned U, S, Vh = torch.linalg.svd(delta) # Decompose lora_A = diag(S[:32].sqrt()) @ Vh[:32, :] lora_B = U[:, :32] @ diag(S[:32].sqrt()) ``` **422 tensors** extracted across: - **MLP layers** (`gate_proj`, `up_proj`, `down_proj`) β knowledge execution - **Expert weights** (256 MoE experts) β specialized routing (3D tensor β flatten β SVD β reshape) - **Shared expert** β common computation - **Norms + lm_head** β output calibration - **Attention layers skipped** β R2 uses `linear_attn` vs Ornith's `self_attn` (incompatible architecture) ### Step 2: Multi-LoRA Fusion The extracted Ornith LoRA (r=32, Ξ±=64) is merged into Qwen35B-Agent-R2 at **scale Ξ±=0.3**: ```python merged = R2 * 0.7 + Ornith_LoRA * 0.3 ``` This preserves **70% of R2's original capabilities** (its 7 LoRAs) while adding **30% of Ornith's algorithm/reasoning strength**. ### Why Ξ±=0.3? | Scale | R2 Preserved | Ornith Added | Best For | |:-----:|:------------:|:------------:|:---------| | 0.3 | 70% | 30% | Balanced β general agent use | | 0.4 | 60% | 40% | Algorithm-heavy tasks | | 0.5+ | <50% | >50% |β οΈ May dilute tool-calling | --- ## π¬ Technique: SVD Weight-Diff for MoE MoE models (256 experts) require special handling for SVD extraction: | Component | Standard Approach | MoE Adaptation | |:----------|:-----------------:|:---------------| | **2D tensors** (MLP, norms) | `SVD(delta)` β normal | Same | | **3D expert tensors** `[out, in, 256]` | N/A | `flatten β SVD β reshape` | | **Attention mismatch** | Direct diff | β Skipped (R2 uses `linear_attn`) | | **language_model prefix** | Exact match | Strip prefix after loading | **Expert tensor handling:** ```python delta = W_a - W_b # [512, 2048, 256] delta_flat = delta.transpose(0,2).reshape(-1, delta.shape[1]) # [131072, 2048] U, S, Vh = torch.linalg.svd(delta_flat) lora_B = U[:, :32] @ diag(S[:32].sqrt()) # [131072, 32] # On merge: reconstruct delta = lora_B @ lora_A # [131072, 2048] delta = delta.reshape(512, 256, 2048).permute(0, 2, 1) # [512, 2048, 256] ``` --- ## π What You Get | Capability | Source | Retained | |:-----------|:------:|:--------:| | π§ **Reasoning** (Opus 4.8) | R2 | β 100% | | π§ **Tool Calling** | R2 | β 100% | | π§ **Agent Routing** | R2 | β 100% | | π **Math** | R2 + Ornith | β Enhanced | | β‘ **Algorithm** | **Ornith** π | β **+30%** | | π¬ **Conversation** (Fable) | R2 | β 100% | | π **Creative** (Mythos) | R2 | β 100% | --- ## π Why Agent-R2O3? | Aspect | Other Models | **Agent-R2O3** | |--------|-------------|:--------------:| | Tool Call Format | β Often malformed | β **Guaranteed valid `
