README.md

---
base_model: Qwen/Qwen-AgentWorld-35B-A3B
library_name: transformers
tags:
- lora
- merged
- qwen3_5_moe
- mlx
pipeline_tag: text-generation
---

# Qwen-AgentWorld-35B-A3B — v130 CPA merged

This is `Qwen/Qwen-AgentWorld-35B-A3B` with the **v130 CPA LoRA adapter** merged
into the weights (full bf16, ready to load with `transformers`).

## Provenance

- **Base model:** `Qwen/Qwen-AgentWorld-35B-A3B` (Qwen3.5-MoE, 35B total / A3B active, bf16)
- **Adapter:** `qwen-cpa-v130-pretrain-fix` — a rank-16 LoRA trained with MLX-LM
  (`scale=32`, `dropout=0.05`, 3300 iters, lr 1e-6), covering the top 12 decoder
  layers (28–39).

The adapter was trained in the MLX format and converted to PEFT before merging
(`adapter_config.json` uses `r=16, lora_alpha=512` so that `scaling = alpha/r = 32`
matches the MLX multiplier).

## Merged modules

Deltas were folded in at fp32 precision, then cast back to bf16:

| Group | Modules |
|-------|---------|
| Full-attention layers (31, 35, 39) | `self_attn.{q,k,v,o}_proj` |
| Linear-attention layers | `linear_attn.{in_proj_a,in_proj_b,in_proj_qkv,in_proj_z,out_proj}` |
| MoE routing / shared expert (all 12 layers) | `mlp.gate`, `mlp.shared_expert.{gate,up,down}_proj`, `mlp.shared_expert_gate` |
| Fused routed experts (all 12 layers) | `mlp.experts.gate_up_proj`, `mlp.experts.down_proj` |

For a plain nn.Linear the delta is `scale · (lora_a @ lora_b)ᵀ`; for the fused
per-expert MoE weights it is `scale · bmm(lora_b, lora_a)`, with the
`gate_proj`/`up_proj` halves written into the two row-blocks of the fused
`gate_up_proj`.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("junaidali/qwenadapters")
model = AutoModelForCausalLM.from_pretrained(
    "junaidali/qwenadapters", torch_dtype="bfloat16", device_map="auto",
    trust_remote_code=True)
```

> Policy note from the training manifest: *v130 CPA SFT candidate — benchmark before deploy.*