File size: 2,729 Bytes
705321b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
license: apache-2.0
base_model:
- Qwen/Qwen3-4B
language:
- ko
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- darwin
- darwin-chimera
- attention-healing
- sliding-window
- qwen3
- vidraft
- research-checkpoint
---
# Darwin-Chimera-4B-Gen1 (Backbone · Research)
> ⚠️ **Generation-1 backbone — a research checkpoint, not a product.** Private repo.
> This is **a Qwen3-4B derivative**, not a from-scratch model. We state this explicitly.
## What this is
`Darwin-Chimera-4B-Gen1` is the first-generation **adapter backbone** of the Darwin-Chimera
line. We take **Qwen/Qwen3-4B** and re-wire only its **attention** via VIDRAFT
attention-healing, while **freezing the FFN, embeddings, and lm_head**, and convert the
attention to a **sliding-window** configuration.
The purpose is to verify that a VIDRAFT-healed attention circuit can sit on a frozen
knowledge core — the foundation for Generation-2 (FFN cross-breeding with other models).
## Honest weight fingerprint (vs Qwen/Qwen3-4B)
Measured relative change `||A−B|| / ||A||` against the original Qwen3-4B:
| Component | Relative change | Note |
|-----------|:---:|------|
| FFN (mlp) | **0.000%** | frozen — identical to Qwen3-4B |
| embed / lm_head | **0.000%** | frozen — identical |
| attention (self_attn) | **3.0%** mean (7.5% max) | healed |
| layernorm | 0.04% | minimal |
| config (hidden/inter/layers/vocab) | identical | only `sliding_window=4096` added |
→ At the weight level this checkpoint is **clearly a Qwen3-4B derivative**. We make **no**
claim of independence or from-scratch training. Knowledge/FFN is 100% Qwen3-4B.
## Training
- **Method**: attention-only healing (self_attn + per-layer norms trainable; FFN/embed/lm_head frozen)
- **Attention**: full → sliding window (4096), 5:1 sliding:full layer ratio
- **Tokens**: ~3B (Korean-centric annealing mix)
- **Base**: Qwen/Qwen3-4B (Apache 2.0)
## Evaluation (base, zero-shot — reference only)
- **Generation**: 6/6 domains coherent (Korean / English / science / code / math / biology), no gibberish
- **KMMLU** (6 subjects, 240Q, zero-shot, greedy): **27.1%** vs Qwen3-4B base **13.3%** (same protocol, +13.8pp)
- Absolute KMMLU is low because this is a **base (non-instruct) checkpoint**; instruction-following
and benchmark quality are expected to come from a later SFT stage. The comparison above is a
same-condition relative measurement, not an absolute SOTA claim.
## Intended use
- Backbone for **Darwin-Chimera Generation-2** (cross-architecture FFN cross-breeding research)
- Research and experimentation only. **Not for production.**
## License & attribution
Apache 2.0, inherited from **Qwen/Qwen3-4B**. Built on Qwen/Qwen3-4B.
|