SeaWolf-AI's picture
license: qwen3 νŒŒμƒμ€ apache-2.0 (google/gemma-4-* 및 Qwen3 λͺ¨λ‘ apache-2.0)
0da5d6e verified
|
Raw
History Blame Contribute Delete
2.58 kB
---
license: apache-2.0
base_model:
- Qwen/Qwen3-4B
- google/gemma-4-E4B
language:
- ko
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- darwin
- darwin-v9
- darwin-chimera
- ffn-crossbreed
- cross-architecture
- evolutionary-merge
- qwen3
- gemma4
- vidraft
- research-checkpoint
---
# Darwin-V9-Chimera-4B (Generation 2)
> **VIDRAFT attention + Qwen3-4B / Gemma4-E4B FFN crossbreed.**
> A **Qwen3-4B Γ— Gemma4-E4B hybrid β€” NOT from-scratch.** Private research checkpoint.
## Lineage (Darwin-Chimera 계보)
| Gen | Model | Composition |
|-----|-------|-------------|
| **Gen 1** | Darwin-Chimera-4B-Gen1 | Qwen3-4B attention-healing adapter (FFN = Qwen3-4B, frozen) |
| **Gen 2 (this)** | **Darwin-V9-Chimera-4B** | Gen1 adapter + **Gemma4-E4B FFN crossbreed** β†’ re-healing |
## What this is
The Gen-1 adapter's FFN is reconstructed by **cross-breeding Qwen3-4B FFN with
Gemma4-E4B FFN (ratio 0.15)**, then the attention is re-healed (VIDRAFT) to adapt
to the fused FFN. This carries the Gen-1 attention forward while blending a second
model's knowledge β€” so the result is **not reducible to any single parent**.
- **attention**: VIDRAFT healing (Qwen3-4B based)
- **FFN**: Qwen3-4B 85% βŠ• Gemma4-E4B 15% (bilinear inter projection 10240β†’9728, layer map 42β†’36)
- **structure**: 2560 / 9728 / 36L (Qwen3-4B coordinates)
- **re-healing**: 0.5B tokens, attention-only, LR 1e-5
## Evaluation (same harness, base zero-shot, KMMLU 3 subjects / 90Q)
| model | KMMLU | stage |
|-------|:---:|------|
| Qwen3-4B (original) | 13.3% | base |
| Gemma4-E4B (base) | 26.7% | base |
| Darwin-Chimera Gen1 | 27.1%* | base |
| fused raw (pre re-heal) | 22.2% | intermediate |
| **Darwin-V9 (this)** | **27.8%** | base |
\* Gen1 measured on 6 subjects. All numbers are **base zero-shot** β€” instruction-following
quality is expected from a later SFT stage (cf. Gemma4-E4B base 26.7% β†’ it 69.4%).
β†’ After blending 15% Gemma4 FFN, performance is **maintained / slightly above** the
Gen-1 baseline and Gemma4-E4B base. Gemma knowledge is visibly incorporated
(multilingual facts, "Germany is Berlin / Italy is ..."), and the intermediate English
degradation is recovered by re-healing.
## Known limitations
- Some Korean repetition remains in greedy single-shot generation β†’ to be resolved by SFT.
- Absolute scores are base-level; this is a **research backbone**, not a production/instruct model.
## License
**Gemma Terms of Use** (Gemma4-E4B weights are blended in) + Apache 2.0 (Qwen3-4B).
Built on **Qwen/Qwen3-4B** and **google/gemma-4-E4B**.