Spaces:
Running
Running
Update README.md
Browse files
README.md
CHANGED
|
@@ -7,4 +7,70 @@ sdk: static
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# ๐ผ DaisyChainAI
|
| 11 |
+
|
| 12 |
+
**Modular minds, not monoliths.** We build capable systems by *daisy-chaining* a handful of
|
| 13 |
+
small, sharp specialists behind a learned router โ instead of training one giant model to do
|
| 14 |
+
everything. Each specialist is cheap, swappable, and crisp on its own domain; chained together,
|
| 15 |
+
they behave like one model at a fraction of the active compute.
|
| 16 |
+
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
## ๐ What "daisy-chaining" means
|
| 20 |
+
|
| 21 |
+
A **daisy chain** links independent units in series so a signal can flow from one to the next,
|
| 22 |
+
each unit handling what it's good at and passing the rest along. That's exactly how our systems work:
|
| 23 |
+
|
| 24 |
+
- **Each link is one small specialist** โ a dense ~74M model trained on a *single* domain. It is
|
| 25 |
+
excellent at its own data and (deliberately) surprised by everything else.
|
| 26 |
+
- **The router is the connector between links.** When an input arrives, it travels down the chain:
|
| 27 |
+
every specialist reports how *surprised* it is (bits/base) and exposes its hidden state, and a tiny
|
| 28 |
+
learned router hands the work to the link that's most at home with it.
|
| 29 |
+
- **The chain grows link by link.** Because the specialists are trained *separately*, you can chain a
|
| 30 |
+
new domain on without retraining the others โ add a link, extend the router, done. Remove or upgrade
|
| 31 |
+
a single link the same way.
|
| 32 |
+
- **One link runs per query.** Only the routed specialist computes, so a chain of four ~74M experts
|
| 33 |
+
costs ~74M of compute per token โ roughly **7ร cheaper** than a 500M monolith of comparable scope.
|
| 34 |
+
|
| 35 |
+
So "DaisyChain" is both the brand and the mechanism: **a chain of specialists, connected by routing,
|
| 36 |
+
that you extend one flower at a time.**
|
| 37 |
+
|
| 38 |
+
---
|
| 39 |
+
|
| 40 |
+
## ๐ ๏ธ How the models are built
|
| 41 |
+
|
| 42 |
+
Each specialist is grown by **interleaving two steps**, per domain:
|
| 43 |
+
|
| 44 |
+
1. **Continued pretraining** โ next-token training on *only* that domain's data, so the specialist
|
| 45 |
+
becomes genuinely crisp on its home distribution (and the router can tell the links apart).
|
| 46 |
+
2. **Per-domain distillation** โ the specialist is distilled from a larger teacher foundation model
|
| 47 |
+
*restricted to its own domain* (soft-target KD, plus a factorized per-nucleotide variant where the
|
| 48 |
+
teacher supports it). It learns the teacher's behavior on its slice without ever becoming a generic
|
| 49 |
+
clone โ the specialization is what makes routing work.
|
| 50 |
+
|
| 51 |
+
We iterate those two steps until each link is as strong as its capacity allows, then train the
|
| 52 |
+
**router**: a small head that reads every specialist's surprise plus a compressed view of its hidden
|
| 53 |
+
state and predicts the home domain โ recovering bias-corrections a plain "lowest-perplexity-wins" rule
|
| 54 |
+
misses.
|
| 55 |
+
|
| 56 |
+
This is, in lineage, a **cluster Branch-Train-Merge (cBTM) mixture of domain experts** โ independent
|
| 57 |
+
experts + perplexity-aware routing โ with iterative distillation from a larger teacher layered on top.
|
| 58 |
+
|
| 59 |
+
---
|
| 60 |
+
|
| 61 |
+
## ๐งฌ Current project โ DaisyChain Genomics
|
| 62 |
+
|
| 63 |
+
Four DNA/RNA specialists (**eukaryote ยท prokaryote ยท mRNA ยท mRNA-splice**, ~74M each, **โ295M total โ
|
| 64 |
+
under 500M**), each distilled per-domain from a 500M genomic foundation model, behind a learned router.
|
| 65 |
+
|
| 66 |
+
| | |
|
| 67 |
+
|---|---|
|
| 68 |
+
| **Routing accuracy** (held-out) | **94.8%** |
|
| 69 |
+
| **Active params / query** | ~74M (one specialist) |
|
| 70 |
+
| **vs the 500M teacher** | within ~6% likelihood; closing with training |
|
| 71 |
+
|
| 72 |
+
- ๐ฆ **Model:** [`DaisyChainAI/daisychain-genomics`](https://huggingface.co/DaisyChainAI/daisychain-genomics)
|
| 73 |
+
- ๐ฎ **Live demo:** [`Daisychain-Genomics-Demo`](https://huggingface.co/spaces/DaisyChainAI/Daisychain-Genomics-Demo) โ paste DNA, watch the chain light up specialist-by-specialist and route in real time.
|
| 74 |
+
|
| 75 |
+
More links on the chain โ and more chains โ coming. ๐ผ
|
| 76 |
+
|