final: 2 anchors + dispatch + card

amoe/moe/README.md

@@ -0,0 +1,69 @@
+---
+license: mit
+tags: [amoe, adapter, mixture-of-experts, sentence-similarity, captionbert, aleph]
+base_model: AbstractPhil/captionbert-8192-v2
+library_name: amoe-lora
+---
+
+# captionbert-8192-v2 :: AMOE 2-anchor mixture
+
+Two [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchors on the
+**frozen** trunk, plus a trained dispatch over them. The trunk never moves.
+
+| anchor | trained on | relation |
+|---|---|---|
+| `equiv` | all-nli triplets | semantic equivalence |
+| `simplify` | simple-wiki + altlex + sentence-compression | simplification / compression |
+
+## Results
+
+| config | STS-B rho | SICK-R rho |
+|---|---|---|
+| bare trunk | .5747 | .6526 |
+| `equiv` alone | .7254 | **.7550** |
+| `simplify` alone | .7400 | .7075 |
+| **2-anchor dispatch** | **.7524** | .7380 |
+
+The two are complementary along the TASK axis -- `simplify` wins STS-B solo,
+`equiv` wins SICK-R -- which is the precondition a mixture needs. SICK-R is
+never trained on and is the honest transfer read.
+
+## Why the dispatch works here
+
+Dispatched amplitude is `(w_k/z) * sigmoid(gate_k) * consume_k(x)`, where
+`w_k/z = sinh(u_k) / SUM_j cosh(u_j)` over ALL anchors (the damping law).
+That gives ~1.0 only when one anchor engages and the other **abstains**
+(`u ~ 0`); if both fire it collapses to ~0.5 and the mixture delivers HALF of
+what either member does alone.
+
+Measured here: mean `|w/z|` moved from **.310/.380 (blend)** before alignment to
+**.645/.223 (specialize)** after 800 keys-only steps, with no starvation
+strikes. That flip is why the mixture beats its best member rather than damping
+itself below it.
+
+## Load
+
+```python
+import amoe
+h = amoe.attach(trunk, ["amoe/moe/equiv.anchor.pt", "amoe/moe/simplify.anchor.pt"],
+                dispatch="amoe/moe/captionbert-v2-moe.dispatch.pt",
+                binding=CaptionBertV2Binding(d=512))   # from modeling_captionbert.py
+base = h.detach()        # bit-exact or raises
+```
+
+All anchors disabled reproduces the bare trunk **bit-exact** (asserted at build
+time). Masking never renormalizes -- that is the damping law, not an oversight.
+
+Anchor `.pt` key layout is `{block}.{param}`. (`blocks.{site}.{param}` in the
+amoe README is the *safetensors* layout, a different serializer.)
+
+## Training
+
+Anchors: MNRL, in-batch + hard negatives where the source has them, 1,500 steps
+at batch 256, pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32/TF32 off.
+1,500 not 4,000: the first solo run's STS-B **peaked at step 1,000** and then
+fell .0178 while SICK-R kept climbing -- 56,825 distinct anchors behind 200,000
+draws (space/draws .284). Dispatch: 800 steps, routing keys only (1,536 params),
+anchors frozen, starvation safeguard armed.
+
+See `metrics.json` for the full table and the routing telemetry.

amoe/moe/captionbert-v2-moe.dispatch.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11979c0f396ac1f5df995c6de4a1df3eafad6382dc62879ac1ce1b21b741f1a7
+size 1587315

amoe/moe/config.json

@@ -0,0 +1,85 @@
+{
+  "run_name": "captionbert-v2-moe",
+  "trunk_repo": "AbstractPhil/captionbert-8192-v2",
+  "trunk_ckpt": "checkpoints/best_model.pt",
+  "tokenizer": "google-bert/bert-base-uncased",
+  "d_model": 512,
+  "n_heads": 8,
+  "n_layers": 12,
+  "d_ff": 2048,
+  "max_len": 8192,
+  "output_dim": 768,
+  "pooling": "mean",
+  "experts": [
+    [
+      "equiv",
+      [
+        [
+          "sentence-transformers/all-nli",
+          "triplet",
+          "anchor",
+          "positive",
+          "negative",
+          200000
+        ]
+      ]
+    ],
+    [
+      "simplify",
+      [
+        [
+          "sentence-transformers/simple-wiki",
+          "pair",
+          "text",
+          "simplified",
+          null,
+          0
+        ],
+        [
+          "sentence-transformers/altlex",
+          "pair",
+          "text",
+          "simplified",
+          null,
+          0
+        ],
+        [
+          "sentence-transformers/sentence-compression",
+          "pair",
+          "text",
+          "simplified",
+          null,
+          0
+        ]
+      ]
+    ]
+  ],
+  "dedup_jaccard": 0.95,
+  "n_slots": 16,
+  "K": 64,
+  "D": 4,
+  "tau": 0.1,
+  "hidden": 178,
+  "gate_init": -3.0,
+  "anchor_steps": 1500,
+  "anchor_lr": 0.001,
+  "batch_size": 256,
+  "temperature": 0.05,
+  "max_tokens": 64,
+  "align_steps": 800,
+  "align_lr": 0.001,
+  "align_emb": 64,
+  "align_tau": 0.1,
+  "check_every": 200,
+  "usage_ppl_floor": 1.5,
+  "usage_min": 0.02,
+  "max_strikes": 3,
+  "seed": 0,
+  "log_every": 100,
+  "eval_every": 500,
+  "out_dir": "/content/amoe_moe",
+  "hf_repo": "AbstractPhil/captionbert-8192-v2",
+  "hf_path": "amoe/moe",
+  "hf_private": false,
+  "hf_push": true
+}

amoe/moe/metrics.json

@@ -0,0 +1,162 @@
+{
+  "baseline": {
+    "STS-B": {
+      "spearman": 0.574721280584061,
+      "self_cos": 0.13962045311927795,
+      "erank": 36.611595622425114
+    },
+    "SICK-R": {
+      "spearman": 0.652602297708298,
+      "self_cos": 0.32337328791618347,
+      "erank": 39.12520123844512
+    }
+  },
+  "rows": {
+    "OFF": {
+      "STS-B": {
+        "spearman": 0.574721280584061,
+        "self_cos": 0.13962045311927795,
+        "erank": 36.611595622425114
+      },
+      "SICK-R": {
+        "spearman": 0.652602297708298,
+        "self_cos": 0.32337328791618347,
+        "erank": 39.12520123844512
+      }
+    },
+    "equiv-only": {
+      "STS-B": {
+        "spearman": 0.731078083351727,
+        "self_cos": 0.1255255937576294,
+        "erank": 50.663957003073875
+      },
+      "SICK-R": {
+        "spearman": 0.7325251878585886,
+        "self_cos": 0.17530100047588348,
+        "erank": 34.694745020767634
+      }
+    },
+    "simplify-only": {
+      "STS-B": {
+        "spearman": 0.6031180553522926,
+        "self_cos": 0.12380383908748627,
+        "erank": 43.472962330224135
+      },
+      "SICK-R": {
+        "spearman": 0.6609382896862828,
+        "self_cos": 0.3064461648464203,
+        "erank": 39.3687757862779
+      }
+    },
+    "MOE": {
+      "STS-B": {
+        "spearman": 0.7524264373550089,
+        "self_cos": 0.12201106548309326,
+        "erank": 54.394680890714454
+      },
+      "SICK-R": {
+        "spearman": 0.7380408425545968,
+        "self_cos": 0.16639426350593567,
+        "erank": 34.39546446727023
+      }
+    }
+  },
+  "telemetry_before": {
+    "equiv": 0.3097170293331146,
+    "simplify": 0.3800048828125
+  },
+  "telemetry_after": {
+    "equiv": 0.6454874873161316,
+    "simplify": 0.2226928472518921
+  },
+  "alarms": [],
+  "anchors": [
+    "equiv.anchor.pt",
+    "simplify.anchor.pt"
+  ],
+  "config": {
+    "run_name": "captionbert-v2-moe",
+    "trunk_repo": "AbstractPhil/captionbert-8192-v2",
+    "trunk_ckpt": "checkpoints/best_model.pt",
+    "tokenizer": "google-bert/bert-base-uncased",
+    "d_model": 512,
+    "n_heads": 8,
+    "n_layers": 12,
+    "d_ff": 2048,
+    "max_len": 8192,
+    "output_dim": 768,
+    "pooling": "mean",
+    "experts": [
+      [
+        "equiv",
+        [
+          [
+            "sentence-transformers/all-nli",
+            "triplet",
+            "anchor",
+            "positive",
+            "negative",
+            200000
+          ]
+        ]
+      ],
+      [
+        "simplify",
+        [
+          [
+            "sentence-transformers/simple-wiki",
+            "pair",
+            "text",
+            "simplified",
+            null,
+            0
+          ],
+          [
+            "sentence-transformers/altlex",
+            "pair",
+            "text",
+            "simplified",
+            null,
+            0
+          ],
+          [
+            "sentence-transformers/sentence-compression",
+            "pair",
+            "text",
+            "simplified",
+            null,
+            0
+          ]
+        ]
+      ]
+    ],
+    "dedup_jaccard": 0.95,
+    "n_slots": 16,
+    "K": 64,
+    "D": 4,
+    "tau": 0.1,
+    "hidden": 178,
+    "gate_init": -3.0,
+    "anchor_steps": 1500,
+    "anchor_lr": 0.001,
+    "batch_size": 256,
+    "temperature": 0.05,
+    "max_tokens": 64,
+    "align_steps": 800,
+    "align_lr": 0.001,
+    "align_emb": 64,
+    "align_tau": 0.1,
+    "check_every": 200,
+    "usage_ppl_floor": 1.5,
+    "usage_min": 0.02,
+    "max_strikes": 3,
+    "seed": 0,
+    "log_every": 100,
+    "eval_every": 500,
+    "out_dir": "/content/amoe_moe",
+    "hf_repo": "AbstractPhil/captionbert-8192-v2",
+    "hf_path": "amoe/moe",
+    "hf_private": false,
+    "hf_push": true
+  }
+}