File size: 3,927 Bytes
d828115 4cb9b61 3ac2c92 d828115 7daa6ee 3ac2c92 d828115 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | ## AlephLM-0 benchmark
| model | params | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|---|---|---|---|---|---|---|---|---|---|---|
| bert-base | 109.5M | 0.4729 | 0.5865 | 0.3087 | 0.5988 | 0.4773 | 0.6029 | 0.6373 | 0.5467 | **0.5289** |
| ModernBERT-base | 149.0M | 0.4215 | 0.5479 | 0.3527 | 0.4247 | 0.3795 | 0.5349 | 0.4174 | 0.5630 | **0.4552** |
| roberta-base | 124.6M | 0.5436 | 0.6296 | 0.3211 | 0.5631 | 0.4522 | 0.6134 | 0.6198 | 0.5777 | **0.5400** |
| albert-base-v2 | 11.7M | 0.4784 | 0.5364 | 0.3101 | 0.4831 | 0.3809 | 0.5542 | 0.5491 | 0.4863 | **0.4723** |
| distilbert | 66.4M | 0.5717 | 0.6424 | 0.4344 | 0.6490 | 0.5410 | 0.6663 | 0.6854 | 0.5162 | **0.5883** |
| all-MiniLM-L6-v2 | 22.7M | 0.8203 | 0.7758 | 0.7237 | 0.8058 | 0.7559 | 0.8539 | 0.7899 | 0.8144 | **0.7925** |
| captionbert-v2 | 58.3M | 0.5747 | 0.6526 | 0.5051 | 0.5995 | 0.5452 | 0.7136 | 0.6776 | 0.5933 | **0.6077** |
| captionbert-v2 + arms | 63.2M | 0.7684 | 0.7391 | 0.6682 | 0.7557 | 0.6921 | 0.8055 | 0.7626 | 0.6382 | **0.7287** |
| captionbert-b | 58.3M | 0.5752 | 0.6548 | 0.5012 | 0.6037 | 0.5470 | 0.7146 | 0.6782 | 0.5500 | **0.6031** |
| captionbert-b + arms | 63.2M | 0.7675 | 0.7374 | 0.6705 | 0.7381 | 0.6945 | 0.8109 | 0.7695 | 0.6472 | **0.7294** |
| alephlm0-a1_anchored-s0 [cls] | 58.3M | 0.5731 | 0.6528 | 0.4996 | 0.6014 | 0.5457 | 0.7121 | 0.6761 | 0.5639 | **0.6031** |
| alephlm0-a1_anchored-s0 OFF [cls] | 58.3M | 0.5809 | 0.6226 | 0.5200 | 0.5720 | 0.5554 | 0.6773 | 0.6232 | 0.4434 | **0.5743** |
| alephlm0-a1_anchored-s1 [cls] | 58.3M | 0.5681 | 0.6512 | 0.4931 | 0.5996 | 0.5423 | 0.7094 | 0.6745 | 0.5673 | **0.6007** |
| alephlm0-a1_anchored-s1 OFF [cls] | 58.3M | 0.5630 | 0.6009 | 0.4968 | 0.5353 | 0.5411 | 0.6401 | 0.5895 | 0.5330 | **0.5625** |
| alephlm0-a2_dense-s0 [cls] | 58.3M | 0.5731 | 0.6507 | 0.4965 | 0.5962 | 0.5430 | 0.7130 | 0.6798 | 0.5683 | **0.6026** |
| alephlm0-a3_random-s0 [cls] | 58.3M | 0.5707 | 0.6538 | 0.4966 | 0.5998 | 0.5448 | 0.7103 | 0.6806 | 0.5703 | **0.6033** |
| alephlm0-a3_random-s0 OFF [cls] | 58.3M | 0.5767 | 0.6309 | 0.4948 | 0.5735 | 0.5486 | 0.6851 | 0.6307 | 0.4776 | **0.5772** |
| alephlm0-small-a1_anchored-s0 [cls] | 58.3M | 0.3026 | 0.5240 | 0.3016 | 0.3204 | 0.3274 | 0.4809 | 0.3686 | 0.4412 | **0.3833** |
| alephlm0-small-a1_anchored-s0 OFF [cls] | 58.3M | 0.2990 | 0.5232 | 0.3007 | 0.3202 | 0.3301 | 0.4736 | 0.3632 | 0.4193 | **0.3787** |
| alephlm0-small-a2_dense-s0 [cls] | 58.3M | 0.2789 | 0.5032 | 0.2834 | 0.2890 | 0.3014 | 0.4040 | 0.3424 | 0.4217 | **0.3530** |
| alephlm0-small-a4_splat-cbk-s0 [cls] | 58.3M | 0.0893 | 0.1972 | -0.1205 | 0.1201 | 0.0964 | 0.1617 | 0.1154 | 0.1085 | **0.0960** |
| alephlm0-small-a4_splat-cbk-s0 OFF [cls] | 58.3M | 0.0702 | 0.1828 | -0.1743 | 0.1203 | 0.0935 | 0.1404 | 0.1133 | 0.1028 | **0.0811** |
| alephlm0-small-a4_splat-gf-s0 [cls] | 58.3M | 0.1009 | 0.1712 | 0.1170 | 0.1295 | 0.0680 | 0.1788 | 0.1784 | -0.1064 | **0.1047** |
| alephlm0-small-a4_splat-gf-s0 OFF [cls] | 58.3M | 0.0969 | 0.1699 | 0.1211 | 0.1337 | 0.0742 | 0.2023 | 0.1743 | -0.0830 | **0.1112** |
| alephlm0-small-a4_splat-rot-s0 [cls] | 58.3M | 0.0505 | 0.1066 | -0.1094 | 0.0984 | 0.0610 | 0.1286 | 0.1682 | -0.0411 | **0.0578** |
| alephlm0-small-a4_splat-rot-s0 OFF [cls] | 58.3M | 0.0351 | 0.1122 | -0.1098 | 0.0907 | 0.0684 | 0.1132 | 0.1637 | -0.0479 | **0.0532** |
| alephlm0-small-a4_splat-s0 [cls] | 58.3M | 0.1179 | 0.2047 | 0.1563 | 0.0109 | 0.0336 | 0.1597 | 0.1832 | 0.1247 | **0.1239** |
| alephlm0-small-a4_splat-s0 OFF [cls] | 58.3M | 0.1067 | 0.1987 | 0.1242 | 0.0350 | 0.0339 | 0.1585 | 0.1799 | 0.1417 | **0.1223** |
AlephLM-0 rows are evaluated with each run's OWN trained pooling (labeled); raw-teacher context rows are mean-pooled per the certified harness. 'OFF' rows = anchored trunks with dispatch disabled (the C6 toggle-law read).
|