Update README.md
Browse files
README.md
CHANGED
|
@@ -2,10 +2,156 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
|
| 5 |
-
prototype 1;
|
|
|
|
| 6 |
|
| 7 |
-
|
| 8 |
|
| 9 |
-
|
| 10 |
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
|
| 5 |
+
# prototype 1;
|
| 6 |
+
full anchor bank, minimal anchors, positive outcome from mini distilled expert student
|
| 7 |
|
| 8 |
+
bert-base aligned.
|
| 9 |
|
| 10 |
+
Positive enough to continue refinement.
|
| 11 |
|
| 12 |
+
```
|
| 13 |
+
=================================================================
|
| 14 |
+
PHASE 3: GEOMETRIC VERIFICATION
|
| 15 |
+
=================================================================
|
| 16 |
+
Passthrough integrity: 1.000000 (should be ~1.000)
|
| 17 |
+
Geo context CV: 0.1651
|
| 18 |
+
Geo context eff_dim: 21.5
|
| 19 |
+
Geo context shape: torch.Size([2000, 64])
|
| 20 |
+
|
| 21 |
+
=================================================================
|
| 22 |
+
PHASE 4: CLASSIFIER STABILITY TEST
|
| 23 |
+
=================================================================
|
| 24 |
+
with_bank : train_acc=0.481 val_acc=0.390 gap=0.091
|
| 25 |
+
without_bank : train_acc=0.443 val_acc=0.330 gap=0.113
|
| 26 |
+
|
| 27 |
+
=================================================================
|
| 28 |
+
DONE
|
| 29 |
+
=================================================================
|
| 30 |
+
|
| 31 |
+
Student: mini_student.pt
|
| 32 |
+
Bank: alignment_bank.pt
|
| 33 |
+
Consensus CV: 0.1316
|
| 34 |
+
Student v_cos: 0.612
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
# prototype 2;
|
| 38 |
+
full anchor bank self aligned, 512 anchors, reduced training overfit, increased validation on distilled expert student
|
| 39 |
+
|
| 40 |
+
bert-base aligned.
|
| 41 |
+
|
| 42 |
+
```
|
| 43 |
+
=================================================================
|
| 44 |
+
PHASE 3: GEOMETRIC VERIFICATION
|
| 45 |
+
=================================================================
|
| 46 |
+
Passthrough: 1.000000 (target: 1.000)
|
| 47 |
+
Emb CV: 0.1592 (consensus: 0.1316)
|
| 48 |
+
Geo context CV: 0.1790
|
| 49 |
+
Geo eff_dim: 33.7 / 128
|
| 50 |
+
Expert cos: 0.794 ± 0.052
|
| 51 |
+
Anchor max cos: 0.656
|
| 52 |
+
Cross-expert: 0.032
|
| 53 |
+
|
| 54 |
+
=================================================================
|
| 55 |
+
PHASE 4: CLASSIFIER STABILITY TEST
|
| 56 |
+
=================================================================
|
| 57 |
+
with_bank : train=0.782 val=0.505 gap=0.277
|
| 58 |
+
without_bank : train=0.512 val=0.372 gap=0.140
|
| 59 |
+
|
| 60 |
+
=================================================================
|
| 61 |
+
SUMMARY
|
| 62 |
+
=================================================================
|
| 63 |
+
Consensus CV: 0.1316
|
| 64 |
+
Consensus eff_dim:223.7
|
| 65 |
+
Student v_cos: 0.612
|
| 66 |
+
Student v_cv: 0.169
|
| 67 |
+
Bank params: 2,918,784
|
| 68 |
+
Bank geo_eff_dim: 33.7
|
| 69 |
+
Bank geo_cv: 0.1790
|
| 70 |
+
|
| 71 |
+
=================================================================
|
| 72 |
+
DONE
|
| 73 |
+
=================================================================
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
# prototype 3;
|
| 77 |
+
full anchor bank, 512 anchors, full procrustes whitening, dynamic ortho, full geometric statistics, even more improved accuracy
|
| 78 |
+
|
| 79 |
+
dynamic alignment, additional utility potential present from dynamic capacity than directly aligning to an expert.
|
| 80 |
+
|
| 81 |
+
```
|
| 82 |
+
=================================================================
|
| 83 |
+
PHASE 3: GEOMETRIC VERIFICATION
|
| 84 |
+
=================================================================
|
| 85 |
+
Passthrough: 1.000000 (target: 1.000)
|
| 86 |
+
Emb CV: 0.1592 (consensus: 0.1316)
|
| 87 |
+
Geo context CV: 0.1817
|
| 88 |
+
Geo eff_dim: 33.5 / 128
|
| 89 |
+
Expert cos: 0.800 ± 0.004
|
| 90 |
+
Anchor max cos: 0.656
|
| 91 |
+
Cross-expert: -0.107
|
| 92 |
+
|
| 93 |
+
=================================================================
|
| 94 |
+
PHASE 4: CLASSIFIER STABILITY TEST
|
| 95 |
+
=================================================================
|
| 96 |
+
with_bank : train=0.773 val=0.513 gap=0.260
|
| 97 |
+
without_bank : train=0.512 val=0.372 gap=0.140
|
| 98 |
+
|
| 99 |
+
=================================================================
|
| 100 |
+
SUMMARY
|
| 101 |
+
=================================================================
|
| 102 |
+
Consensus CV: 0.1316
|
| 103 |
+
Consensus eff_dim:223.7
|
| 104 |
+
Student v_cos: 0.612
|
| 105 |
+
Student v_cv: 0.169
|
| 106 |
+
Bank params: 2,920,320
|
| 107 |
+
Bank geo_eff_dim: 33.5
|
| 108 |
+
Bank geo_cv: 0.1817
|
| 109 |
+
|
| 110 |
+
=================================================================
|
| 111 |
+
DONE
|
| 112 |
+
=================================================================
|
| 113 |
+
```
|
| 114 |
+
|
| 115 |
+
# prototype 4;
|
| 116 |
+
full anchor bank, 512 anchors, full procrestes whitening, dynamic ortho, improved and robust relative geometric statistics, even more improved accuracy
|
| 117 |
+
|
| 118 |
+
with full dynamic alignment the system is now cross-expert positive with a dynamic series of ratios for loss and alignment
|
| 119 |
+
|
| 120 |
+
stability still present, additional complexity can be included.
|
| 121 |
+
|
| 122 |
+
```
|
| 123 |
+
=================================================================
|
| 124 |
+
PHASE 3: GEOMETRIC VERIFICATION
|
| 125 |
+
=================================================================
|
| 126 |
+
Passthrough: 1.000000 (target: 1.000)
|
| 127 |
+
Emb CV: 0.1557 (consensus: 0.1316)
|
| 128 |
+
Geo context CV: 0.1872
|
| 129 |
+
Geo eff_dim: 33.2 / 128
|
| 130 |
+
Expert cos: 0.784 �� 0.005
|
| 131 |
+
Anchor max cos: 0.655
|
| 132 |
+
Disagreement:
|
| 133 |
+
Cross-expert: 0.0463 ± 0.0154
|
| 134 |
+
Ratio: 0.001912 (target: 0.000000)
|
| 135 |
+
Norm spread: 0.1279
|
| 136 |
+
|
| 137 |
+
=================================================================
|
| 138 |
+
PHASE 4: CLASSIFIER STABILITY TEST
|
| 139 |
+
=================================================================
|
| 140 |
+
with_bank : train=0.775 val=0.508 gap=0.266
|
| 141 |
+
without_bank : train=0.513 val=0.392 gap=0.122
|
| 142 |
+
|
| 143 |
+
=================================================================
|
| 144 |
+
SUMMARY
|
| 145 |
+
=================================================================
|
| 146 |
+
Consensus CV: 0.1316
|
| 147 |
+
Consensus eff_dim:223.7
|
| 148 |
+
Student v_cos: 0.612
|
| 149 |
+
Student v_cv: 0.169
|
| 150 |
+
Bank params: 2,921,088
|
| 151 |
+
Bank geo_eff_dim: 33.2
|
| 152 |
+
Bank geo_cv: 0.1872
|
| 153 |
+
|
| 154 |
+
=================================================================
|
| 155 |
+
DONE
|
| 156 |
+
=================================================================
|
| 157 |
+
```
|