| ================================================================= |
| RAPID PROTOTYPE v2: Differentiation-Centered Bank |
| ================================================================= |
| Device: cuda |
|
|
| ================================================================= |
| PHASE 0: EXTRACTION |
| ================================================================= |
| Captions: 20,000 |
|
|
| Extracting: bert... |
| Loading weights: 100% |
| 199/199 [00:00<00:00, 4038.86it/s, Materializing param=pooler.dense.weight] |
| BertModel LOAD REPORT from: google-bert/bert-base-uncased |
| Key | Status | | |
| -------------------------------------------+------------+--+- |
| cls.predictions.bias | UNEXPECTED | | |
| cls.predictions.transform.dense.bias | UNEXPECTED | | |
| cls.predictions.transform.LayerNorm.weight | UNEXPECTED | | |
| cls.predictions.transform.LayerNorm.bias | UNEXPECTED | | |
| cls.seq_relationship.weight | UNEXPECTED | | |
| cls.predictions.transform.dense.weight | UNEXPECTED | | |
| cls.seq_relationship.bias | UNEXPECTED | | |
|
|
| Notes: |
| - UNEXPECTED :can be ignored when loading from different task/architecture; not ok if you expect identical arch. |
| bert: 100%|██████████| 157/157 [00:23<00:00, 6.55it/s] |
| Shape: torch.Size([20000, 768]) |
|
|
| Extracting: modern... |
| Loading weights: 100% |
| 134/134 [00:00<00:00, 4016.84it/s, Materializing param=layers.21.mlp_norm.weight] |
| ModernBertModel LOAD REPORT from: answerdotai/ModernBERT-base |
| Key | Status | | |
| ------------------+------------+--+- |
| head.dense.weight | UNEXPECTED | | |
| head.norm.weight | UNEXPECTED | | |
| decoder.bias | UNEXPECTED | | |
|
|
| Notes: |
| - UNEXPECTED :can be ignored when loading from different task/architecture; not ok if you expect identical arch. |
| modern: 100%|██████████| 157/157 [00:35<00:00, 4.39it/s] |
| Shape: torch.Size([20000, 768]) |
|
|
| ================================================================= |
| PHASE 0b: GENERALIZED PROCRUSTES ALIGNMENT (no reference bias) |
| ================================================================= |
| GPA iter 1: delta=1.19668072 |
| GPA iter 3: delta=0.00029225 |
| GPA iter 6: delta=0.00006347 |
| GPA iter 9: delta=0.00002718 |
| bert : cos_after=0.8541 cos_to_mean=0.9865 |
| modern : cos_after=0.8577 cos_to_mean=0.9867 |
| cos(consensus, bert): 0.9867 |
| cos(consensus, modern): 0.9868 |
| Equidistance range: 0.0001 (should be near 0) |
|
|
| Measuring consensus statistics... |
| CV: 0.1771 |
| Mean cos: 0.0018 |
| Eff dim: 109.5 |
| Spectral: [0.0343, 0.0322, 0.0275, 0.0240, 0.0222...] |
|
|
| ================================================================= |
| PHASE 1: TRAIN STUDENT |
| ================================================================= |
| Student: 11,269,632 params |
| CV target: 0.1771 |
| E1: 2s loss=2.9588 t_acc=0.362 t_cos=0.334 v_acc=0.494 v_cos=0.503 v_cv=0.223 |
| E2: 2s loss=1.4268 t_acc=0.761 t_cos=0.543 v_acc=0.704 v_cos=0.588 v_cv=0.212 |
| E3: 2s loss=0.9784 t_acc=0.887 t_cos=0.604 v_acc=0.822 v_cos=0.639 v_cv=0.182 |
| E4: 2s loss=0.7289 t_acc=0.943 t_cos=0.641 v_acc=0.912 v_cos=0.676 v_cv=0.182 |
| E5: 2s loss=0.5807 t_acc=0.968 t_cos=0.666 v_acc=0.920 v_cos=0.686 v_cv=0.182 |
|
|
| Student saved. v_cos=0.686, v_cv=0.182 |
|
|
| ================================================================= |
| PHASE 2: TRAIN ALIGNMENT BANK (student frozen) |
| ================================================================= |
| Pre-encoding through frozen student... |
| Student embeddings: torch.Size([18000, 768]) |
| Expert 0 (bert): rotation + whitener + mean loaded, cos_after=0.8541 |
| Expert 1 (modern): rotation + whitener + mean loaded, cos_after=0.8577 |
| Anchors: 512 initialized from consensus embeddings |
| Targets: CV=0.1771, mean_cos=0.0018 |
| Bank: 2,921,088 params |
| Bank targets: CV=0.1771, mean_cos=0.0018 |
| Calibrated disagreement: |
| cross_cos: 0.0794 ± 0.0035 |
| disagree_ratio: 0.000000 |
|
|
| E 1: 1s loss=0.4775 v_loss=0.4350 |
| Geometry: b_cv=0.2673 e_cv=0.1680 spread=0.03938 a_max=0.653 |
| Experts: cos=0.791±0.006 agr=0.000142 ortho=0.000440 |
| Disagree: x_cos=0.0833±0.0019 ratio=0.004748 preserve=0.014587 norms=0.1814 |
|
|
| E 2: 1s loss=0.3981 v_loss=0.3833 |
| Geometry: b_cv=0.2230 e_cv=0.1651 spread=0.02783 a_max=0.669 |
| Experts: cos=0.809±0.006 agr=0.000008 ortho=0.000344 |
| Disagree: x_cos=0.0817±0.0018 ratio=0.003509 preserve=0.000024 norms=0.1719 |
|
|
| E 3: 1s loss=0.3730 v_loss=0.3757 |
| Geometry: b_cv=0.2162 e_cv=0.1648 spread=0.02493 a_max=0.670 |
| Experts: cos=0.830±0.005 agr=0.000004 ortho=0.000186 |
| Disagree: x_cos=0.0799±0.0019 ratio=0.002291 preserve=0.000013 norms=0.1513 |
|
|
| E 4: 1s loss=0.3623 v_loss=0.3708 |
| Geometry: b_cv=0.2187 e_cv=0.1615 spread=0.02314 a_max=0.670 |
| Experts: cos=0.832±0.005 agr=0.000003 ortho=0.000115 |
| Disagree: x_cos=0.0793±0.0020 ratio=0.003285 preserve=0.000011 norms=0.1422 |
|
|
| E 5: 1s loss=0.3554 v_loss=0.3539 |
| Geometry: b_cv=0.2141 e_cv=0.1621 spread=0.02139 a_max=0.669 |
| Experts: cos=0.853±0.004 agr=0.000002 ortho=0.000079 |
| Disagree: x_cos=0.0781±0.0021 ratio=0.001270 preserve=0.000011 norms=0.0980 |
|
|
| E 6: 1s loss=0.3507 v_loss=0.3571 |
| Geometry: b_cv=0.2124 e_cv=0.1633 spread=0.02019 a_max=0.669 |
| Experts: cos=0.829±0.005 agr=0.000001 ortho=0.000058 |
| Disagree: x_cos=0.0788±0.0022 ratio=0.001736 preserve=0.000010 norms=0.1789 |
|
|
| E 7: 1s loss=0.3460 v_loss=0.3465 |
| Geometry: b_cv=0.2059 e_cv=0.1607 spread=0.01903 a_max=0.669 |
| Experts: cos=0.845±0.005 agr=0.000001 ortho=0.000045 |
| Disagree: x_cos=0.0819±0.0023 ratio=0.001425 preserve=0.000008 norms=0.1536 |
|
|
| E 8: 1s loss=0.3449 v_loss=0.3421 |
| Geometry: b_cv=0.2060 e_cv=0.1592 spread=0.01841 a_max=0.670 |
| Experts: cos=0.833±0.005 agr=0.000003 ortho=0.000035 |
| Disagree: x_cos=0.0885±0.0021 ratio=0.001539 preserve=0.000017 norms=0.1313 |
|
|
| E 9: 1s loss=0.3422 v_loss=0.3451 |
| Geometry: b_cv=0.2040 e_cv=0.1626 spread=0.01793 a_max=0.669 |
| Experts: cos=0.822±0.005 agr=0.000003 ortho=0.000031 |
| Disagree: x_cos=0.0761±0.0024 ratio=0.001610 preserve=0.000037 norms=0.2032 |
|
|
| E10: 1s loss=0.3416 v_loss=0.3497 |
| Geometry: b_cv=0.2077 e_cv=0.1647 spread=0.01735 a_max=0.669 |
| Experts: cos=0.782±0.007 agr=0.000003 ortho=0.000029 |
| Disagree: x_cos=0.0825±0.0023 ratio=0.004691 preserve=0.000025 norms=0.2039 |
|
|
| E11: 1s loss=0.3387 v_loss=0.3507 |
| Geometry: b_cv=0.2019 e_cv=0.1640 spread=0.01701 a_max=0.668 |
| Experts: cos=0.811±0.005 agr=0.000002 ortho=0.000024 |
| Disagree: x_cos=0.0780±0.0023 ratio=0.000957 preserve=0.000015 norms=0.1889 |
|
|
| E12: 1s loss=0.3391 v_loss=0.3381 |
| Geometry: b_cv=0.2006 e_cv=0.1588 spread=0.01675 a_max=0.668 |
| Experts: cos=0.778±0.006 agr=0.000003 ortho=0.000021 |
| Disagree: x_cos=0.0729±0.0021 ratio=0.001148 preserve=0.000024 norms=0.1404 |
|
|
| E13: 1s loss=0.3373 v_loss=0.3434 |
| Geometry: b_cv=0.1987 e_cv=0.1635 spread=0.01671 a_max=0.668 |
| Experts: cos=0.703±0.007 agr=0.000013 ortho=0.000021 |
| Disagree: x_cos=0.0680±0.0026 ratio=0.003978 preserve=0.000085 norms=0.2265 |
|
|
| E14: 1s loss=0.3383 v_loss=0.3351 |
| Geometry: b_cv=0.2027 e_cv=0.1658 spread=0.01634 a_max=0.668 |
| Experts: cos=0.779±0.005 agr=0.000007 ortho=0.000024 |
| Disagree: x_cos=0.0849±0.0022 ratio=0.002337 preserve=0.000085 norms=0.1472 |
|
|
| E15: 1s loss=0.3366 v_loss=0.3357 |
| Geometry: b_cv=0.1999 e_cv=0.1612 spread=0.01584 a_max=0.668 |
| Experts: cos=0.671±0.008 agr=0.000008 ortho=0.000023 |
| Disagree: x_cos=0.0777±0.0024 ratio=0.011179 preserve=0.000061 norms=0.1758 |
|
|
| E16: 1s loss=0.3363 v_loss=0.3467 |
| Geometry: b_cv=0.1983 e_cv=0.1612 spread=0.01575 a_max=0.668 |
| Experts: cos=0.737±0.005 agr=0.000010 ortho=0.000022 |
| Disagree: x_cos=0.0839±0.0022 ratio=0.006047 preserve=0.000049 norms=0.1216 |
|
|
| E17: 1s loss=0.3343 v_loss=0.3376 |
| Geometry: b_cv=0.1974 e_cv=0.1655 spread=0.01591 a_max=0.668 |
| Experts: cos=0.718±0.005 agr=0.000002 ortho=0.000020 |
| Disagree: x_cos=0.0723±0.0023 ratio=0.002539 preserve=0.000042 norms=0.0947 |
|
|
| E18: 1s loss=0.3354 v_loss=0.3457 |
| Geometry: b_cv=0.1955 e_cv=0.1580 spread=0.01588 a_max=0.668 |
| Experts: cos=0.763±0.005 agr=0.000007 ortho=0.000019 |
| Disagree: x_cos=0.0796±0.0022 ratio=0.004057 preserve=0.000069 norms=0.1001 |
|
|
| E19: 1s loss=0.3344 v_loss=0.3313 |
| Geometry: b_cv=0.1962 e_cv=0.1602 spread=0.01560 a_max=0.668 |
| Experts: cos=0.687±0.005 agr=0.000005 ortho=0.000018 |
| Disagree: x_cos=0.0862±0.0024 ratio=0.005997 preserve=0.000030 norms=0.1218 |
|
|
| E20: 1s loss=0.3331 v_loss=0.3651 |
| Geometry: b_cv=0.1950 e_cv=0.1631 spread=0.01556 a_max=0.668 |
| Experts: cos=0.729±0.005 agr=0.000007 ortho=0.000018 |
| Disagree: x_cos=0.0826±0.0021 ratio=0.006963 preserve=0.000065 norms=0.0781 |
|
|
| ================================================================= |
| PHASE 3: GEOMETRIC VERIFICATION |
| ================================================================= |
| Passthrough: 1.000000 (target: 1.000) |
| Emb CV: 0.1660 (consensus: 0.1771) |
| Geo context CV: 0.2053 |
| Geo eff_dim: 30.5 / 128 |
| Expert cos: 0.729 ± 0.005 |
| Anchor max cos: 0.668 |
| Disagreement: |
| Cross-expert: 0.0826 ± 0.0021 |
| Ratio: 0.006963 (target: 0.000000) |
| Norm spread: 0.0781 |
|
|
| ================================================================= |
| PHASE 4: CLASSIFIER STABILITY TEST |
| ================================================================= |
| with_bank : train=0.746 val=0.500 gap=0.246 |
| without_bank : train=0.490 val=0.363 gap=0.126 |
|
|
| ================================================================= |
| SUMMARY |
| ================================================================= |
| Consensus CV: 0.1771 |
| Consensus eff_dim:109.5 |
| Student v_cos: 0.686 |
| Student v_cv: 0.182 |
| Bank params: 2,921,088 |
| Bank geo_eff_dim: 30.5 |
| Bank geo_cv: 0.2053 |
|
|
| ================================================================= |
| DONE |
| ================================================================= |
