Update README.md

README.md

@@ -9,6 +9,7 @@ tags:
 - caption-embedding
 - sentence-similarity
 - feature-extraction
+- caption_encoder
 language: en
 pipeline_tag: feature-extraction
 datasets:
@@ -17,121 +18,164 @@ base_model:
 - AbstractPhil/geolip-bertenstein
 ---
 
-# GEOLIP Consensus-Distilled Caption Encoder
+# GEOLIP CaptionBERT-8192
 
-**A standalone 23M-parameter caption encoder trained via geometric consensus distillation from 5 BERT-family models.**
+A 26M-parameter caption encoder whose embedding space is the geometric intersection of five independently trained language models. Trained from scratch via consensus distillation — no pretrained weights, no expert models at inference.
 
-No expert models needed at inference. Just a tokenizer and this model.
+## Benchmarks
 
-## What Is This?
+Evaluated against all five consensus teachers on STS-B, SICK-R, and MRPC. All models use mean-pooled embeddings with cosine similarity. No fine-tuning on any benchmark task.
 
-Five independently trained language models — BERT-base, ModernBERT-base, RoBERTa-base, ALBERT-base-v2, and DistilBERT-base — were aligned into a shared geometric space via whitened Procrustes rotation. Their normalized centroid (the **geometric consensus**) was proven to be a mathematical constant: five different random seeds produced the same consensus point to three decimal places.
+### Semantic Textual Similarity (STS-B)
 
-This model was trained from scratch to reproduce that consensus directly from text. It distills the geometric intersection of five experts — the subspace where all five agree — into a single small transformer.
+| Model | Params | Spearman ρ | Pearson r |
+|---|---|---|---|
+| DistilBERT-base | 66M | 0.5717 | — |
+| RoBERTa-base | 125M | 0.5436 | — |
+| **CaptionBERT-8192** | **26M** | **0.5032** | **0.5100** |
+| ALBERT-base-v2 | 12M | 0.4784 | — |
+| BERT-base | 110M | 0.4729 | — |
+| ModernBERT-base | 149M | 0.4215 | — |
 
-The form of distillation is an experimental multi-teacher form of aligned distillation through anchoring geometric cooperation rather than adversarial conflict.
-The math aligns the data and the model no longer needs custom architecture now that the math aligns the structure so robustly and the losses maintain
-that analysis to such an extent.
+Beats BERT-base (4.2× larger) and ModernBERT-base (5.7× larger) on general sentence similarity despite being trained exclusively on image captions.
 
-The system works now. The geolip patchwork isn't the patchwork itself, it's the topological differentiation between what is and what is needed, and the most
-robust utility of that difference is based on analytical differentiation through direct distillation rather than utility through architectural grinding mechanisms.
+### SICK-R (Compositional Similarity)
 
-With this experimentation result, it shows the alignment itself can be directly distilled. The upcoming heads training will show that not only can this work,
-but the actual differentation can be applied DIRECTLY to downstream tasks such as head training, analysis through procrustes with additional experts,
-decoder processing through multiple INDEPENDENT experts from what the encoder was trained on, and entirely typeless functional transfer from one model
-to another.
+| Model | Params | Spearman ρ | Pearson r |
+|---|---|---|---|
+| DistilBERT-base | 66M | 0.6424 | — |
+| RoBERTa-base | 125M | 0.6296 | — |
+| **CaptionBERT-8192** | **26M** | **0.6138** | **0.6645** |
+| BERT-base | 110M | 0.5865 | — |
+| ModernBERT-base | 149M | 0.5479 | — |
+| ALBERT-base-v2 | 12M | 0.5364 | — |
 
-This unlocks the entire spectrum of geometric function, and the processing depends entirely on fp32 or higher structural integrity, or else the system collapses.
+\#3/6 on compositional/syntactic similarity. Beats BERT-base, ModernBERT-base, and ALBERT on a task requiring structural language understanding.
 
-## Results
+### MRPC (Paraphrase Detection)
+
+| Model | Params | F1 | Accuracy | Threshold |
+|---|---|---|---|---|
+| RoBERTa-base | 125M | 0.8122 | — | — |
+| **CaptionBERT-8192** | **26M** | **0.8068** | **0.6881** | **0.71** |
+| ALBERT-base-v2 | 12M | 0.8067 | — | — |
+| BERT-base | 110M | 0.8062 | — | — |
+| DistilBERT-base | 66M | 0.8055 | — | — |
+| ModernBERT-base | 149M | 0.8038 | — | — |
+
+**\#2/6 on paraphrase detection.** 0.005 F1 behind RoBERTa, ahead of every other teacher. No classification head — pure cosine similarity with auto-discovered threshold. A model that has never seen a paraphrase pair during training nearly wins paraphrase detection.
+
+### Caption Embedding Quality
+
+| Metric | Value |
+|---|---|
+| Self-similarity mean | 0.0040 |
+| Self-similarity max | 0.7181 |
+| Top-1 retrieval cosine | 0.5477 |
+| Top-5 retrieval cosine | 0.4853 |
+
+Near-zero average self-similarity across 1000 random captions — the embedding space has excellent discrimination. Every caption occupies its own distinct region on the hypersphere.
+
+### Consensus Fidelity
 
 | Metric | Value |
 |---|---|
-| **Val cosine to consensus** | **0.8621352314949036** |
-| **Val R@1** | **1.0** |
-| **Val CV** | **0.0817226113729792** |
-| Training data | CC12M captions (500000 samples) |
+| Val cosine to consensus | 0.862 |
+| Val R@1 | 1.000 |
+| Pentachoron CV | 0.082 |
+| Training data | 500K CC12M captions |
 | Epochs | 30 |
-| Warm-started | True |
-| Parameters | ~23M |
 | Position capacity | 8,192 tokens |
+| Parameters | 25,958,016 |
 
-### STS-B Comparison (mean-pooled, no fine-tuning)
-
-| Model | Params | STS-B Spearman |
-|---|---|---|
-| DistilBERT-base | 66M | 0.5717 |
-| RoBERTa-base | 125M | 0.5436 |
-| **Consensus Student** | **23M** | **0.4814** |
-| ALBERT-base-v2 | 12M | 0.4784 |
-| BERT-base | 110M | 0.4729 |
-| ModernBERT-base | 149M | 0.4215 |
-
-The student beats BERT-base (5x larger) and ModernBERT-base (7x larger) on STS-B despite being trained from scratch on image captions — out of domain for sentence similarity.
-
-### Training Curve
-
-| Epoch | t_acc | t_cos | v_acc | v_cos | v_cv | Time |
-|---|---|---|---|---|---|---|
-| 1 | 1.000 | 0.804 | 1.000 | 0.803 | 0.104 | 689s |
-| 2 | 1.000 | 0.807 | 1.000 | 0.810 | 0.085 | 688s |
-| 3 | 1.000 | 0.811 | 1.000 | 0.820 | 0.103 | 688s |
-| 4 | 1.000 | 0.815 | 1.000 | 0.825 | 0.084 | 689s |
-| 5 | 1.000 | 0.819 | 1.000 | 0.819 | 0.086 | 689s |
-| 6 | 1.000 | 0.821 | 1.000 | 0.821 | 0.095 | 689s |
-| 7 | 1.000 | 0.824 | 1.000 | 0.820 | 0.091 | 688s |
-| 8 | 1.000 | 0.827 | 1.000 | 0.834 | 0.088 | 689s |
-| 9 | 1.000 | 0.829 | 1.000 | 0.829 | 0.088 | 688s |
-| 10 | 1.000 | 0.831 | 1.000 | 0.829 | 0.087 | 689s |
-| 11 | 1.000 | 0.833 | 1.000 | 0.836 | 0.082 | 689s |
-| 12 | 1.000 | 0.835 | 1.000 | 0.838 | 0.084 | 689s |
-| 13 | 1.000 | 0.837 | 1.000 | 0.842 | 0.083 | 688s |
-| 14 | 1.000 | 0.839 | 1.000 | 0.842 | 0.081 | 689s |
-| 15 | 1.000 | 0.842 | 1.000 | 0.840 | 0.078 | 688s |
-| 16 | 1.000 | 0.843 | 1.000 | 0.843 | 0.086 | 689s |
-| 17 | 1.000 | 0.846 | 1.000 | 0.845 | 0.086 | 689s |
-| 18 | 1.000 | 0.847 | 1.000 | 0.848 | 0.087 | 689s |
-| 19 | 1.000 | 0.849 | 1.000 | 0.849 | 0.082 | 688s |
-| 20 | 1.000 | 0.851 | 1.000 | 0.849 | 0.078 | 690s |
-| 21 | 1.000 | 0.853 | 1.000 | 0.855 | 0.087 | 689s |
-| 22 | 1.000 | 0.855 | 1.000 | 0.856 | 0.083 | 689s |
-| 23 | 1.000 | 0.857 | 1.000 | 0.855 | 0.078 | 689s |
-| 24 | 1.000 | 0.858 | 1.000 | 0.857 | 0.093 | 688s |
-| 25 | 1.000 | 0.860 | 1.000 | 0.859 | 0.092 | 689s |
-| 26 | 1.000 | 0.861 | 1.000 | 0.860 | 0.079 | 689s |
-| 27 | 1.000 | 0.863 | 1.000 | 0.862 | 0.084 | 689s |
-| 28 | 1.000 | 0.863 | 1.000 | 0.862 | 0.091 | 688s |
-| 29 | 1.000 | 0.863 | 1.000 | 0.862 | 0.081 | 688s |
-| 30 | 1.000 | 0.863 | 1.000 | 0.862 | 0.082 | 689s |
+## How It Works
 
+Five language models were aligned into a shared geometric space via whitened Procrustes rotation. Their normalized centroid — the **geometric consensus** — was proven to be a mathematical constant: five different random seeds produced the same consensus point to three decimal places.
 
-## Usage
+This model was trained from scratch to reproduce that consensus directly from text. It distills the geometric intersection of five experts into a single small transformer.
 
-```python
-import torch
-from transformers import AutoTokenizer
-from caption_encoder import CaptionEncoder
+The distillation is not standard knowledge distillation. It is multi-teacher geometric consensus distillation: the target is not any single teacher's output but the fixed point where all five teachers agree. Individual model errors cancel. What remains is the structural invariant of language understanding that five different architectures and training objectives independently discovered.
 
-# Load
-tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
-model = CaptionEncoder(
-    vocab_size=30522, max_len=8192, d_model=384,
-    n_heads=6, n_layers=6, d_ff=1536, output_dim=768,
-    dropout=0.0, pad_token_id=0)
-model.load_state_dict(torch.load("best_model.pt", weights_only=True))
-model.eval()
+The alignment itself is directly distillable. The geometric structure is so robust that a from-scratch model learns it with R@1=1.000 from 18K examples in 80 seconds. The consensus manifold has pentachoron CV=0.084 — the tightest geometric regularity measured across all GEOLIP experiments — which means the function from text to embedding is smooth enough that sparse sampling covers it completely.
 
-# Encode
-texts = ["A cat sitting on a windowsill", "A dog playing fetch on the beach"]
-tokens = tokenizer(texts, max_length=512, padding="max_length",
-                   truncation=True, return_tensors="pt")
-with torch.no_grad():
-    embeddings = model(tokens["input_ids"], tokens["attention_mask"])
-
-# embeddings: (2, 768) L2-normalized
-similarity = embeddings[0] @ embeddings[1]
-print(f"Similarity: {similarity:.3f}")
 ```
+5 Expert Models (frozen)
+    │
+    ├── BERT-base-uncased        (110M, MLM)
+    ├── ModernBERT-base          (149M, MLM + rotary, 8192 ctx)
+    ├── RoBERTa-base             (125M, MLM + dynamic masking)
+    ├── ALBERT-base-v2           (12M, MLM + SOP + factorized)
+    └── DistilBERT-base          (66M, distilled from BERT)
+        │
+        ├── Extract pooled embeddings on 500K CC12M captions
+        ├── Whitened Procrustes alignment to shared space
+        ├── Consensus = normalized centroid (geometric constant)
+        │
+        └── Train student with:
+            ├── InfoNCE(student, consensus)   — retrieval alignment
+            ├── MSE(student, consensus)       — direct regression
+            └── Pentachoron CV → 0.084        — geometric regularity
+```
+
+## Planned Task Heads
+
+The 768-dim consensus embedding serves as a frozen feature extractor. Linear heads trained on task-specific data snap on top.
+
+### Priority Heads
+
+| Head | Architecture | Training Data | Use Case |
+|---|---|---|---|
+| **NLI / Entailment** | cat(a, b, \|a-b\|, a*b) → Linear(3072, 3) | MNLI, SNLI | Agent reasoning validation |
+| **Semantic Similarity** | Linear(768, 1) → sigmoid×5 | STS-B train | Push STS-B toward 0.80+ |
+| **Multi-Label Tagging** | Linear(768, n_tags) → sigmoid | COCO categories, Visual Genome | Predict objects/attributes from captions |
+| **Paraphrase Detection** | cos(a, b) → threshold (already works) | MRPC, QQP | Deduplication, reformulation detection |
+| **Sentiment** | Linear(768, n_classes) | SST-2, IMDB | Content routing, sentiment analysis |
+
+### Extended Heads
+
+| Head | Architecture | Training Data | Use Case |
+|---|---|---|---|
+| Caption Quality | Linear(768, 2) | Hallucination-annotated captions | Filter AI-generated training data |
+| Cross-Encoder Reranker | cat(query, doc) → Linear(1536, 1) | MS MARCO | Two-stage retrieval scoring |
+| Clustering | Linear(768, 256) → normalize | Unsupervised | Caption taxonomy, dataset organization |
+| Relation Extraction | cat(subj_emb, obj_emb) → Linear(1536, n_rel) | Visual Genome relationships | Structured scene understanding |
+| Caption-Image Score | Linear(768, 256) → cos with CLIP visual | CC12M image-caption pairs | Cross-modal retrieval without CLIP |
+
+### Consensus Head Distillation
+
+The same consensus trick applies to task heads. Train five separate NLI heads on the five frozen expert models, take the consensus prediction, distill into a single head on CaptionBERT. The head learns where all five experts agree on entailment — same noise cancellation, one layer instead of five.
+
+## Training Datasets — Current and Planned
+
+### Current
+
+| Dataset | Samples Used | Content | Notes |
+|---|---|---|---|
+| [CC12M LLaVA-Next](https://huggingface.co/datasets/CaptionEmporium/conceptual-captions-cc12m-llavanext) | 500K | Re-captioned CC12M with LLaVA-Next | Primary training data, mean ~92 tokens |
+
+### Planned — Caption Saturation
+
+The model tokenizes to 512 but has 8,192 position capacity. Longer, more complex captions will exercise the full context window and push v_cos beyond 0.862.
+
+| Dataset | Size | Content | Why |
+|---|---|---|---|
+| [ShareGPT4V](https://huggingface.co/datasets/Lin-Chen/ShareGPT4V) | 1.2M | GPT-4V detailed image descriptions | Longer captions (200-500 tokens), richer vocabulary |
+| [DOCCI](https://huggingface.co/datasets/google/docci) | 15K | Expert-written dense image descriptions | Extremely detailed, 100-300 words per image |
+| [Localized Narratives](https://huggingface.co/datasets/google/localized-narratives) | 850K | Spoken descriptions with mouse traces | Narrative structure, temporal ordering |
+| [DenseCap](https://huggingface.co/datasets/visual-genome/dense-captions) | 5.4M | Region-level dense captions | Fine-grained spatial descriptions |
+| [TextCaps](https://huggingface.co/datasets/lmms-lab/TextCaps) | 145K | Captions requiring OCR reading | Text-in-image understanding |
+| [VizWiz](https://huggingface.co/datasets/lmms-lab/VizWiz-VQA) | 32K | Captions from blind/low-vision users | Diverse, real-world, often longer descriptions |
+| [COCO Captions](https://huggingface.co/datasets/HuggingFaceM4/COCO) | 600K | 5 captions per image, human-written | Short but high-quality, broad coverage |
+| [SBU Captions](https://huggingface.co/datasets/sbu_captions) | 1M | Web-crawled image-caption pairs | Scale and diversity |
+
+### Planned — Domain Extension
+
+| Dataset | Size | Content | Why |
+|---|---|---|---|
+| [BookCorpus](https://huggingface.co/datasets/bookcorpus) | 11K books | Long-form narrative text | Exercise 8K context, literary language |
+| [Wikipedia](https://huggingface.co/datasets/wikipedia) | 6M articles | Encyclopedic text | General knowledge, factual density |
+| [Natural Questions](https://huggingface.co/datasets/google-research-datasets/natural_questions) | 300K | Question-answer pairs | QA capability for retrieval heads |
+| [MS MARCO](https://huggingface.co/datasets/microsoft/ms_marco) | 1M | Passages + queries | Retrieval training for reranker head |
 
 ## Architecture
 
@@ -152,51 +196,61 @@ Input text
         └── (B, 768) consensus-aligned embedding
 ```
 
-## The Consensus Distillation Pipeline
+## Usage
 
-```
-5 Expert Models (frozen)
-    │
-    ├── BERT-base-uncased        (110M, MLM)
-    ├── ModernBERT-base          (149M, MLM + rotary)
-    ├── RoBERTa-base             (125M, MLM + dynamic masking)
-    ├── ALBERT-base-v2           (12M, MLM + SOP + factorized)
-    └── DistilBERT-base          (66M, distilled from BERT)
-        │
-        ├── Extract embeddings on CC12M captions
-        ├── Whitened Procrustes alignment to shared space
-        ├── Consensus = normalized centroid
-        │   (proven constant to 3 decimal places across 5 seeds)
-        │
-        └── Train student with:
-            ├── InfoNCE(student, consensus)  — retrieval alignment
-            ├── MSE(student, consensus)      — direct regression
-            └── Pentachoron CV → 0.084       — geometric regularity
-```
+```python
+import torch
+from transformers import AutoTokenizer
+from caption_encoder import CaptionEncoder
 
-## Key Properties
+# Load
+tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
+model = CaptionEncoder(
+    vocab_size=30522, max_len=8192, d_model=384,
+    n_heads=6, n_layers=6, d_ff=1536, output_dim=768,
+    dropout=0.0, pad_token_id=0)
+model.load_state_dict(torch.load("best_model.pt", weights_only=True))
+model.eval()
 
-**Geometric regularity.** The embedding space has pentachoron CV ≈ 0.08–0.10, meaning local neighborhoods are uniformly distributed. The space is smooth, interpolable, and well-conditioned for downstream operations.
+# Encode
+texts = ["A cat sitting on a windowsill", "A dog playing fetch on the beach"]
+tokens = tokenizer(texts, max_length=512, padding="max_length",
+                   truncation=True, return_tensors="pt")
+with torch.no_grad():
+    embeddings = model(tokens["input_ids"], tokens["attention_mask"])
+
+# embeddings: (2, 768) L2-normalized
+similarity = embeddings[0] @ embeddings[1]
+print(f"Similarity: {similarity:.3f}")
+```
 
-**Multi-teacher consensus.** The target is the geometric intersection of five experts, not any single teacher. Individual model errors cancel. What remains is what five independent systems agree on.
+## Training Curve
 
-**Minimal data requirement.** The consensus manifold is so smooth (CV=0.084) that 18K examples were sufficient for R@1=1.000 on held-out data. The function from text to consensus embedding has a low Lipschitz constant.
+| Epoch | t_cos | v_cos | v_cv | Time |
+|---|---|---|---|---|
+| 1 | 0.804 | 0.803 | 0.104 | 689s |
+| 5 | 0.819 | 0.819 | 0.086 | 689s |
+| 10 | 0.831 | 0.829 | 0.087 | 689s |
+| 15 | 0.842 | 0.840 | 0.078 | 688s |
+| 20 | 0.851 | 0.849 | 0.078 | 690s |
+| 25 | 0.860 | 0.859 | 0.092 | 689s |
+| 30 | 0.863 | 0.862 | 0.082 | 689s |
 
-**8K position capacity.** Trained on 512-token sequences but position embeddings extend to 8,192. Ready for long-context applications without retraining.
+R@1=1.000 and t_acc=1.000 throughout all 30 epochs. Train/val gap < 0.002 — no overfitting on 500K samples.
 
 ## GEOLIP Family
 
-| System | Type | Output |
-|---|---|---|
-| [CLIP-L ctx576](https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576) | Memory bank | pooled (768,) |
-| [CLIP-L seq77](https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576-seq77) | Memory + sequence | pooled + seq (77, 768) |
-| [Meridian bigG](https://huggingface.co/AbstractPhil/geolip-clip-vit-bigG-patch14-ctx576-seq77) | Memory + sequence | pooled + seq (77, 1280) |
-| [Conduit v0](https://huggingface.co/AbstractPhil/geolip-bertenstein) | Multi-expert hub | aligned (1024,) |
-| **Consensus Distilled** | **Student** | **consensus (768,)** |
+| System | Type | Params | Output |
+|---|---|---|---|
+| [CLIP-L ctx576](https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576) | Memory bank | 34M | pooled (768,) |
+| [CLIP-L seq77](https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576-seq77) | Memory + sequence | 53M | pooled + seq (77, 768) |
+| [Meridian bigG](https://huggingface.co/AbstractPhil/geolip-clip-vit-bigG-patch14-ctx576-seq77) | Memory + sequence | 167M | pooled + seq (77, 1280) |
+| [Conduit v0](https://huggingface.co/AbstractPhil/geolip-bertenstein) | Multi-expert hub | 8.8M | aligned (1024,) |
+| **CaptionBERT-8192** | **Consensus distilled** | **26M** | **consensus (768,)** |
 
 ## Citation
 
-See [Geometric Memory Part I](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft1) and Part II for the full methodology.
+See [Geometric Memory Part I](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft1) and Part II for the full methodology, including the pentachoron consensus proof, whitened Procrustes alignment, compositional convolution experiments, and the path from accumulation-based memory to alignment-based distillation.
 
 ## License