Mithil Maske commited on
Rename package gelato_local -> quadembed, drop GELATO from project naming (credit stays in text)
Browse files
README.md
CHANGED
|
@@ -13,13 +13,19 @@ tags:
|
|
| 13 |
library_name: pytorch
|
| 14 |
---
|
| 15 |
|
| 16 |
-
#
|
| 17 |
|
| 18 |
-
A local, from-scratch-trained
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
This is not a copy of Jina's released checkpoint β the three encoders below
|
| 25 |
are frozen, publicly available source models, and only two small projector
|
|
@@ -70,12 +76,11 @@ stayed under 2.7GB β well inside an 8GB budget.
|
|
| 70 |
|
| 71 |
These numbers come after seven rounds of iterating on the vision projector
|
| 72 |
specifically (architecture fixes, data scale, data diversity, projector
|
| 73 |
-
capacity)
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
image-caption training set was scaled to ~172k pairs (round 7).
|
| 79 |
|
| 80 |
## Usage
|
| 81 |
|
|
@@ -84,10 +89,10 @@ pip install -r requirements.txt
|
|
| 84 |
```
|
| 85 |
|
| 86 |
```python
|
| 87 |
-
from
|
| 88 |
from PIL import Image
|
| 89 |
|
| 90 |
-
model =
|
| 91 |
|
| 92 |
text_embeds = model.embed_text(["a dog running on the beach"])
|
| 93 |
image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
|
|
@@ -137,7 +142,8 @@ Jina AI directly.
|
|
| 137 |
|
| 138 |
## Citation
|
| 139 |
|
| 140 |
-
|
|
|
|
| 141 |
|
| 142 |
```
|
| 143 |
@article{gelato2026,
|
|
@@ -146,5 +152,6 @@ This repo reproduces the architecture described in:
|
|
| 146 |
}
|
| 147 |
```
|
| 148 |
|
| 149 |
-
|
| 150 |
-
reproduction
|
|
|
|
|
|
| 13 |
library_name: pytorch
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# QuadEmbed
|
| 17 |
|
| 18 |
+
A local, from-scratch-trained multimodal embedding model. "Quad" for the
|
| 19 |
+
four modalities it covers: text, image, audio, and video, all mapped into
|
| 20 |
+
one shared 768-dimensional embedding space, trained end-to-end on a single
|
| 21 |
+
8GB consumer GPU (RTX 4060 laptop).
|
| 22 |
+
|
| 23 |
+
The architecture reproduces **GELATO** (Geometry-preserving Embeddings via
|
| 24 |
+
Locked Aligned TOwers), described by Jina AI in the paper behind
|
| 25 |
+
[jina-embeddings-v5-omni](https://arxiv.org/abs/2605.08384). Full credit for
|
| 26 |
+
the architecture and training recipe goes to that paper; this repo is an
|
| 27 |
+
independent, from-scratch reproduction of it, not a copy of Jina's released
|
| 28 |
+
weights.
|
| 29 |
|
| 30 |
This is not a copy of Jina's released checkpoint β the three encoders below
|
| 31 |
are frozen, publicly available source models, and only two small projector
|
|
|
|
| 76 |
|
| 77 |
These numbers come after seven rounds of iterating on the vision projector
|
| 78 |
specifically (architecture fixes, data scale, data diversity, projector
|
| 79 |
+
capacity). Short version: architecture wasn't the bottleneck after round 3;
|
| 80 |
+
raw training data volume was β R@1 only moved once the image-caption
|
| 81 |
+
training set was scaled to ~172k pairs (round 7). The full round-by-round
|
| 82 |
+
write-up, with what did and didn't move the needle, lives in the source
|
| 83 |
+
repo's README and feasibility notes (linked once published).
|
|
|
|
| 84 |
|
| 85 |
## Usage
|
| 86 |
|
|
|
|
| 89 |
```
|
| 90 |
|
| 91 |
```python
|
| 92 |
+
from quadembed import QuadEmbed
|
| 93 |
from PIL import Image
|
| 94 |
|
| 95 |
+
model = QuadEmbed.from_pretrained("checkpoints") # auto-detects cuda/cpu
|
| 96 |
|
| 97 |
text_embeds = model.embed_text(["a dog running on the beach"])
|
| 98 |
image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
|
|
|
|
| 142 |
|
| 143 |
## Citation
|
| 144 |
|
| 145 |
+
QuadEmbed reproduces the architecture described in Jina AI's GELATO paper.
|
| 146 |
+
All credit for the original architecture and training recipe belongs there:
|
| 147 |
|
| 148 |
```
|
| 149 |
@article{gelato2026,
|
|
|
|
| 152 |
}
|
| 153 |
```
|
| 154 |
|
| 155 |
+
QuadEmbed is not affiliated with or endorsed by Jina AI β an independent,
|
| 156 |
+
from-scratch reproduction of their published architecture, built as a
|
| 157 |
+
learning and portfolio project.
|
examples/inference_example.py
CHANGED
|
@@ -8,9 +8,9 @@ Run from the repo root:
|
|
| 8 |
"""
|
| 9 |
from PIL import Image
|
| 10 |
|
| 11 |
-
from
|
| 12 |
|
| 13 |
-
model =
|
| 14 |
|
| 15 |
texts = ["a photo of a dog running on the beach", "a bowl of ramen noodles"]
|
| 16 |
text_embeds = model.embed_text(texts)
|
|
|
|
| 8 |
"""
|
| 9 |
from PIL import Image
|
| 10 |
|
| 11 |
+
from quadembed import QuadEmbed
|
| 12 |
|
| 13 |
+
model = QuadEmbed.from_pretrained("checkpoints") # device auto-detects cuda/cpu
|
| 14 |
|
| 15 |
texts = ["a photo of a dog running on the beach", "a bowl of ramen noodles"]
|
| 16 |
text_embeds = model.embed_text(texts)
|
{gelato_local β quadembed}/__init__.py
RENAMED
|
@@ -1,11 +1,12 @@
|
|
| 1 |
-
"""
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
|
|
|
| 5 |
|
| 6 |
-
from
|
| 7 |
|
| 8 |
-
model =
|
| 9 |
text_embeds = model.embed_text(["a dog running on the beach"])
|
| 10 |
image_embeds = model.embed_image([Image.open("photo.jpg")])
|
| 11 |
similarity = text_embeds @ image_embeds.T
|
|
@@ -20,7 +21,7 @@ from .projectors import AudioProjector, VisionProjector
|
|
| 20 |
from .video import embed_video, sample_frames
|
| 21 |
|
| 22 |
__all__ = [
|
| 23 |
-
"
|
| 24 |
"TextEncoder",
|
| 25 |
"VisionEncoder",
|
| 26 |
"AudioEncoder",
|
|
@@ -31,7 +32,7 @@ __all__ = [
|
|
| 31 |
]
|
| 32 |
|
| 33 |
|
| 34 |
-
class
|
| 35 |
"""Loads all three frozen encoders plus the trained projectors, and
|
| 36 |
exposes one `embed_*` method per modality, each returning an L2-normalized
|
| 37 |
[batch, 768] tensor in the shared space -- so any two outputs from any
|
|
|
|
| 1 |
+
"""QuadEmbed: a local, from-scratch-trained multimodal embedding model
|
| 2 |
+
covering all four modalities, text, image, audio, and video, in one shared
|
| 3 |
+
768-dim space. Reproduces the architecture behind Jina AI's GELATO
|
| 4 |
+
(jina-embeddings-v5-omni): frozen text/vision/audio encoders plus two small
|
| 5 |
+
trained projectors.
|
| 6 |
|
| 7 |
+
from quadembed import QuadEmbed
|
| 8 |
|
| 9 |
+
model = QuadEmbed.from_pretrained("checkpoints", device="cuda")
|
| 10 |
text_embeds = model.embed_text(["a dog running on the beach"])
|
| 11 |
image_embeds = model.embed_image([Image.open("photo.jpg")])
|
| 12 |
similarity = text_embeds @ image_embeds.T
|
|
|
|
| 21 |
from .video import embed_video, sample_frames
|
| 22 |
|
| 23 |
__all__ = [
|
| 24 |
+
"QuadEmbed",
|
| 25 |
"TextEncoder",
|
| 26 |
"VisionEncoder",
|
| 27 |
"AudioEncoder",
|
|
|
|
| 32 |
]
|
| 33 |
|
| 34 |
|
| 35 |
+
class QuadEmbed:
|
| 36 |
"""Loads all three frozen encoders plus the trained projectors, and
|
| 37 |
exposes one `embed_*` method per modality, each returning an L2-normalized
|
| 38 |
[batch, 768] tensor in the shared space -- so any two outputs from any
|
{gelato_local β quadembed}/encoders.py
RENAMED
|
File without changes
|
{gelato_local β quadembed}/projectors.py
RENAMED
|
File without changes
|
{gelato_local β quadembed}/video.py
RENAMED
|
File without changes
|