Mithil Maske commited on
Commit
544e32b
Β·
verified Β·
1 Parent(s): 0af30f5

Rename package gelato_local -> quadembed, drop GELATO from project naming (credit stays in text)

Browse files
README.md CHANGED
@@ -13,13 +13,19 @@ tags:
13
  library_name: pytorch
14
  ---
15
 
16
- # gelato-local-nano
17
 
18
- A local, from-scratch-trained reproduction of **GELATO** (Geometry-preserving
19
- Embeddings via Locked Aligned TOwers), the architecture behind
20
- [jina-embeddings-v5-omni](https://arxiv.org/abs/2605.08384). It maps text,
21
- images, audio, and video into one shared 768-dimensional embedding space,
22
- trained end-to-end on a single 8GB consumer GPU (RTX 4060 laptop).
 
 
 
 
 
 
23
 
24
  This is not a copy of Jina's released checkpoint β€” the three encoders below
25
  are frozen, publicly available source models, and only two small projector
@@ -70,12 +76,11 @@ stayed under 2.7GB β€” well inside an 8GB budget.
70
 
71
  These numbers come after seven rounds of iterating on the vision projector
72
  specifically (architecture fixes, data scale, data diversity, projector
73
- capacity) β€” full write-up of what did and didn't move the needle is in this
74
- project's [feasibility write-up and README](https://github.com/) *(local
75
- project β€” see the blog post that accompanies this model for the full
76
- round-by-round story)*. Short version: architecture wasn't the bottleneck
77
- after round 3; raw training data volume was β€” R@1 only moved once the
78
- image-caption training set was scaled to ~172k pairs (round 7).
79
 
80
  ## Usage
81
 
@@ -84,10 +89,10 @@ pip install -r requirements.txt
84
  ```
85
 
86
  ```python
87
- from gelato_local import GelatoLocal
88
  from PIL import Image
89
 
90
- model = GelatoLocal.from_pretrained("checkpoints") # auto-detects cuda/cpu
91
 
92
  text_embeds = model.embed_text(["a dog running on the beach"])
93
  image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
@@ -137,7 +142,8 @@ Jina AI directly.
137
 
138
  ## Citation
139
 
140
- This repo reproduces the architecture described in:
 
141
 
142
  ```
143
  @article{gelato2026,
@@ -146,5 +152,6 @@ This repo reproduces the architecture described in:
146
  }
147
  ```
148
 
149
- Not affiliated with or endorsed by Jina AI β€” an independent, from-scratch
150
- reproduction built for a learning/portfolio project.
 
 
13
  library_name: pytorch
14
  ---
15
 
16
+ # QuadEmbed
17
 
18
+ A local, from-scratch-trained multimodal embedding model. "Quad" for the
19
+ four modalities it covers: text, image, audio, and video, all mapped into
20
+ one shared 768-dimensional embedding space, trained end-to-end on a single
21
+ 8GB consumer GPU (RTX 4060 laptop).
22
+
23
+ The architecture reproduces **GELATO** (Geometry-preserving Embeddings via
24
+ Locked Aligned TOwers), described by Jina AI in the paper behind
25
+ [jina-embeddings-v5-omni](https://arxiv.org/abs/2605.08384). Full credit for
26
+ the architecture and training recipe goes to that paper; this repo is an
27
+ independent, from-scratch reproduction of it, not a copy of Jina's released
28
+ weights.
29
 
30
  This is not a copy of Jina's released checkpoint β€” the three encoders below
31
  are frozen, publicly available source models, and only two small projector
 
76
 
77
  These numbers come after seven rounds of iterating on the vision projector
78
  specifically (architecture fixes, data scale, data diversity, projector
79
+ capacity). Short version: architecture wasn't the bottleneck after round 3;
80
+ raw training data volume was β€” R@1 only moved once the image-caption
81
+ training set was scaled to ~172k pairs (round 7). The full round-by-round
82
+ write-up, with what did and didn't move the needle, lives in the source
83
+ repo's README and feasibility notes (linked once published).
 
84
 
85
  ## Usage
86
 
 
89
  ```
90
 
91
  ```python
92
+ from quadembed import QuadEmbed
93
  from PIL import Image
94
 
95
+ model = QuadEmbed.from_pretrained("checkpoints") # auto-detects cuda/cpu
96
 
97
  text_embeds = model.embed_text(["a dog running on the beach"])
98
  image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
 
142
 
143
  ## Citation
144
 
145
+ QuadEmbed reproduces the architecture described in Jina AI's GELATO paper.
146
+ All credit for the original architecture and training recipe belongs there:
147
 
148
  ```
149
  @article{gelato2026,
 
152
  }
153
  ```
154
 
155
+ QuadEmbed is not affiliated with or endorsed by Jina AI β€” an independent,
156
+ from-scratch reproduction of their published architecture, built as a
157
+ learning and portfolio project.
examples/inference_example.py CHANGED
@@ -8,9 +8,9 @@ Run from the repo root:
8
  """
9
  from PIL import Image
10
 
11
- from gelato_local import GelatoLocal
12
 
13
- model = GelatoLocal.from_pretrained("checkpoints") # device auto-detects cuda/cpu
14
 
15
  texts = ["a photo of a dog running on the beach", "a bowl of ramen noodles"]
16
  text_embeds = model.embed_text(texts)
 
8
  """
9
  from PIL import Image
10
 
11
+ from quadembed import QuadEmbed
12
 
13
+ model = QuadEmbed.from_pretrained("checkpoints") # device auto-detects cuda/cpu
14
 
15
  texts = ["a photo of a dog running on the beach", "a bowl of ramen noodles"]
16
  text_embeds = model.embed_text(texts)
{gelato_local β†’ quadembed}/__init__.py RENAMED
@@ -1,11 +1,12 @@
1
- """gelato-local-nano: a local, from-scratch-trained reproduction of GELATO
2
- (the architecture behind jina-embeddings-v5-omni) -- frozen text/vision/audio
3
- encoders plus two small trained projectors, sharing one 768-dim embedding
4
- space across text, image, audio, and video.
 
5
 
6
- from gelato_local import GelatoLocal
7
 
8
- model = GelatoLocal.from_pretrained("checkpoints", device="cuda")
9
  text_embeds = model.embed_text(["a dog running on the beach"])
10
  image_embeds = model.embed_image([Image.open("photo.jpg")])
11
  similarity = text_embeds @ image_embeds.T
@@ -20,7 +21,7 @@ from .projectors import AudioProjector, VisionProjector
20
  from .video import embed_video, sample_frames
21
 
22
  __all__ = [
23
- "GelatoLocal",
24
  "TextEncoder",
25
  "VisionEncoder",
26
  "AudioEncoder",
@@ -31,7 +32,7 @@ __all__ = [
31
  ]
32
 
33
 
34
- class GelatoLocal:
35
  """Loads all three frozen encoders plus the trained projectors, and
36
  exposes one `embed_*` method per modality, each returning an L2-normalized
37
  [batch, 768] tensor in the shared space -- so any two outputs from any
 
1
+ """QuadEmbed: a local, from-scratch-trained multimodal embedding model
2
+ covering all four modalities, text, image, audio, and video, in one shared
3
+ 768-dim space. Reproduces the architecture behind Jina AI's GELATO
4
+ (jina-embeddings-v5-omni): frozen text/vision/audio encoders plus two small
5
+ trained projectors.
6
 
7
+ from quadembed import QuadEmbed
8
 
9
+ model = QuadEmbed.from_pretrained("checkpoints", device="cuda")
10
  text_embeds = model.embed_text(["a dog running on the beach"])
11
  image_embeds = model.embed_image([Image.open("photo.jpg")])
12
  similarity = text_embeds @ image_embeds.T
 
21
  from .video import embed_video, sample_frames
22
 
23
  __all__ = [
24
+ "QuadEmbed",
25
  "TextEncoder",
26
  "VisionEncoder",
27
  "AudioEncoder",
 
32
  ]
33
 
34
 
35
+ class QuadEmbed:
36
  """Loads all three frozen encoders plus the trained projectors, and
37
  exposes one `embed_*` method per modality, each returning an L2-normalized
38
  [batch, 768] tensor in the shared space -- so any two outputs from any
{gelato_local β†’ quadembed}/encoders.py RENAMED
File without changes
{gelato_local β†’ quadembed}/projectors.py RENAMED
File without changes
{gelato_local β†’ quadembed}/video.py RENAMED
File without changes