Mithil Maske commited on
Point install instructions at the published PyPI package
Browse files
README.md
CHANGED
|
@@ -32,8 +32,10 @@ are frozen, publicly available source models, and only two small projector
|
|
| 32 |
heads (a few million parameters total) were trained from scratch on public
|
| 33 |
datasets to align them into a shared space.
|
| 34 |
|
| 35 |
-
|
| 36 |
-
|
|
|
|
|
|
|
| 37 |
|
| 38 |
## Architecture
|
| 39 |
|
|
@@ -91,14 +93,15 @@ as a full narrative write-up on Medium:
|
|
| 91 |
## Usage
|
| 92 |
|
| 93 |
```bash
|
| 94 |
-
pip install
|
|
|
|
| 95 |
```
|
| 96 |
|
| 97 |
```python
|
| 98 |
from quadembed import QuadEmbed
|
| 99 |
from PIL import Image
|
| 100 |
|
| 101 |
-
model = QuadEmbed.from_pretrained(
|
| 102 |
|
| 103 |
text_embeds = model.embed_text(["a dog running on the beach"])
|
| 104 |
image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
|
|
@@ -106,8 +109,16 @@ image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
|
|
| 106 |
similarity = text_embeds @ image_embeds.T # already L2-normalized -> cosine similarity
|
| 107 |
```
|
| 108 |
|
| 109 |
-
|
| 110 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
## Important caveat: two different vision checkpoints
|
| 113 |
|
|
|
|
| 32 |
heads (a few million parameters total) were trained from scratch on public
|
| 33 |
datasets to align them into a shared space.
|
| 34 |
|
| 35 |
+
馃摝 **`pip install quadembed`** ([PyPI](https://pypi.org/project/quadembed/)) 路
|
| 36 |
+
馃捇 [Source on GitHub](https://github.com/mithilai/QuadEmbed) 路
|
| 37 |
+
馃摑 [Full write-up on Medium](https://medium.com/@mithilmaske/i-built-a-multimodal-embedding-model-from-scratch-on-an-rtx-4060-text-image-audio-and-video-ab1fef04f1cd)
|
| 38 |
+
(how it was built, and seven rounds of what did and didn't work)
|
| 39 |
|
| 40 |
## Architecture
|
| 41 |
|
|
|
|
| 93 |
## Usage
|
| 94 |
|
| 95 |
```bash
|
| 96 |
+
pip install quadembed # text + image + audio
|
| 97 |
+
pip install quadembed[video] # adds video support
|
| 98 |
```
|
| 99 |
|
| 100 |
```python
|
| 101 |
from quadembed import QuadEmbed
|
| 102 |
from PIL import Image
|
| 103 |
|
| 104 |
+
model = QuadEmbed.from_pretrained() # downloads these weights automatically
|
| 105 |
|
| 106 |
text_embeds = model.embed_text(["a dog running on the beach"])
|
| 107 |
image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
|
|
|
|
| 109 |
similarity = text_embeds @ image_embeds.T # already L2-normalized -> cosine similarity
|
| 110 |
```
|
| 111 |
|
| 112 |
+
Each encoder costs memory and download time, so load only what you need:
|
| 113 |
+
|
| 114 |
+
```python
|
| 115 |
+
model = QuadEmbed.from_pretrained(modalities=("text", "vision")) # skip audio
|
| 116 |
+
model = QuadEmbed.from_pretrained(device="cpu") # force CPU
|
| 117 |
+
```
|
| 118 |
+
|
| 119 |
+
Audio takes mono float32 arrays at 16 kHz; video takes a file path
|
| 120 |
+
(`model.embed_video_file("clip.mp4")`). See
|
| 121 |
+
`examples/inference_example.py` for a complete runnable script.
|
| 122 |
|
| 123 |
## Important caveat: two different vision checkpoints
|
| 124 |
|