Mithil Maske commited on
Commit
12b0c79
verified
1 Parent(s): 0e0e4ff

Point install instructions at the published PyPI package

Browse files
Files changed (1) hide show
  1. README.md +17 -6
README.md CHANGED
@@ -32,8 +32,10 @@ are frozen, publicly available source models, and only two small projector
32
  heads (a few million parameters total) were trained from scratch on public
33
  datasets to align them into a shared space.
34
 
35
- 馃摑 Full write-up (how it was built, and seven rounds of what did and didn't
36
- work): [I Built a Multimodal Embedding Model From Scratch on an RTX 4060](https://medium.com/@mithilmaske/i-built-a-multimodal-embedding-model-from-scratch-on-an-rtx-4060-text-image-audio-and-video-ab1fef04f1cd)
 
 
37
 
38
  ## Architecture
39
 
@@ -91,14 +93,15 @@ as a full narrative write-up on Medium:
91
  ## Usage
92
 
93
  ```bash
94
- pip install -r requirements.txt
 
95
  ```
96
 
97
  ```python
98
  from quadembed import QuadEmbed
99
  from PIL import Image
100
 
101
- model = QuadEmbed.from_pretrained("checkpoints") # auto-detects cuda/cpu
102
 
103
  text_embeds = model.embed_text(["a dog running on the beach"])
104
  image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
@@ -106,8 +109,16 @@ image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
106
  similarity = text_embeds @ image_embeds.T # already L2-normalized -> cosine similarity
107
  ```
108
 
109
- See `examples/inference_example.py` for a complete runnable script, including
110
- audio and video usage.
 
 
 
 
 
 
 
 
111
 
112
  ## Important caveat: two different vision checkpoints
113
 
 
32
  heads (a few million parameters total) were trained from scratch on public
33
  datasets to align them into a shared space.
34
 
35
+ 馃摝 **`pip install quadembed`** ([PyPI](https://pypi.org/project/quadembed/))
36
+ 馃捇 [Source on GitHub](https://github.com/mithilai/QuadEmbed)
37
+ 馃摑 [Full write-up on Medium](https://medium.com/@mithilmaske/i-built-a-multimodal-embedding-model-from-scratch-on-an-rtx-4060-text-image-audio-and-video-ab1fef04f1cd)
38
+ (how it was built, and seven rounds of what did and didn't work)
39
 
40
  ## Architecture
41
 
 
93
  ## Usage
94
 
95
  ```bash
96
+ pip install quadembed # text + image + audio
97
+ pip install quadembed[video] # adds video support
98
  ```
99
 
100
  ```python
101
  from quadembed import QuadEmbed
102
  from PIL import Image
103
 
104
+ model = QuadEmbed.from_pretrained() # downloads these weights automatically
105
 
106
  text_embeds = model.embed_text(["a dog running on the beach"])
107
  image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])
 
109
  similarity = text_embeds @ image_embeds.T # already L2-normalized -> cosine similarity
110
  ```
111
 
112
+ Each encoder costs memory and download time, so load only what you need:
113
+
114
+ ```python
115
+ model = QuadEmbed.from_pretrained(modalities=("text", "vision")) # skip audio
116
+ model = QuadEmbed.from_pretrained(device="cpu") # force CPU
117
+ ```
118
+
119
+ Audio takes mono float32 arrays at 16 kHz; video takes a file path
120
+ (`model.embed_video_file("clip.mp4")`). See
121
+ `examples/inference_example.py` for a complete runnable script.
122
 
123
  ## Important caveat: two different vision checkpoints
124