Gabriele commited on
Commit ·
6b864bd
1
Parent(s): 190b924
Remove zero-shot segmentation snippet, link to GitHub for advanced use cases
Browse files
README.md
CHANGED
|
@@ -12,7 +12,7 @@ pipeline_tag: zero-shot-image-classification
|
|
| 12 |
|
| 13 |
# TIPSv2 — SO400m/14
|
| 14 |
|
| 15 |
-
TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the SO400m variant with 412M vision params and 448M text params.
|
| 16 |
|
| 17 |
| Variant | Vision params | Text params | Embed dim | DPT Heads |
|
| 18 |
|---------|--------------|-------------|-----------|-----------|
|
|
@@ -78,16 +78,6 @@ similarity = cls @ text_emb.T
|
|
| 78 |
print(classes[similarity.argmax()]) # bus — predicted class
|
| 79 |
```
|
| 80 |
|
| 81 |
-
### Zero-shot segmentation
|
| 82 |
-
|
| 83 |
-
```python
|
| 84 |
-
classes = ["bus", "snow", "mountain", "house", "road"]
|
| 85 |
-
patch_feats = F.normalize(out.patch_tokens, dim=-1)
|
| 86 |
-
text_emb = F.normalize(model.encode_text(classes), dim=-1)
|
| 87 |
-
seg_map = (patch_feats @ text_emb.T).reshape(32, 32, len(classes)).argmax(dim=-1)
|
| 88 |
-
print(seg_map.shape) # (32, 32) — per-patch class prediction
|
| 89 |
-
```
|
| 90 |
-
|
| 91 |
### Visualize spatial features
|
| 92 |
|
| 93 |
```python
|
|
|
|
| 12 |
|
| 13 |
# TIPSv2 — SO400m/14
|
| 14 |
|
| 15 |
+
TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the SO400m variant with 412M vision params and 448M text params. Try the code snippets below or check out the [GitHub repo](https://github.com/google-deepmind/tips) for more use cases and visualizations, including zero-shot segmentation.
|
| 16 |
|
| 17 |
| Variant | Vision params | Text params | Embed dim | DPT Heads |
|
| 18 |
|---------|--------------|-------------|-----------|-----------|
|
|
|
|
| 78 |
print(classes[similarity.argmax()]) # bus — predicted class
|
| 79 |
```
|
| 80 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
### Visualize spatial features
|
| 82 |
|
| 83 |
```python
|