Gabriele commited on
Commit ·
8c6c309
1
Parent(s): d075dda
Remove zero-shot segmentation snippet, link to GitHub for advanced use cases
Browse files
README.md
CHANGED
|
@@ -12,7 +12,7 @@ pipeline_tag: zero-shot-image-classification
|
|
| 12 |
|
| 13 |
# TIPSv2 — B/14
|
| 14 |
|
| 15 |
-
TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the Base variant with 86M vision params and 110M text params.
|
| 16 |
|
| 17 |
| Variant | Vision params | Text params | Embed dim | DPT Heads |
|
| 18 |
|---------|--------------|-------------|-----------|-----------|
|
|
@@ -78,16 +78,6 @@ similarity = cls @ text_emb.T
|
|
| 78 |
print(classes[similarity.argmax()]) # bus — predicted class
|
| 79 |
```
|
| 80 |
|
| 81 |
-
### Zero-shot segmentation
|
| 82 |
-
|
| 83 |
-
```python
|
| 84 |
-
classes = ["bus", "snow", "mountain", "house", "road"]
|
| 85 |
-
patch_feats = F.normalize(out.patch_tokens, dim=-1)
|
| 86 |
-
text_emb = F.normalize(model.encode_text(classes), dim=-1)
|
| 87 |
-
seg_map = (patch_feats @ text_emb.T).reshape(32, 32, len(classes)).argmax(dim=-1)
|
| 88 |
-
print(seg_map.shape) # (32, 32) — per-patch class prediction
|
| 89 |
-
```
|
| 90 |
-
|
| 91 |
### Visualize spatial features
|
| 92 |
|
| 93 |
```python
|
|
|
|
| 12 |
|
| 13 |
# TIPSv2 — B/14
|
| 14 |
|
| 15 |
+
TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the Base variant with 86M vision params and 110M text params. Try the code snippets below or check out the [GitHub repo](https://github.com/google-deepmind/tips) for more use cases and visualizations, including zero-shot segmentation.
|
| 16 |
|
| 17 |
| Variant | Vision params | Text params | Embed dim | DPT Heads |
|
| 18 |
|---------|--------------|-------------|-----------|-----------|
|
|
|
|
| 78 |
print(classes[similarity.argmax()]) # bus — predicted class
|
| 79 |
```
|
| 80 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
### Visualize spatial features
|
| 82 |
|
| 83 |
```python
|