File size: 3,764 Bytes
f4c6329 20596b4 68e8925 f4c6329 68e8925 f4c6329 20596b4 f4c6329 20596b4 f4c6329 40c975a 20596b4 40c975a 20596b4 68e8925 20596b4 a7da308 20596b4 68e8925 20596b4 f4c6329 20596b4 f4c6329 20596b4 68e8925 20596b4 68e8925 20596b4 f4c6329 20596b4 68e8925 20596b4 68e8925 20596b4 68e8925 20596b4 68e8925 20596b4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
pipeline_tag: image-segmentation
license: mit
base_model: tue-mps/coco_panoptic_eomt_base_640_2x
library_name: zeromodels
tags:
- keras
- zeromodels
- eomt
- panoptic-segmentation
- image-segmentation
- arxiv:2503.19108
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/eomt-6a8eaf6dac390eae4c54893b) for all versions of EoMT.***
# Run EoMT with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/eomt/) [](https://huggingface.co/collections/zeromodels/eomt-6a8eaf6dac390eae4c54893b)
# zeromodels/eomt_base_coco_panoptic_640
Paper: [Your ViT is Secretly an Image Segmentation Model (arXiv:2503.19108)](https://arxiv.org/abs/2503.19108) · [HF Papers](https://huggingface.co/papers/2503.19108)
EoMT (Encoder-only Mask Transformer) keeps segmentation inside a plain ViT: learned query tokens are concatenated with patch tokens and run through the same ViT blocks. No pixel decoder, no deformable attention decoder.
For more details on the model, please go to the upstream [model card](https://huggingface.co/tue-mps/coco_panoptic_eomt_base_640_2x).
Pure-**Keras 3** conversion of [`tue-mps/coco_panoptic_eomt_base_640_2x`](https://huggingface.co/tue-mps/coco_panoptic_eomt_base_640_2x) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is a **panoptic** checkpoint (`EoMTUniversalSegment`).
## ✨ Quick start
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from PIL import Image
from zeromodels.models.eomt import EoMTUniversalSegment, EoMTImageProcessor
model = EoMTUniversalSegment.from_weights("zeromodels/eomt_base_coco_panoptic_640")
processor = EoMTImageProcessor.from_weights("zeromodels/eomt_base_coco_panoptic_640")
image = Image.open("your_image.jpg").convert("RGB")
output = model(processor(image)["pixel_values"], training=False)
result = processor.post_process_panoptic_segmentation(
output, target_size=(image.height, image.width)
)
print(result["segmentation"].shape)
```
Load any EoMT variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Task |
|---|---|---|
| `eomt_small_coco_panoptic_640` | [`zeromodels/eomt_small_coco_panoptic_640`](https://huggingface.co/zeromodels/eomt_small_coco_panoptic_640) | panoptic |
| `eomt_base_coco_panoptic_640` | [`zeromodels/eomt_base_coco_panoptic_640`](https://huggingface.co/zeromodels/eomt_base_coco_panoptic_640) | panoptic |
| `eomt_large_coco_panoptic_640` | [`zeromodels/eomt_large_coco_panoptic_640`](https://huggingface.co/zeromodels/eomt_large_coco_panoptic_640) | panoptic |
| `eomt_large_coco_instance_640` | [`zeromodels/eomt_large_coco_instance_640`](https://huggingface.co/zeromodels/eomt_large_coco_instance_640) | instance |
| `eomt_large_ade20k_semantic_512` | [`zeromodels/eomt_large_ade20k_semantic_512`](https://huggingface.co/zeromodels/eomt_large_ade20k_semantic_512) | semantic |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Use the post-processor that matches the checkpoint task (panoptic / instance / semantic).
- See [EoMT docs]({DOCS_URL}) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream weights: `EoMTUniversalSegment.from_weights("hf:tue-mps/coco_panoptic_eomt_base_640_2x")`.
## Special Thanks
A huge thank you to the TU/e MPS EoMT authors for creating and releasing these models.
License: MIT.
|