You can deploy this model and start earning money today!

kerasformers/
eomt_base_coco_panoptic_640

Name: eomt_base_coco_panoptic_640
Availability: InStock

kerasformers

keras

eomt

panoptic-segmentation

tue-mps/coco_panoptic_eomt_base_640_2x

finetune

mit

Views

No views yet

See our collection for all versions of EoMT.

Run EoMT with Keras 3: JAX, PyTorch, or TensorFlow

GitHub

zeromodels/eomt_base_coco_panoptic_640

Paper: Your ViT is Secretly an Image Segmentation Model (arXiv:2503.19108) · HF Papers

EoMT (Encoder-only Mask Transformer) keeps segmentation inside a plain ViT: learned query tokens are concatenated with patch tokens and run through the same ViT blocks. No pixel decoder, no deformable attention decoder.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of tue-mps/coco_panoptic_eomt_base_640_2x for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a panoptic checkpoint (EoMTUniversalSegment).

✨ Quick start

python

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.eomt import EoMTUniversalSegment, EoMTImageProcessor

model = EoMTUniversalSegment.from_weights("zeromodels/eomt_base_coco_panoptic_640")
processor = EoMTImageProcessor.from_weights("zeromodels/eomt_base_coco_panoptic_640")

image = Image.open("your_image.jpg").convert("RGB")
output = model(processor(image)["pixel_values"], training=False)
result = processor.post_process_panoptic_segmentation(
    output, target_size=(image.height, image.width)
)
print(result["segmentation"].shape)

Load any EoMT variant the same way with from_weights("zeromodels/<variant>"):

Variant	Hub	Task
`eomt_small_coco_panoptic_640`	`zeromodels/eomt_small_coco_panoptic_640`	panoptic
`eomt_base_coco_panoptic_640`	`zeromodels/eomt_base_coco_panoptic_640`	panoptic
`eomt_large_coco_panoptic_640`	`zeromodels/eomt_large_coco_panoptic_640`	panoptic
`eomt_large_coco_instance_640`	`zeromodels/eomt_large_coco_instance_640`	instance
`eomt_large_ade20k_semantic_512`	`zeromodels/eomt_large_ade20k_semantic_512`	semantic

Tips

Set KERAS_BACKEND before importing Keras / zeromodels.
Use the post-processor that matches the checkpoint task (panoptic / instance / semantic).
See EoMT docs and Loading Weights.
Community / upstream weights: EoMTUniversalSegment.from_weights("hf:tue-mps/coco_panoptic_eomt_base_640_2x").

Special Thanks

A huge thank you to the TU/e MPS EoMT authors for creating and releasing these models.

License: MIT.

kerasformers/eomt_base_coco_panoptic_640