Running a Pretrained Segmentation Model

Logits per class per pixel

A semantic segmentation model outputs, for every pixel, a score per class; taking the argmax gives the class map. Pretrained models (for example DeepLabV3 on the 21 PASCAL VOC classes) handle common categories; for your own classes you fine-tune with pixel labels. Post-processing (removing tiny regions, smoothing boundaries) often improves results. Inspect predictions visually: small spurious regions are common and easy to miss in averages.

DeepLabV3 (MobileNetV3) on the astronaut photo, run

I ran this on CPU with Python 3, OpenCV 5.0.0, scikit-image 0.26.0, PyTorch 2.14.1 and torchvision 0.29.1, using scikit-image's bundled sample photos and torchvision's published pretrained weights. The 520 x 520 output mask labels 53.0% of pixels as background and 46.3% as person, plus 0.7% wrongly labelled bottle: a small spurious region of the kind post-processing or a better model would remove.

import torch
from torchvision.models.segmentation import deeplabv3_mobilenet_v3_large, DeepLabV3_MobileNet_V3_Large_Weights
from skimage import data
weights = DeepLabV3_MobileNet_V3_Large_Weights.DEFAULT
model = deeplabv3_mobilenet_v3_large(weights=weights).eval()
img = torch.tensor(data.astronaut()).permute(2, 0, 1)
x = weights.transforms()(img)[None]
with torch.no_grad():
    mask = model(x)["out"].argmax(1)[0]
cats = weights.meta["categories"]
counts = torch.bincount(mask.flatten(), minlength=len(cats))
print("mask shape:", tuple(mask.shape))
for i in counts.argsort(descending=True)[:3]:
    if counts[i] > 0:
        print(f"{cats[i]:<12} {counts[i].item() / mask.numel():.1%} of pixels")

Output:

mask shape: (520, 520)
__background__ 53.0% of pixels
person       46.3% of pixels
bottle       0.7% of pixels

Overlay masks on images

Always look at masks overlaid on the original images; numbers alone hide boundary and small-region errors.

Quick check: How is the final class map usually obtained from a segmentation model?

• Using the image's file name
• Averaging all pixels
• Choosing a random class
• Taking the argmax over class scores for each pixel