Zero-Shot Classification and Retrieval

Classes as sentences

Zero-shot classification means classifying into labels the model was never explicitly trained on. With CLIP, you write each label as a sentence, typically "a photo of a {label}", embed all sentences with the text tower, embed the image with the image tower, and pick the sentence with the highest similarity. A softmax over the similarities gives probabilities. Prompt wording matters: "a photo of a dog" usually works better than the bare word "dog" because it resembles the captions seen in training, and averaging embeddings over several templates ("a blurry photo of a {}", "a close-up photo of a {}") often helps further. The same embeddings power retrieval: embed a text query and rank stored image vectors by similarity. Zero-shot is a strong baseline, but on specialised domains such as medical scans or satellite images, fine-tuning or a linear probe on your labelled data usually beats it.

Zero-shot classification with toy embeddings

The image vector is closest to the "cat" prompt, so the softmax puts almost all probability there.

import numpy as np

def normalize(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

image_emb = normalize(np.array([0.2, 0.9, 0.1]))
labels = ["dog", "cat", "car"]
prompts = [f"a photo of a {l}" for l in labels]
text_emb = normalize(np.array([[0.9, 0.2, 0.1],    # "a photo of a dog"
                               [0.3, 0.9, 0.0],    # "a photo of a cat"
                               [0.0, 0.1, 1.0]]))  # "a photo of a car"

scores = text_emb @ image_emb * 10       # logit scale (CLIP learns its own)
probs = np.exp(scores - scores.max())
probs /= probs.sum()
for p, prob in zip(prompts, probs):
    print(f"{p:20s} {prob:.3f}")

Output:

a photo of a dog     0.004
a photo of a cat     0.996
a photo of a car     0.000

The real thing with Hugging Face

The same idea with a real CLIP checkpoint. It downloads weights and needs an image file, so the output is not shown.

import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

name = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(name)
processor = CLIPProcessor.from_pretrained(name)

image = Image.open("street.jpg")
labels = ["a photo of an auto-rickshaw", "a photo of a bus", "a photo of a bicycle"]
inputs = processor(text=labels, images=image, return_tensors="pt", padding=True)

with torch.no_grad():
    probs = model(**inputs).logits_per_image.softmax(dim=-1)[0]
for label, p in zip(labels, probs):
    print(f"{label}: {p:.3f}")

Softmax hides "none of the above"

Probabilities always sum to 1 across your labels, so a photo of a dosa will still be confidently assigned to one of "bus", "bicycle" or "auto-rickshaw". Add an explicit catch-all prompt or threshold the raw similarity score.

त्वरित जाँच: You want CLIP to recognise a new class it has never been trained on as a label. What do you need to add?

• A new output neuron and retraining
• Thousands of labelled images of the class
• A text prompt describing the class
• A larger image encoder