Running a Model by Hand

From tokens to probabilities without a pipeline

Running a model yourself means repeating the pipeline's three steps explicitly. First tokenize with return_tensors="pt". Then call model(**inputs); the double star unpacks input_ids and attention_mask as keyword arguments. The result is a ModelOutput, an object you can read by attribute (outputs.logits) or by key. Logits are raw, unnormalised scores, one per class; they can be any real number. To get probabilities, apply softmax along the last dimension, and to get a prediction take the argmax and look it up in model.config.id2label. Two habits matter at inference time. Call model.eval() so layers such as dropout behave deterministically (models loaded with from_pretrained already start in eval mode, but training code may switch it). And wrap the call in torch.no_grad() or torch.inference_mode() so PyTorch does not store the activations needed for gradients, which saves memory and time. Pass output_hidden_states=True when you need every layer's vectors.

Manual inference with PyTorch

The same result a pipeline gives, with every step visible.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

inputs = tok(["Loved the new syllabus", "The lab was a waste of time"],
             padding=True, return_tensors="pt")
with torch.inference_mode():
    outputs = model(**inputs)

probs = outputs.logits.softmax(dim=-1)   # shape: (batch, num_labels)
for p in probs:
    label_id = int(p.argmax())
    print(model.config.id2label[label_id], round(float(p[label_id]), 3))

Softmax and argmax, no library needed

The postprocessing step in plain Python, so you can see that logits are just scores.

import math

# Logits a sentiment classifier might return for one review
logits = [-1.2, 2.3]
id2label = {0: "NEGATIVE", 1: "POSITIVE"}

exps = [math.exp(x) for x in logits]
probs = [e / sum(exps) for e in exps]
best = max(range(len(probs)), key=probs.__getitem__)

print([round(p, 3) for p in probs])
print(id2label[best])

Output:

[0.029, 0.971]
POSITIVE

Logits are not probabilities

A logit of 2.3 is not "230 percent" or "2.3 times more likely". Only after softmax do the values sum to 1. When comparing models, compare probabilities or metrics, never raw logits from different models.

Quick check: Why wrap inference in `torch.inference_mode()` or `torch.no_grad()`?

• It turns logits into probabilities
• It moves the model to the GPU
• It stops PyTorch from recording operations for gradients, saving memory and time
• It switches dropout off