Picking the Right Model Family

Which family for which job

Choosing a model starts with the task. Encoder-only models (BERT, RoBERTa, DeBERTa and smaller distilled versions) read the whole input bidirectionally and are strong, cheap choices for classification, NER, extractive QA and embeddings for semantic search. Decoder-only models (GPT-2, Llama, Mistral, Qwen, Gemma and similar) predict the next token and are loaded with AutoModelForCausalLM; they power chatbots, code assistants and open-ended generation. Encoder-decoder models (T5, BART, many translation models) are loaded with AutoModelForSeq2SeqLM and map one sequence to another. A large decoder can also classify by prompting, but a small fine-tuned encoder is often faster, cheaper and more consistent for a fixed label set. To get a sentence embedding from an encoder, a common recipe is mean pooling: average the token vectors, ignoring padding via the attention mask. Dedicated embedding models trained for similarity usually beat raw BERT vectors.

Mean-pooled sentence embeddings

Multiply by the mask so padding vectors do not drag the average, then compare with cosine similarity.

import torch
from transformers import AutoTokenizer, AutoModel

model_id = "sentence-transformers/all-MiniLM-L6-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

sentences = ["How do I reset my password?", "I forgot my login password"]
inputs = tok(sentences, padding=True, truncation=True, return_tensors="pt")
with torch.inference_mode():
    hidden = model(**inputs).last_hidden_state          # (batch, tokens, dim)

mask = inputs["attention_mask"].unsqueeze(-1).float()  # (batch, tokens, 1)
emb = (hidden * mask).sum(dim=1) / mask.sum(dim=1)     # mean over real tokens
emb = torch.nn.functional.normalize(emb, dim=-1)
print(float(emb[0] @ emb[1]))  # cosine similarity, high for paraphrases

Reader, writer, translator

An encoder is the student who reads the whole question paper before answering. A decoder is the storyteller who speaks one word at a time and cannot take words back. An encoder-decoder is the court interpreter who listens to the full sentence first, then speaks it again in another language.

Quick check: Which Auto class should you use to load a decoder-only chat model for text generation?

• AutoModelForSeq2SeqLM
• AutoModelForCausalLM
• AutoModelForMaskedLM
• AutoModelForSequenceClassification