Auto Classes, Configs and Heads

Body plus head

Every checkpoint on the Hub has a config.json that records its architecture (for example BertForSequenceClassification) and its hyperparameters: number of layers, hidden size, vocabulary size and label names. Auto classes read that config and build the right Python class for you, so AutoModel.from_pretrained(id) works for BERT, RoBERTa or Llama without you naming the class. It helps to see a model as a body plus a head. AutoModel loads only the body, which returns hidden states: one vector per token. Task-specific classes such as AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoModelForQuestionAnswering, AutoModelForCausalLM and AutoModelForSeq2SeqLM add a small head on top that turns hidden states into task logits. If a checkpoint has no trained weights for the head you ask for, the library initialises the head randomly and warns you that some weights are newly initialised. That warning is expected before fine-tuning and a red flag at inference time.

Config, body and head

The config decides the architecture, the body produces hidden states and the head turns them into task-specific logits.

Loading a body and a body-with-head

num_labels sizes a new classification head. The warning about newly initialised weights is expected here because bert-base-uncased was never trained for this task.

from transformers import AutoConfig, AutoModel, AutoModelForSequenceClassification

model_id = "google-bert/bert-base-uncased"
config = AutoConfig.from_pretrained(model_id)
print(config.model_type, config.num_hidden_layers, config.hidden_size)  # bert 12 768

body = AutoModel.from_pretrained(model_id)  # hidden states only
clf = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    num_labels=3,
    id2label={0: "negative", 1: "neutral", 2: "positive"},
    label2id={"negative": 0, "neutral": 1, "positive": 2},
)
print(type(clf).__name__)  # BertForSequenceClassification

Read the "newly initialized" warning

If you see "Some weights of ... were not initialized from the model checkpoint" while you expect a ready-made classifier, you loaded the wrong checkpoint or the wrong Auto class. The predictions will be random until you fine-tune.

त्वरित जाँच: You want a model that outputs one score per class for a whole sentence. Which class do you load?

• AutoModelForSequenceClassification
• AutoModel
• AutoModelForTokenClassification
• AutoTokenizer