Chat Models and Chat Templates

Each chat model has its own prompt format

Instruction-tuned chat models were trained on conversations wrapped in special markers that separate the system, user and assistant turns. The exact markers differ between model families, and using the wrong format quietly degrades answers. Instead of hand-writing these strings, store the conversation as a list of messages, each a dictionary with a role and content, and call tokenizer.apply_chat_template(). It uses a Jinja template saved with the tokenizer to render the correct string. Pass add_generation_prompt=True so the template ends with the marker that tells the model "now the assistant speaks"; without it the model may continue the user's message instead. The text-generation pipeline also accepts a list of messages directly, applies the template for you and returns the conversation with the new assistant turn appended. Base (non-chat) checkpoints usually have no chat template, so they should be prompted with plain text.

Rendering a conversation with the model's template

return_dict=True gives input_ids and attention_mask together, ready for generate().

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")

messages = [
    {"role": "system", "content": "You are a concise tutor for engineering students."},
    {"role": "user", "content": "Explain overfitting in one sentence."},
]
print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))

inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_dict=True, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The chat-aware pipeline

Give the pipeline messages; it returns the whole conversation with the assistant's reply as the last item.

from transformers import pipeline

chat = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")
messages = [{"role": "user", "content": "Give me two tips for a viva exam."}]
result = chat(messages, max_new_tokens=80)
print(result[0]["generated_text"][-1]["content"])

Quick check: What does `add_generation_prompt=True` add to a rendered chat prompt?

• A system message with safety rules
• The model's end-of-sequence token
• The marker that starts the assistant's turn, so the model replies instead of continuing the user
• A random seed for sampling