Common Pitfalls and Evaluating Quality

Measure on your task, not on vibes

The usual failures are predictable. Wrong variant or template: a base model or broken template gives rambling output. Silent truncation: a context smaller than your documents drops text without an error. Spilling to CPU: a model slightly too large for VRAM runs several times slower. Over-trusting small models: a 3B model can sound confident while inventing facts, citations or API names. Unpinned versions: pulling latest months later can bring a different model. The fix for most of these is a small evaluation set: twenty to fifty real inputs from your task with expected outputs or a scoring rule. Run every candidate model, quantization or prompt change against it with a fixed seed and low temperature, and record accuracy, latency and memory. For a pure quality signal between quantizations of one model, llama.cpp's llama-perplexity measures how well each predicts a reference text; lower is better, but task scores matter more.

A tiny repeatable eval harness

Needs a running server. Swap MODELS to compare tags; the seed and temperature keep runs comparable.

import time
from ollama import chat

CASES = [("2 + 2 * 3 = ? Reply with the number only.", "8"),
         ("Capital of Karnataka? One word.", "Bengaluru")]
MODELS = ["llama3.2:3b", "qwen2.5:7b"]

for model in MODELS:
    correct, start = 0, time.perf_counter()
    for prompt, expected in CASES:
        out = chat(model=model, messages=[{"role": "user", "content": prompt}],
                   options={"temperature": 0, "seed": 1}).message.content
        correct += expected.lower() in out.lower()
    print(f"{model}: {correct}/{len(CASES)} in {time.perf_counter() - start:.1f}s")

Mock tests before the real exam

You would not judge your exam preparation by how confident you feel; you take a mock test with known answers. An eval set is a mock test for a model: same questions every time, scored the same way, so you can see whether a change actually helped.

Quick check: You want to know whether Q4_K_M is good enough compared to Q8_0 for your support-ticket classifier. What is the best approach?

• Ask the model which quantization it prefers
• Pick whichever downloads faster
• Assume Q8_0 is always required
• Run both on the same labelled set of real tickets with fixed settings and compare accuracy and speed