# Common Pitfalls and Evaluating Quality — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-pitfalls-eval

> Avoid typical local-LLM mistakes and compare models or quantizations with a small, repeatable evaluation.

## Measure on your task, not on vibes

The usual failures are predictable. **Wrong variant or template**: a base model or broken template gives rambling output. **Silent truncation**: a context smaller than your documents drops text without an error. **Spilling to CPU**: a model slightly too large for VRAM runs several times slower. **Over-trusting small models**: a 3B model can sound confident while inventing facts, citations or API names. **Unpinned versions**: pulling `latest` months later can bring a different model. The fix for most of these is a small **evaluation set**: twenty to fifty real inputs from your task with expected outputs or a scoring rule. Run every candidate model, quantization or prompt change against it with a fixed seed and low temperature, and record accuracy, latency and memory. For a pure quality signal between quantizations of one model, llama.cpp's `llama-perplexity` measures how well each predicts a reference text; lower is better, but task scores matter more.

## A tiny repeatable eval harness

Needs a running server. Swap MODELS to compare tags; the seed and temperature keep runs comparable.

```python
import time
from ollama import chat

CASES = [("2 + 2 * 3 = ? Reply with the number only.", "8"),
         ("Capital of Karnataka? One word.", "Bengaluru")]
MODELS = ["llama3.2:3b", "qwen2.5:7b"]

for model in MODELS:
    correct, start = 0, time.perf_counter()
    for prompt, expected in CASES:
        out = chat(model=model, messages=[{"role": "user", "content": prompt}],
                   options={"temperature": 0, "seed": 1}).message.content
        correct += expected.lower() in out.lower()
    print(f"{model}: {correct}/{len(CASES)} in {time.perf_counter() - start:.1f}s")
```

## Mock tests before the real exam

You would not judge your exam preparation by how confident you feel; you take a mock test with known answers. An eval set is a mock test for a model: same questions every time, scored the same way, so you can see whether a change actually helped.

**Quiz:** You want to know whether Q4_K_M is good enough compared to Q8_0 for your support-ticket classifier. What is the best approach?

- [ ] Ask the model which quantization it prefers
- [ ] Pick whichever downloads faster
- [ ] Assume Q8_0 is always required
- [x] Run both on the same labelled set of real tickets with fixed settings and compare accuracy and speed

*Answer:* Run both on the same labelled set of real tickets with fixed settings and compare accuracy and speed. A task-specific evaluation with fixed settings answers the question directly; perplexity is a useful secondary signal.