LLM Engineering Foundations interview questions

31 questions from the LLM Engineering Foundations course, each with the correct answer and an explanation. Try to answer before you open the answer.

From Demo to Product

1. Which property of LLM components most changes how we test them?

  1. They always run on laptops
  2. They are written in Python
  3. Outputs are non-deterministic and have no single correct answer
  4. They never change
Show answer

Answer: Outputs are non-deterministic and have no single correct answer

We therefore measure behaviour statistically over many cases.

Lesson: What LLM Engineering Is

2. Which component sits after the model call and before the user sees a response?

  1. Parsing, validation and safety checks
  2. Training the base model
  3. Choosing the GPU
  4. Writing the system prompt
Show answer

Answer: Parsing, validation and safety checks

Output handling turns raw text into something safe and usable.

Lesson: A Reference Architecture for an LLM Feature

3. Why define metrics and targets before building the prompt?

  1. There is no reason
  2. Because prompts cannot be written otherwise
  3. To make documentation longer
  4. So you can tell objectively whether a change improved the system
Show answer

Answer: So you can tell objectively whether a change improved the system

Without a target, every change is an opinion.

Lesson: Defining Success: Requirements and Metrics

4. What is the most valuable early investment in an LLM project?

  1. The longest possible system prompt
  2. A realistic evaluation set with a scoring method
  3. The largest model available
  4. A fancy logo
Show answer

Answer: A realistic evaluation set with a scoring method

Everything else depends on being able to measure.

Lesson: From Prototype to Production: A Staged Path

Evaluation That You Can Trust

5. Why keep a held-out test set that you do not tune against?

  1. Held-out sets run faster
  2. Tuning on the same cases overstates quality (overfitting to the test)
  3. They are required by APIs
  4. They remove the need for scoring
Show answer

Answer: Tuning on the same cases overstates quality (overfitting to the test)

You need an honest measurement on cases the system was not tuned to.

Lesson: An Evaluation Harness

6. What happens to the confidence interval as the test set grows?

  1. It stays exactly the same
  2. It widens
  3. It disappears
  4. It narrows, so small differences become detectable
Show answer

Answer: It narrows, so small differences become detectable

More cases mean less uncertainty about the true rate.

Lesson: Noise and Confidence Intervals

7. In a paired comparison, which cases carry information about which version is better?

  1. Those where one version passes and the other fails
  2. Those where both pass
  3. Those where both fail
  4. All cases equally
Show answer

Answer: Those where one version passes and the other fails

Agreement cannot separate the versions; disagreements can.

Lesson: Comparing Two Versions Fairly

8. What is a simple way to reduce position bias in pairwise judging?

  1. Use a shorter rubric
  2. Always show the better answer first
  3. Judge each pair in both orders and combine the results
  4. Ignore the judge
Show answer

Answer: Judge each pair in both orders and combine the results

Swapping order cancels a bias that favours position.

Lesson: LLM-as-Judge: Useful, Biased, Must Be Calibrated

9. Why is it a problem if test questions appear in the few-shot examples of the prompt?

  1. Few-shot examples are forbidden
  2. It makes the prompt shorter
  3. The system has seen the answers, so the score overstates real performance
  4. It has no effect
Show answer

Answer: The system has seen the answers, so the score overstates real performance

Evaluation requires cases the system has not been shown.

Lesson: Test-Set Hygiene: Leakage and Contamination

Engineering for Quality

10. Why record a release version ID with every request?

  1. So you can trace any answer to the exact prompt, model and settings
  2. To make requests larger
  3. Because APIs require it
  4. It has no use
Show answer

Answer: So you can trace any answer to the exact prompt, model and settings

Traceability is the basis of debugging and rollback.

Lesson: Prompts, Models and Settings as Versioned Artefacts

11. What should you do when the release gate fails?

  1. Delete the failing cases
  2. Lower the threshold
  3. Skip the gate once
  4. Investigate the flipped cases and fix the cause
Show answer

Answer: Investigate the flipped cases and fix the cause

A gate is only useful if failing it leads to understanding, not bypass.

Lesson: Release Gates and CI for LLM Changes

12. Why bound the repair loop?

  1. Because models dislike retries
  2. To avoid endless retries and runaway cost; fall back safely instead
  3. To make outputs longer
  4. Repair loops should be unlimited
Show answer

Answer: To avoid endless retries and runaway cost; fall back safely instead

Every retry costs money and time; cap them and have a safe default.

Lesson: Structured Output, Validation and Repair Loops

13. Fine-tuning is best suited to which need?

  1. Adding facts that change daily
  2. Consistent style, format or a narrow skill
  3. Avoiding any evaluation
  4. Replacing access control
Show answer

Answer: Consistent style, format or a narrow skill

Use retrieval for changing facts and fine-tuning for stable behaviour.

Lesson: Prompting, RAG or Fine-Tuning: A Decision Process

Latency, Cost and Capacity

14. What is the main risk of semantic caching?

  1. It deletes the prompt
  2. It makes embeddings unusable
  3. It always costs more
  4. Returning an answer to a similar but different question
Show answer

Answer: Returning an answer to a similar but different question

Use strict thresholds and limit it to safe, non-personalised categories.

Lesson: Caching: Exact, Normalised and Semantic

15. How should a cascade be evaluated?

  1. End to end, comparing quality and cost with a strong-model baseline, including per category
  2. Only by its cost
  3. Only by the cheap model's score
  4. It cannot be evaluated
Show answer

Answer: End to end, comparing quality and cost with a strong-model baseline, including per category

A cheap path that fails on hard categories can hide a serious quality loss.

Lesson: Model Routing and Cascades

16. Why track p95 and p99 rather than only the average?

  1. Averages are always wrong
  2. Percentiles are cheaper to compute
  3. Averages hide the slow requests that users notice
  4. Users ignore speed
Show answer

Answer: Averages hide the slow requests that users notice

The tail determines perceived reliability.

Lesson: Tail Latency, Timeouts and Hedged Requests

17. If average latency doubles at the same request rate, what happens to requests in flight?

  1. They stay the same
  2. They halve
  3. They double
  4. They become zero
Show answer

Answer: They double

In-flight requests = rate x latency, so they scale with latency.

Lesson: Capacity Planning With Little's Law

18. In the example, what dominates the cost per ticket?

  1. The human hand-off for unresolved tickets
  2. The token price
  3. The font size
  4. The server colour
Show answer

Answer: The human hand-off for unresolved tickets

Improving resolution rate can matter more than cheaper tokens.

Lesson: Unit Economics: Cost per Resolved Task

Reliability and Observability

19. What does a trace let you do that an aggregate metric does not?

  1. Hide failures
  2. Make the model smarter
  3. Remove the need for tests
  4. See where time and errors occurred inside one specific request
Show answer

Answer: See where time and errors occurred inside one specific request

Aggregates say that something is slow; traces show where.

Lesson: Tracing and Logging LLM Calls

20. A system's quality drops although no code changed. Which is a plausible cause?

  1. The code compiled differently
  2. The provider updated the model, or users now ask about new topics
  3. Python forgot how to run
  4. It cannot happen
Show answer

Answer: The provider updated the model, or users now ask about new topics

Inputs, data and the model can all change under you.

Lesson: Drift: When the World or the Model Changes

21. What is the safest response when retrieval returns nothing relevant?

  1. Invent a plausible answer
  2. Say the answer was not found, and offer a human
  3. Return an empty page
  4. Crash the app
Show answer

Answer: Say the answer was not found, and offer a human

An honest "not found" beats a confident guess.

Lesson: Failure Handling in Layers

Serving, Deployment and Release

22. What helps most when you might switch between models or providers later?

  1. Never reading release notes
  2. Hard-coding one provider's details everywhere
  3. Avoiding tests
  4. A thin abstraction over the model call and an evaluation set
Show answer

Answer: A thin abstraction over the model call and an evaluation set

Abstraction makes switching cheap; evaluation makes it safe.

Lesson: Hosted APIs vs Self-Hosting

23. What does quantisation mainly do?

  1. Reduces memory (and often cost) by storing weights in fewer bits, with some quality loss to measure
  2. Makes the model larger
  3. Removes the need for evaluation
  4. Encrypts the weights
Show answer

Answer: Reduces memory (and often cost) by storing weights in fewer bits, with some quality loss to measure

Always re-run your evaluation set on the quantised model.

Lesson: What Limits Serving Throughput

24. What does shadow mode do?

  1. Deletes the old version
  2. Shows the new version to everyone immediately
  3. Runs the new version on live requests without showing its output to users, for comparison
  4. Disables logging
Show answer

Answer: Runs the new version on live requests without showing its output to users, for comparison

It gives real-traffic evidence with no user risk.

Lesson: Staged Rollouts, Shadow Mode and Rollback

Safety, Governance and Team Practice

25. Which is part of an honest UX for an LLM feature?

  1. Hiding all uncertainty
  2. Pretending to be a human
  3. Telling users it is an AI, showing sources and offering a human
  4. Removing citations
Show answer

Answer: Telling users it is an AI, showing sources and offering a human

Transparency builds appropriate trust and reduces harm from wrong answers.

Lesson: Safety Essentials Every LLM Feature Needs

26. What makes deleting a person's data hard in LLM systems?

  1. Copies can exist in logs, caches, vector indexes and fine-tuning data
  2. Models forbid deletion
  3. Data is never stored
  4. It is easy everywhere
Show answer

Answer: Copies can exist in logs, caches, vector indexes and fine-tuning data

Map data flows so every copy can be found and removed.

Lesson: Privacy, Compliance and Data Handling

27. Why include "out of scope" uses in a system card?

  1. There is no reason
  2. To shorten the document
  3. Because law forbids listing uses
  4. To prevent misuse of a feature in situations it was not tested or designed for
Show answer

Answer: To prevent misuse of a feature in situations it was not tested or designed for

Clear limits set expectations and reduce harm.

Lesson: Documentation, Roles and Team Practice

Putting It Together

28. Why is the cascade checked end to end before adoption?

  1. Cascades never work
  2. Cost savings can hide quality loss in hard categories
  3. It is required by APIs
  4. To increase token use
Show answer

Answer: Cost savings can hide quality loss in hard categories

Always compare quality and cost with the strong-only baseline.

Lesson: Case Study: Taking a Support Assistant From Demo to Production

29. Prompt B passes 3 more cases than prompt A out of 40, and A passes 2 that B fails. Is B clearly better?

  1. Yes, always pick the higher score
  2. No: 5 vs 2 disagreements is within noise; collect more cases before deciding
  3. Yes, because B is newer
  4. Neither can be evaluated
Show answer

Answer: No: 5 vs 2 disagreements is within noise; collect more cases before deciding

Small gaps on small sets are indistinguishable from chance.

Lesson: Revision: Cheat Sheet and Self-Check

30. Your LLM judge agrees with humans 70% of the time, but 70% of the labels are "good". What should you check?

  1. Whether the judge is longer than the rubric
  2. Nothing, 70% is great
  3. The font of the prompt
  4. Cohen's kappa, since raw agreement is inflated by the label imbalance
Show answer

Answer: Cohen's kappa, since raw agreement is inflated by the label imbalance

A judge that always says "good" would also score 70% raw agreement.

Lesson: Revision: Cheat Sheet and Self-Check

31. Requests are 40 per second and each takes 5 seconds on average. About how many are in flight at once?

  1. 200
  2. 8
  3. 45
  4. 0.125
Show answer

Answer: 200

In-flight = rate x latency = 40 x 5.

Lesson: Revision: Cheat Sheet and Self-Check

← Back to the course