AI Safety, Evaluation and Cost Control interview questions

27 questions from the AI Safety, Evaluation and Cost Control course, each with the correct answer and an explanation. Try to answer before you open the answer.

Safety Foundations

1. Who is responsible for how an LLM application behaves?

  1. Only the model provider
  2. Nobody
  3. The team that builds and deploys the application
  4. Only the end user
Show answer

Answer: The team that builds and deploys the application

Providers add safeguards, but the application owner controls context, data, tools and deployment.

Lesson: What AI Safety Means for Applications

2. A model confidently states a policy that does not exist. Which failure mode is this?

  1. Hallucination
  2. Rate limiting
  3. Caching
  4. Compression
Show answer

Answer: Hallucination

A fluent invented answer is the classic definition of hallucination.

Lesson: Common Failure Modes

3. Why stack several safeguards instead of relying on one?

  1. It removes the need for testing
  2. One safeguard is illegal
  3. Layers make the model faster
  4. Each has gaps, so layers catch what others miss
Show answer

Answer: Each has gaps, so layers catch what others miss

Independent layers reduce the chance that a single miss becomes harm.

Lesson: Safety by Design: Layers

Input and Output Safeguards

4. What happens to recall when you raise the moderation threshold?

  1. It always rises
  2. It usually falls, more harmful items are missed
  3. It becomes undefined
  4. It stays exactly the same
Show answer

Answer: It usually falls, more harmful items are missed

A stricter cut-off blocks fewer items overall, so some real harm slips under it.

Lesson: Moderation and Thresholds

5. When is the best time to mask personal data?

  1. Never
  2. Before it enters prompts, logs or analytics
  3. After a breach
  4. Only in the UI
Show answer

Answer: Before it enters prompts, logs or analytics

Data that never leaves your boundary in raw form cannot be leaked from downstream systems.

Lesson: Handling Personal Data

6. What is a good instruction for reducing hallucination in a document Q&A bot?

  1. Guess when unsure
  2. Always sound confident
  3. Never mention sources
  4. Answer only from the provided text and say if it is not there
Show answer

Answer: Answer only from the provided text and say if it is not there

Restricting the answer to supplied evidence and allowing "not found" removes the pressure to invent.

Lesson: Grounding and Hallucination Control

7. What is over-refusal?

  1. Declining a harmless request
  2. Answering a harmful request
  3. Refusing to start
  4. Running out of tokens
Show answer

Answer: Declining a harmless request

Over-refusal makes the product less useful and is measured alongside under-refusal.

Lesson: Refusal Calibration

Misuse and Abuse

8. Which defence against prompt injection is most reliable?

  1. Hiding the system prompt
  2. Only writing "be careful" in the prompt
  3. Giving the model no dangerous tools and requiring approval for sensitive actions
  4. Using longer user messages
Show answer

Answer: Giving the model no dangerous tools and requiring approval for sensitive actions

Capability limits still protect you when the model is fooled.

Lesson: Injection and Jailbreaks

9. What does a token bucket allow?

  1. Only one request ever
  2. Unlimited requests
  3. Short bursts with a steady average rate
  4. No requests at all
Show answer

Answer: Short bursts with a steady average rate

Capacity permits bursts; the refill rate bounds the long-run average.

Lesson: Rate Limiting and Abuse Control

Fairness, Privacy and Accountability

10. Why report accuracy by slice instead of only overall?

  1. An overall average can hide a group that is served badly
  2. Slices are always more accurate
  3. Overall numbers are illegal
  4. It saves compute
Show answer

Answer: An overall average can hide a group that is served badly

Gaps between groups only appear when results are broken down.

Lesson: Checking for Bias with Slices

11. What is data minimisation?

  1. Collecting everything just in case
  2. Compressing files
  3. Deleting the model
  4. Sending the model only the data the task needs
Show answer

Answer: Sending the model only the data the task needs

Less data collected and shared means less that can leak or be misused.

Lesson: Privacy and Data Governance

12. Who should make high-stakes decisions such as loan approval?

  1. The AI alone, always
  2. A human who can review and override the AI
  3. A random number generator
  4. Nobody
Show answer

Answer: A human who can review and override the AI

Significant decisions need accountable human oversight and a record of reasons.

Lesson: Documentation, Transparency and Human Oversight

Evaluation

13. Why keep some evaluation cases separate from tuning?

  1. To save disk space
  2. So the final score is not inflated by tuning to the same cases
  3. Because they are secret
  4. It makes tests slower
Show answer

Answer: So the final score is not inflated by tuning to the same cases

Tuning on the test cases leads to an optimistic score that will not hold in production.

Lesson: Building an Evaluation Set

14. What does recall measure?

  1. How many tokens were used
  2. Of flagged items, how many were truly positive
  3. How fast the model replies
  4. Of all truly positive items, how many were caught
Show answer

Answer: Of all truly positive items, how many were caught

Recall is TP / (TP + FN): the share of real positives the system finds.

Lesson: Metrics: Accuracy, Precision, Recall, F1

15. Which gives a tighter confidence interval for the same 90% score?

  1. 200 test cases
  2. 20 test cases
  3. Both are identical
  4. Neither has an interval
Show answer

Answer: 200 test cases

More data reduces uncertainty, narrowing the plausible range.

Lesson: How Sure Is Your Score?

16. What does Cohen's kappa measure?

  1. Cost per token
  2. Model speed
  3. Agreement between two raters beyond what chance would give
  4. Disk usage
Show answer

Answer: Agreement between two raters beyond what chance would give

Kappa corrects raw agreement for the agreement expected by chance.

Lesson: LLM Judges and Human Agreement

17. Why block a change when regression-check scores drop?

  1. Dropping scores are always noise
  2. Scores never matter
  3. The change may have made the system worse in ways you did not intend
  4. It speeds up deployment
Show answer

Answer: The change may have made the system worse in ways you did not intend

Automatic checks catch silent quality loss before users do.

Lesson: A/B Tests and Regression Checks

Monitoring in Production

18. Why record the prompt version in every trace?

  1. To link a change in behaviour to the exact version that caused it
  2. Because logs must be long
  3. To hide the prompt
  4. It reduces costs by itself
Show answer

Answer: To link a change in behaviour to the exact version that caused it

Version labels make regressions traceable to a specific change.

Lesson: What to Log and Watch

19. Why track p95 latency in addition to the average?

  1. It reduces the bill
  2. p95 is always smaller than the mean
  3. Averages are illegal
  4. The slow tail affects real users but barely moves the average
Show answer

Answer: The slow tail affects real users but barely moves the average

Percentiles expose the worst experiences that an average smooths over.

Lesson: Latency Percentiles and Alerts

Cost Control

20. Which usually drives LLM cost most in a chat product?

  1. The colour of the UI
  2. Large inputs repeated across many requests
  3. The font size
  4. The number of buttons
Show answer

Answer: Large inputs repeated across many requests

Long prompts and histories are paid for again on every call.

Lesson: Token Economics

21. What is the main risk of routing requests to a cheaper model?

  1. The bill always rises
  2. Quality may drop on requests the router misjudged as easy
  3. Small models cannot read text
  4. Routing is illegal
Show answer

Answer: Quality may drop on requests the router misjudged as easy

Savings depend on the router being right, so quality must be measured on the cheap path.

Lesson: Model Routing and Cascades

22. Where should the changing user question go for prompt caching to work well?

  1. Nowhere
  2. At the very start
  3. Inside the API key
  4. At the end, after the stable prefix
Show answer

Answer: At the end, after the stable prefix

Only an identical prefix can be reused, so volatile content belongs after it.

Lesson: Caching, Batching and Trimming

23. Why use separate API keys for development and production?

  1. So a test script cannot drain the production budget, and spend is attributable
  2. Keys must be unique by law
  3. It makes the model faster
  4. It removes the need for budgets
Show answer

Answer: So a test script cannot drain the production budget, and spend is attributable

Separate keys isolate environments and make costs traceable.

Lesson: Budgets, Quotas and FinOps for AI

Putting It Together

24. Why release to a small share of users first?

  1. Because small launches are free
  2. To avoid collecting feedback
  3. To limit the impact of problems your tests missed
  4. To skip evaluation
Show answer

Answer: To limit the impact of problems your tests missed

Staged rollouts catch real-world problems while few people are affected.

Lesson: Case Study: Launching a Support Assistant

25. A harmful-content filter is 99% accurate because only 1% of messages are harmful and it never flags anything. What is the problem?

  1. It is too fast
  2. Its precision is too high
  3. Its recall is zero for the rare harmful class
  4. Nothing is wrong
Show answer

Answer: Its recall is zero for the rare harmful class

Accuracy hides the failure on the rare class; recall exposes it.

Lesson: Revision: Cheat Sheet and Self-Check

26. Two versions score 90% and 88% on 25 test cases. What should you conclude?

  1. There is no real evidence of a difference; gather more cases
  2. Version A is clearly better
  3. Version B is clearly better
  4. Scores cannot be compared
Show answer

Answer: There is no real evidence of a difference; gather more cases

With so few cases the confidence intervals overlap heavily, so the gap is within noise.

Lesson: Revision: Cheat Sheet and Self-Check

27. Which change usually cuts LLM cost without lowering quality when done carefully?

  1. Disabling monitoring
  2. Removing the evaluation set
  3. Doubling the context length
  4. Routing easy requests to a smaller model after testing quality
Show answer

Answer: Routing easy requests to a smaller model after testing quality

Routing, caching and trimming reduce spend, and your evaluation set confirms quality holds.

Lesson: Revision: Cheat Sheet and Self-Check

← Back to the course