Safe Rollout Plans for AI Features interview questions

25 questions from the Safe Rollout Plans for AI Features course, each with the correct answer and an explanation. Try to answer before you open the answer.

Why AI Rollouts Need Their Own Plan

1. Why is "it passed QA" not enough for an AI feature?

  1. QA cannot run AI code
  2. AI features never fail
  3. Outputs vary, so quality must be measured as a rate across many cases
  4. Tests make models worse
Show answer

Answer: Outputs vary, so quality must be measured as a rate across many cases

Measure distributions, not single passes.

Lesson: How AI Features Differ From Ordinary Features

2. Why write the rollout plan before launch?

  1. To agree on metrics, bars and rollback rules before launch pressure builds
  2. To avoid measuring anything
  3. Because launches cannot be paused
  4. To skip user communication
Show answer

Answer: To agree on metrics, bars and rollback rules before launch pressure builds

Decide the rules while calm.

Lesson: The Rollout Plan Document

3. Which is a guardrail metric for an AI feature?

  1. Office temperature
  2. Number of marketing emails sent
  3. Lines of code written
  4. Rate of safety flags per thousand responses
Show answer

Answer: Rate of safety flags per thousand responses

Guardrails measure harm and risk.

Lesson: Success Metrics and Guardrail Metrics

Pre-Launch Gates

4. Why set a floor per slice and not only an overall bar?

  1. Slices always score higher
  2. A good overall average can hide a slice that performs badly
  3. Overall rates cannot be computed
  4. It reduces evaluation cost to zero
Show answer

Answer: A good overall average can hide a slice that performs badly

Protect every important group.

Lesson: Offline Evaluation and the Launch Bar

5. Which red-team category should usually have a zero-tolerance limit?

  1. Slightly verbose answers
  2. Revealing another user's data
  3. Polite refusals
  4. Formatting preferences
Show answer

Answer: Revealing another user's data

Some harms are never acceptable.

Lesson: Red-Team and Safety Gates

6. Why compute cost per rollout stage before launching?

  1. It replaces evaluation
  2. Costs never change with usage
  3. Providers do not charge for tokens
  4. Costs that are small in testing can become very large at full scale
Show answer

Answer: Costs that are small in testing can become very large at full scale

Cost scales with exposure.

Lesson: Cost and Latency Budgets

Feature Flags and Targeting

7. Why hash the user id for percent rollouts instead of choosing randomly per request?

  1. Each user gets a consistent experience and raising the percentage only adds users
  2. Hashing makes the model faster
  3. Random choice is illegal
  4. Hashing reduces token cost
Show answer

Answer: Each user gets a consistent experience and raising the percentage only adds users

Stable buckets avoid flip-flopping.

Lesson: Feature Flags, Percent Rollouts and Kill Switches

8. Who usually sees an AI feature first?

  1. Only regulated customers
  2. All users at once
  3. Internal employees (dogfooding)
  4. Nobody until it is perfect
Show answer

Answer: Internal employees (dogfooding)

Start with people who can give fast feedback and tolerate rough edges.

Lesson: Targeting Cohorts

9. Why version prompts and model settings like code?

  1. Versioning makes models free
  2. Prompts never affect output
  3. They change behaviour, so they need review, evaluation and rollback
  4. Providers require it
Show answer

Answer: They change behaviour, so they need review, evaluation and rollback

Behavioural config deserves the same discipline as code.

Lesson: Prompts and Models as Versioned Configuration

Staged Exposure

10. What happens at the end of each ramp stage?

  1. A gate check against guardrail metrics before advancing
  2. Automatic jump to 100%
  3. The feature is deleted
  4. Metrics are reset without review
Show answer

Answer: A gate check against guardrail metrics before advancing

Advance only on evidence.

Lesson: A Ramp Schedule

11. What can shadow mode NOT tell you?

  1. Disagreement with the current system
  2. Latency on real traffic
  3. Cost per request
  4. How users react to seeing the AI output
Show answer

Answer: How users react to seeing the AI output

Users never see shadow results.

Lesson: Shadow Mode

12. A guardrail metric is significantly worse in the canary. What should happen?

  1. Advance to 100% quickly
  2. Stop the ramp and investigate
  3. Ignore it because success metrics matter more
  4. Delete the metric
Show answer

Answer: Stop the ramp and investigate

Guardrails are stop signs.

Lesson: Canary Comparisons

13. What happens to the required sample size as the effect you want to detect gets smaller?

  1. It shrinks
  2. It grows rapidly
  3. It stays the same
  4. It becomes zero
Show answer

Answer: It grows rapidly

Small effects need large samples.

Lesson: A/B Tests and Sample Size

Runtime Protection

14. What should happen when an output fails a schema or safety check?

  1. Retry forever
  2. Show it anyway
  3. Disable logging
  4. Regenerate or fall back rather than show it
Show answer

Answer: Regenerate or fall back rather than show it

Bounded retries, then fallback.

Lesson: Input and Output Checks

15. Why use a circuit breaker for model calls?

  1. To stop sending requests to a failing provider and serve a fallback quickly
  2. To make the model smarter
  3. To increase costs
  4. To skip timeouts entirely
Show answer

Answer: To stop sending requests to a failing provider and serve a fallback quickly

Fail fast, recover automatically.

Lesson: Fallbacks, Timeouts and Circuit Breakers

16. What should happen when an AI feature reaches its daily spend cap?

  1. Crash the app
  2. Charge users more automatically
  3. Serve the fallback experience and alert the owners
  4. Silently keep spending
Show answer

Answer: Serve the fallback experience and alert the owners

Degrade gracefully and tell someone.

Lesson: Rate Limits and Spend Caps

Monitoring and Rollback

17. Why break monitoring down by config version?

  1. Versions do not affect behaviour
  2. To hide problems
  3. To link regressions to specific prompt or model changes
  4. It reduces token usage
Show answer

Answer: To link regressions to specific prompt or model changes

Change tracking makes diagnosis fast.

Lesson: What to Monitor After Launch

18. Which breach should usually trigger an automatic kill switch?

  1. A spike in errors or safety flags
  2. A small dip in a success metric for one hour
  3. A new feature request
  4. A positive review
Show answer

Answer: A spike in errors or safety flags

Severe harm warrants automatic action.

Lesson: Automatic Rollback Triggers

19. Why include random samples, not only thumbs-down items, in review?

  1. It reduces privacy
  2. Random samples are always bad
  3. Thumbs-down items are always wrong
  4. Many problems are never reported by users
Show answer

Answer: Many problems are never reported by users

Complaints are a biased sample.

Lesson: Human Review of Sampled Outputs

Users, Support and Incidents

20. Why label AI-generated content in the product?

  1. To hide how it works
  2. So users know to check it and can calibrate their trust
  3. Labels improve model accuracy
  4. It removes the need for review
Show answer

Answer: So users know to check it and can calibrate their trust

Transparency reduces over-trust.

Lesson: Disclosure and Setting Expectations

21. What should support teams have before an AI launch stage?

  1. Nothing; they learn from complaints
  2. A briefing on limits, who has the feature, and how to escalate or disable it
  3. Access to all model weights
  4. A promise that nothing will fail
Show answer

Answer: A briefing on limits, who has the feature, and how to escalate or disable it

Prepared support catches problems early.

Lesson: Support Readiness and Feedback Loops

22. What is the first step when an AI feature produces harmful outputs at scale?

  1. Wait for the weekly meeting
  2. Start a long investigation while it keeps running
  3. Delete all logs
  4. Contain it with the kill switch or a config rollback
Show answer

Answer: Contain it with the kill switch or a config rollback

Stop the harm, then investigate.

Lesson: Incident Response for AI Features

Changes After Launch, a Case Study and a Checklist

23. Why can a model with a higher eval score still be blocked?

  1. Other guardrails, such as refusal rate or cost, got worse
  2. Higher scores are always fake
  3. New models cannot be used
  4. Eval scores do not matter
Show answer

Answer: Other guardrails, such as refusal rate or cost, got worse

Check every guardrail, not only quality.

Lesson: Model Upgrades and Prompt Changes

24. In the case study, how was the thumbs-down spike at 5% handled?

  1. The metric was removed
  2. The ramp jumped to 100%
  3. The config was rolled back to the previous prompt version and the ramp paused
  4. Users were told to stop giving feedback
Show answer

Answer: The config was rolled back to the previous prompt version and the ramp paused

Versioned config makes rollback quick.

Lesson: Case Study: AI Summaries for Support Agents

25. Which item belongs on an AI rollout checklist?

  1. Skipping slice-level evaluation
  2. Launching to 100% on day one
  3. A tested kill switch with a designed fallback experience
  4. Hiding that content is AI-generated
Show answer

Answer: A tested kill switch with a designed fallback experience

Control, measure, communicate.

Lesson: An AI Rollout Checklist

← Back to the course