SkillByAIOpen interactive version →

LLM Engineering Foundations

Turn LLM demos into dependable products: evaluation and statistics, prompt and model versioning, structured output, cost and latency engineering, reliability and observability, serving, governance and team practice, with every experiment run.

Start course →

Syllabus

From Demo to Product

  1. What LLM Engineering Is
  2. A Reference Architecture for an LLM Feature
  3. Defining Success: Requirements and Metrics
  4. From Prototype to Production: A Staged Path

Evaluation That You Can Trust

  1. An Evaluation Harness
  2. Noise and Confidence Intervals
  3. Comparing Two Versions Fairly
  4. LLM-as-Judge: Useful, Biased, Must Be Calibrated
  5. Test-Set Hygiene: Leakage and Contamination

Engineering for Quality

  1. Prompts, Models and Settings as Versioned Artefacts
  2. Release Gates and CI for LLM Changes
  3. Structured Output, Validation and Repair Loops
  4. Prompting, RAG or Fine-Tuning: A Decision Process

Latency, Cost and Capacity

  1. Caching: Exact, Normalised and Semantic
  2. Model Routing and Cascades
  3. Tail Latency, Timeouts and Hedged Requests
  4. Capacity Planning With Little's Law
  5. Unit Economics: Cost per Resolved Task

Reliability and Observability

  1. Tracing and Logging LLM Calls
  2. Drift: When the World or the Model Changes
  3. Failure Handling in Layers

Serving, Deployment and Release

  1. Hosted APIs vs Self-Hosting
  2. What Limits Serving Throughput
  3. Staged Rollouts, Shadow Mode and Rollback

Safety, Governance and Team Practice

  1. Safety Essentials Every LLM Feature Needs
  2. Privacy, Compliance and Data Handling
  3. Documentation, Roles and Team Practice

Putting It Together

  1. Case Study: Taking a Support Assistant From Demo to Production
  2. Revision: Cheat Sheet and Self-Check