AI Engineering: Building Applications with Foundation Models (O’Reilly, 2025) by Chip Huyen addresses a gap that has become increasingly important as large language models have moved from research into production: the engineering discipline of building applications on top of AI models, as distinct from training or researching those models.

Huyen brings a practical orientation grounded in her experience at NVIDIA and Clarifai and her widely-read writing on machine learning systems. The result is a book that takes software engineers seriously as the primary audience, rather than treating AI application development as a subset of ML research.

Part of the Top 5 Books Every Software Engineer Should Read series.

What the Book Is About

The opening framing is precise and useful:

“AI engineering refers to the process of building applications on top of foundation models.”

This distinguishes the book’s scope from both classic machine learning (which focuses on training models from scratch) and from software engineering broadly. The target audience is engineers who need to understand how to integrate, evaluate, and operate AI systems — not how to train transformers.

Foundation models — large pretrained models capable of a wide range of tasks — have changed what AI application development looks like. Instead of assembling datasets and training task-specific models, engineers can now adapt a general-purpose model to a specific application through prompting, retrieval, or fine-tuning.

The book is organized around the engineering challenges that arise at each stage of this process.

Core Concepts

1. Foundation Models and the AI Stack

The book begins with a conceptual map of the foundation model landscape: what foundation models are, how they differ from earlier ML systems, and why they change the engineering discipline.

Foundation models are trained on large, broad datasets and can be adapted to many tasks. This has practical consequences:

  • The cost of training is externalized to model providers (OpenAI, Anthropic, Google, open-source alternatives)
  • Application teams work primarily with pre-trained weights and adapt them through prompting or fine-tuning
  • Model capabilities and limitations are probabilistic rather than deterministic, which changes how engineers test and evaluate systems

The shift from deterministic to probabilistic outputs is one of the most significant engineering challenges the book addresses. Traditional software testing validates exact outputs. AI systems produce outputs that vary — and must be evaluated against criteria like relevance, accuracy, and safety rather than simple equality checks.

2. Prompt Engineering

Prompt engineering is the practice of designing the instructions and context provided to a model in order to improve the quality and consistency of its outputs.

Huyen covers this systematically. The key concepts include:

Zero-shot prompting Asking the model to perform a task without examples. Works well for tasks within the model’s training distribution.

Few-shot prompting Providing examples in the prompt. Substantially improves performance on tasks where the desired format or approach is not obvious from the task description alone.

Chain-of-thought prompting Instructing the model to show its reasoning before producing an answer. Measurably improves performance on multi-step reasoning tasks.

System prompts Instructions that establish context, persona, and constraints at the start of a conversation. Critical for production applications that require consistent behavior across many interactions.

The book makes an important observation: prompt engineering is not a permanent solution. Techniques that work well with one model version may not transfer to another. Teams that rely heavily on prompt engineering should plan for re-evaluation each time a model is updated.

3. Retrieval-Augmented Generation (RAG)

RAG is one of the most important architectural patterns for production AI applications. It solves the problem that foundation models have a fixed training cutoff and no access to private or proprietary data.

The architecture has three stages:

  1. Indexing: source documents are chunked, converted to vector embeddings, and stored in a vector database
  2. Retrieval: at inference time, the query is embedded and semantically similar chunks are retrieved from the index
  3. Generation: the retrieved context and the query are combined in a prompt, and the model generates a response grounded in the retrieved information

RAG addresses several common failure modes of pure prompting:

  • Hallucination: the model fabricates facts not in its training data. RAG provides a grounding context.
  • Staleness: the model’s knowledge is frozen at training time. RAG allows current information to be provided at inference time.
  • Privacy: proprietary data cannot be included in training. RAG enables the model to access it at inference time without sharing it with a model provider.

The book covers practical considerations: chunking strategies, embedding model selection, retrieval evaluation, hybrid search (keyword + semantic), and the problem of context window management — not all retrieved chunks will fit in the context window, so ranking and selection strategies matter.

4. Fine-Tuning

Fine-tuning adapts a pretrained model to a specific task using additional training data. It is appropriate when:

  • Prompt engineering cannot achieve the required output quality or consistency
  • The task requires domain-specific vocabulary or reasoning not well-represented in the base model’s training
  • Latency or cost constraints require a smaller, more specialized model

The book distinguishes between:

Full fine-tuning: updating all model weights. Maximally flexible but computationally expensive.

Parameter-efficient fine-tuning (PEFT): techniques such as LoRA (Low-Rank Adaptation) that update only a small fraction of weights. Far less expensive while retaining most of the performance gains.

Instruction fine-tuning: training the model to follow instructions in a particular style or for a particular domain.

An important caution: fine-tuning does not reliably improve factual accuracy. If the goal is to make the model aware of new facts, RAG is usually the better choice. Fine-tuning is better suited to changing the model’s style, format, or behavior.

5. Evaluation

This is where the book makes its most distinctive engineering contribution. Evaluation is identified as the hardest problem in AI application development — not technically, but methodologically.

The challenge is that AI systems produce natural language outputs. Determining whether an output is “correct” or “good” requires criteria that are often subjective, task-specific, and difficult to define programmatically.

The book covers several evaluation strategies:

Human evaluation Direct human judgment of output quality. Most accurate but expensive and slow. Appropriate for establishing ground truth and calibrating automated methods.

Reference-based evaluation Comparing model outputs to known-good reference answers. Effective when reference answers can be obtained, but the space of correct answers is often large.

LLM-as-judge Using another language model to evaluate outputs. Fast and scalable. The book addresses the limitations: LLM judges can be biased toward their own outputs, verbose responses, and confident-sounding text regardless of correctness.

Task-specific metrics For structured outputs — code, classification, information extraction — standard metrics (pass rate, precision, recall, F1) apply directly.

The key insight is that evaluation must be designed alongside the system, not added at the end. Teams that defer evaluation to the end of development often discover fundamental quality issues that require architectural changes.

6. Production AI Systems

The final sections address operating AI applications in production. The challenges are different from those of traditional software:

Latency and cost Foundation model inference is expensive relative to a database query. Caching, batch processing, smaller model selection, and quantization are all levers.

Observability Traditional monitoring tracks metrics and logs. AI systems also require output monitoring: detecting hallucinations, off-topic responses, and safety violations in production traffic. This often requires building or adopting LLM-based evaluation pipelines that run on real user outputs.

Model updates Model providers release new versions frequently. A prompt that works well with one model version may perform differently with the next. Teams need evaluation pipelines that can run before and after model version changes.

Safety and content moderation Production AI applications require guardrails: input classification to detect misuse, output filtering for harmful content, and rate limiting for abuse prevention.

The Engineering Shift AI Requires

One of the book’s recurring themes is the nature of AI uncertainty and how it differs from traditional software uncertainty.

Traditional software behaves deterministically:

input → deterministic logic → predictable output

AI applications introduce probabilistic behavior at the core:

input → retrieval → context → model → probabilistic output → evaluation → monitoring

This changes the engineering practice in several ways:

  • Testing shifts from unit tests with expected outputs to evaluation suites with quality criteria
  • Debugging becomes statistical — “this class of inputs degrades quality by 15%” rather than “this function throws an error”
  • Deployment must include evaluation of the new version against the current version before full rollout

These are not problems unique to AI; probabilistic systems in other domains (recommendation engines, search ranking) face the same challenges. But the ubiquity of LLM-based applications means more engineers are encountering them for the first time.

Who Should Read It

AI Engineering is well-suited for:

  • Software engineers building applications that use LLM APIs
  • Backend engineers designing AI-powered features (search, summarization, generation, classification)
  • Architects evaluating how to integrate AI into existing system designs
  • Technical leads setting standards for how teams evaluate and monitor AI features

ML researchers or engineers primarily working on model training will find the book less directly relevant to their work, as the focus is explicitly on application engineering rather than model development.

Caveats and Limitations

The AI engineering space evolves quickly. Some specific techniques and tools discussed in the book — particularly around the RAG and fine-tuning sections — have continued to develop since publication. The principles remain valid, but specific implementation details may benefit from cross-reference with current documentation.

The book is also deliberately language and framework agnostic, which is appropriate for a principles-focused text but means engineers will need to look elsewhere for framework-specific implementation guidance (LangChain, LlamaIndex, Semantic Kernel, and similar tools all have their own documentation).

Summary

AI Engineering fills a genuine gap in the technical literature. It provides software engineers with the vocabulary, mental models, and architectural patterns needed to build AI-powered applications thoughtfully — with appropriate attention to evaluation, reliability, and production operation.

The field is moving quickly, but the engineering challenges Huyen identifies — evaluation, observability, consistency, latency, and safety — are not going away. Engineers who develop a clear-eyed understanding of them now will build better systems regardless of which models or frameworks are dominant next year.

Related posts in this series: