Martin Kleppmann’s Designing Data-Intensive Applications (O’Reilly, 2017) is widely regarded as one of the most important technical books of the past decade. It sits at the intersection of databases, distributed systems, and software architecture — three areas that every backend engineer eventually needs to understand deeply.

This review covers the book’s main ideas, explains why they matter, and highlights the lessons most directly applicable to production engineering work.

Part of the Top 5 Books Every Software Engineer Should Read series.

What the Book Is About

The central premise is that modern applications are data-intensive rather than compute-intensive. The limiting factor is not raw CPU speed but the volume, complexity, and rate of change of the data itself.

The book is organized into three parts:

  1. Foundations of Data Systems — reliable storage, data models, query languages, storage engines
  2. Distributed Data — replication, partitioning, transactions, consistency
  3. Derived Data — batch processing, stream processing, combining systems

Part One: Foundations

Data Models and Query Languages

Kleppmann opens with a tour of data models: relational, document, graph, and others. Rather than declaring a winner, the book frames each model in terms of the relationships it handles well.

Relational databases excel at many-to-many relationships and ad-hoc querying. Document databases fit hierarchical data with known access patterns. Graph databases handle highly connected data where relationships are first-class entities.

The practical lesson for engineers is to understand the access patterns of the data before selecting a storage model. Most production systems end up using multiple models for different parts of the domain.

Storage Engines

The book explains how databases store and retrieve data by contrasting two fundamentally different approaches:

  • Log-structured storage (LSM trees, as used in LevelDB, Cassandra, RocksDB): writes are fast because data is appended sequentially; reads require merging multiple files.
  • Page-oriented storage (B-trees, as used in PostgreSQL, MySQL): reads are efficient because data is organized in fixed-size pages; writes update pages in place.

Understanding this distinction explains why different databases have different performance characteristics. A write-heavy time-series workload behaves very differently on an LSM-backed engine than on a B-tree engine.

Part Two: Distributed Data

This is where the book delivers its highest value. The distributed data section explains the real mechanics of the systems that power large-scale applications.

Replication

Replication — maintaining copies of data on multiple machines — serves three purposes: tolerating node failures, serving reads from multiple geographic locations, and improving read throughput.

The book explains three primary replication topologies:

Single-leader replication All writes go through one node (the leader). Replicas receive a stream of changes. This is the model used by PostgreSQL, MySQL, and MongoDB in their default configurations. It is simple but introduces a single point of failure for writes.

Multi-leader replication Multiple nodes can accept writes. Used in multi-datacenter deployments and collaborative applications. The challenge is write conflicts — two leaders may accept conflicting writes to the same record. Conflict resolution strategies (last-write-wins, application-defined merging) have real correctness implications.

Leaderless replication Any replica can accept writes; reads query multiple replicas and take the most recent value. Used in Cassandra and Dynamo-style systems. Consistency is managed through quorums: if a write succeeds on w of n replicas, and a read queries r replicas, then w + r > n guarantees overlap.

The deeper lesson is that replication lag is unavoidable in asynchronous replication. Applications must be designed to tolerate — or compensate for — eventual consistency.

Partitioning

Partitioning (also called sharding) divides a large dataset across multiple nodes. The goal is to distribute both data volume and query load evenly.

Two common strategies:

Range partitioning Keys are sorted, and each partition owns a contiguous range. Efficient for range queries but susceptible to hot spots if writes concentrate in one key range.

Hash partitioning Keys are hashed, and the hash determines the partition. Even distribution but no efficient range queries.

Secondary indexes add complexity. A local index (partition index) can be maintained per partition but requires scatter-gather queries. A global index offers efficient queries but updates become distributed operations.

Transactions

The ACID acronym is familiar, but Kleppmann unpacks what each property actually means in practice:

  • Atomicity: all operations in a transaction succeed or all are rolled back — not about concurrency.
  • Consistency: a property of the application, not the database.
  • Isolation: concurrently executing transactions appear as if they ran serially — the hardest property to implement efficiently.
  • Durability: committed data survives node failures.

Isolation deserves particular attention. The book catalogs the read phenomena that arise at weak isolation levels:

  • Dirty reads: reading data written by an uncommitted transaction
  • Non-repeatable reads: the same read returns different values within one transaction
  • Phantom reads: a re-executed query returns additional rows that weren’t there before

Serializable isolation prevents all of these but carries a performance cost. Most databases default to weaker isolation levels (read committed, repeatable read) that are faster but require applications to handle edge cases explicitly.

Consistency and Consensus

Distributed systems introduce a class of problems that do not exist in single-node systems.

The CAP theorem (Consistency, Availability, Partition tolerance) is often misapplied. Kleppmann is precise: during a network partition, a system must choose between consistency (returning accurate results or an error) and availability (returning potentially stale results). Systems operating without network partitions can offer both — the tension only manifests during failure.

The linearizability property — the strongest consistency guarantee — means the system behaves as if there is only one copy of the data, and every operation appears instantaneous. Linearizability is expensive to implement across distributed nodes.

Consensus — getting multiple nodes to agree on a single value — is required for leader election and atomic commits. The book explains Paxos and Raft conceptually. In practice, most engineers do not implement consensus protocols directly; they rely on coordination services like ZooKeeper or etcd, or databases with built-in consensus.

Part Three: Derived Data

The final section covers batch processing (MapReduce and its successors) and stream processing (Kafka, Flink, Spark Streaming).

The key insight is that complex data systems are often composed of multiple storage and processing components — a primary database, a search index, a cache, a data warehouse, and a stream processor. These components stay in sync through data pipelines.

Kleppmann frames this as a choice between two approaches:

  • Synchronous writes to multiple systems: higher consistency, higher coupling, reduced availability.
  • Event logs as the source of truth: one system of record emits events; other systems derive their state from the log. This is the CDC (Change Data Capture) pattern.

The event log approach maps closely to event sourcing architectures and has strong production advantages: the log can replay events to rebuild derived views, makes data flow auditable, and decouples producers from consumers.

Key Mental Models

The book provides several durable ways of thinking about distributed systems:

Systems are defined by their guarantees, not their names “Eventual consistency” means different things in different systems. Ask what the actual failure modes are, not just what the marketing page says.

Partial failures are the normal case A distributed system must assume that some nodes are slow, some messages are lost, and some clocks are wrong — simultaneously. Design for graceful degradation.

The data model is an architectural decision Changing the data model in a distributed system is hard. Choose it with the same care as the API contract.

Who Should Read It

Designing Data-Intensive Applications is appropriate for:

  • Backend engineers working with databases at any depth
  • Architects designing data-heavy systems
  • Senior engineers building or evaluating distributed data infrastructure
  • Any engineer who wants to move from “which database should I use?” to “what guarantees do I need and what are the trade-offs?”

A basic familiarity with relational databases and HTTP services is sufficient. No advanced mathematics is required.

Further Reading

After completing DDIA, the following resources extend specific topics:

  • Database Internals by Alex Petrov — goes deeper into storage engine implementation
  • Kafka: The Definitive Guide — extends the streaming concepts from Part Three
  • The original Raft and Paxos papers — for engineers curious about the consensus protocols referenced in the book

Summary

Designing Data-Intensive Applications teaches the mental models needed to reason about modern distributed systems. Its value is not in memorizing every algorithm but in developing a richer vocabulary for asking better architecture questions.

The engineer who has read this book approaches data architecture differently: not with a preferred technology, but with a set of well-formed questions about consistency, availability, failure modes, and operational cost.

Related posts in this series: