The Write/Read Boundary: A Decision Framework for Derived Data

The Write/Read Boundary: Mechanics of Derived Data

Section 1: Shifting the Workload

In the architecture of data-intensive systems, the fundamental challenge is not merely storing data, but managing the cost of accessing it. The primary lever for this management is the write/read boundary. As Martin Kleppmann establishes in Designing Data-Intensive Applications, "Viewed like this, the role of caches, indexes, and materialized views is simple: they shift the boundary between the read path and the write path." This shift allows engineers to "do more work on the write path, by precomputing results, in order to save effort on the read path."

The decision of where to draw this line is not arbitrary; it is a response to the specific load characteristics of the application. Without an index, "a search query would have to scan over all documents (like grep), which would get very expensive if you had a large number of documents." Conversely, "No index means less work on the write path (no index to update), but a lot more work on the read path." The introduction of derived data—whether an inverted index, a cache, or a materialized view—is an explicit trade-off: we accept the cost of maintaining a redundant data structure to avoid the cost of computing the answer from scratch every time a user asks.

The Mechanics of Fan-Out: Push vs. Pull

The Twitter feed example serves as the canonical illustration of this trade-off, specifically regarding fan-out. In a social network, a single write (a tweet) must be made visible to potentially millions of readers (followers). There are two primary architectural patterns for this, often described as "fan-out on write" and "fan-out on read."

Fan-Out on Write (The Push Model)

In the "fan-out on write" approach, the system eagerly distributes content to all consumers at the moment of writing. The source text describes this as: "Maintain a cache for each user's home timeline—like a mailbox of tweets for each recipient user... When a user posts a tweet, look up all the people who follow that user, and insert the new tweet into each of their home timeline caches."

The advantage is immediate and predictable read performance. "The request to read the home timeline is then cheap, because its result has been computed ahead of time." This aligns with the principle of "precomputing results" to minimize latency for the reader. However, the cost is borne entirely by the writer. As the text notes, "posting a tweet now requires a lot of extra work."

The mechanics of this cost are quantifiable. With an average of 75 followers per user, "4.6k tweets per second become 345k writes per second to the home timeline caches." The write amplification is significant. In a system designed by kindatechnical.com, the average write amplification for a standard user with 200 followers is calculated as a "200x" multiplier: "Total cache writes/sec = W * F_avg = 4,600 * 200 = 920,000 writes/sec."

Fan-Out on Read (The Pull Model)

The alternative is "fan-out on read," where no pre-computation happens. Here, the system "look[s] up all the people they follow, find all the tweets for each of those users, and merge them (sorted by time)" only when the user requests their timeline. The source describes this as: "When a user requests their home timeline, look up all the people they follow, find all the tweets for each of those users, and merge them."

This approach inverts the cost structure. The write path is "fairly easy," requiring only a single insert into a global collection. "Simply handling 12,000 writes per second (the peak rate for posting tweets) would be fairly easy." However, the read path becomes the bottleneck. The system must perform a complex join operation for every timeline request, which is "very expensive" when the number of followees is large.

The Decision Factor: Data Skew

The choice between these two models is rarely binary; it is dictated by the distribution of the workload. The primary decision factor is data skew. As the text explains, "the number of followers per user varies wildly, and some users have over 30 million followers."

In a "fan-out on write" system, this skew is catastrophic. For a user with 30 million followers, "a single tweet may result in over 30 million writes to home timelines!" The source highlights the operational impossibility: "Doing this in a timely manner—Twitter tries to deliver tweets to followers within five seconds—is a significant challenge." The write amplification for a celebrity becomes unsustainable; kindatechnical.com calculates that a celebrity with 10 million followers generates "46 million writes/sec," requiring "460 nodes" just to handle that single user's fan-out, rendering the pure push model "unsustainable and wasteful."

Conversely, "fan-out on read" handles skew gracefully. A celebrity's tweet is just one more entry in the global store; the cost of merging it into a follower's timeline is the same whether the tweeter has 10 followers or 10 million. The cost is linear to the number of readers, not the followees of the author.

The Hybrid Reality and the "Celebrity" Example

The ultimate solution for systems facing extreme load skew is a hybrid approach that adapts the write/read boundary based on the user's position in the distribution. The source text explicitly states that "Twitter is moving to a hybrid of both approaches." For the vast majority of users, the system maintains the eager precomputation strategy: "tweets continue to be fanned out to home timelines at the time when they are posted." However, for the extreme tail of the distribution, the strategy shifts back to on-demand computation: "a small number of users with a very large number of followers (i.e., celebrities) are excepted from this fan-out."

For these excluded users, the system reverts to the mechanics of "approach 1": "Tweets from any celebrities that a user may follow are fetched separately and merged with that user's home timeline when it is read." This hybrid strategy ensures "consistently good performance" by avoiding the write amplification storm that would occur if the system attempted to push a single tweet to 30 million caches, while still preserving low-latency reads for the 99.9% of interactions involving ordinary users.

It is essential to distinguish between the mechanics described in the text and the industry shorthand often applied to them. While the text details the handling of "users with a very large number of followers (i.e., celebrities)" and the resulting need to "fetch separately and merge" their content at read time, the specific term "celebrity user" does not appear as a formal, capitalized design pattern definition in Designing Data-Intensive Applications. The text treats "celebrity" as a descriptive label for a statistical outlier—a user whose follower count creates a load parameter that violates the assumptions of a uniform fan-out-on-write strategy. The mechanics of this skew, where the distribution of followers determines the load, are explicitly held in the evidence, but the shorthand "celebrity vs. ordinary" is a colloquialism derived from the example rather than a distinct term defined by the author. The principle remains: when the distribution of load is skewed, a uniform strategy fails, and a hybrid or adaptive strategy becomes necessary to balance write amplification against read latency.

Consistency and Latency Trade-offs

The choice of boundary also dictates consistency guarantees. "Fan-out on write" typically introduces a slight delay, as the system must asynchronously propagate the update to all caches. The source notes that Twitter "tries to deliver tweets to followers within five seconds," implying a window of inconsistency where a follower might not see a tweet immediately after it is posted. The derived data is eventually consistent.

In contrast, "fan-out on read" offers stronger freshness guarantees. Because the timeline is assembled from the source of truth at the moment of the request, "the result has been computed ahead of time" is replaced by "the system assembles the feed dynamically." This means the user sees the most current state, but at the cost of higher latency during the read operation.

The decision process, therefore, is not just about speed, but about the nature of the data and the expectations of the users. If the data is write-heavy and read-sparse, or if the read latency must be near-zero, precomputation (fan-out on write) is preferred. If the data is read-heavy with skewed access patterns, or if the cost of maintaining the derived state is prohibitive, dynamic assembly (fan-out on read) is the superior choice. As the text concludes, "Viewed like this, the role of caches, indexes, and materialized views is simple: they shift the boundary between the read path and the write path." The engineer's task is to find the point where the cost of that shift is minimized for the specific load profile of the system.