Yashveer Singh
Connect
<- All posts
Backend, APIs, and System Design11 min read

Eventual Consistency in Plain Language for Founders

Eventual consistency means the state of the system reaches a consistent value some time after the change. The window where state is inconsistent is small in well designed systems and meaningful in poorly designed ones. Almost every modern web application has eventual consistency somewhere. The teams that handle it well design for the consistency window. The teams that pretend it does not exist ship bugs that confuse customers.

Written by Yashveer Singh, founder of Yashveer Labs.

What you actually need to know

  • Almost every modern web app has eventual consistency somewhere.
  • Read from primary for the user's own data immediately after writes.
  • Short windows for low stakes data is acceptable.
  • Long windows or high stakes data is a bug.
  • The discipline is to know where the window matters.
Data typeConsistency requirement
User's own settings after changeRead your writes
Public counts and aggregatesEventual is fine
Payment confirmationStrong
Permission changesRead your writes
Cached search resultsEventual is fine
Audit log queriesEventual is acceptable
Authentication tokensStrong
Display name in a listEventual is fine

The core argument

Eventual consistency is the kind of distributed systems topic that sounds academic and matters in production. Almost every modern web application has some form of eventual consistency. Read replicas. Cached views. Asynchronous updates. Search indexes that lag behind the database. Each is a place where the system is consistent at some point after the change, not immediately at the change.

Founders need to understand the concept because it shapes product decisions. The customer who updated their plan should see the new plan immediately. The customer browsing a list of public projects can tolerate a count that lags by a few seconds. The product manager who knows the distinction designs flows that match. The product manager who does not design for the window ships bugs that confuse customers.

The engineering discipline is mechanical. The user's own data is read from the primary or from a cache that was updated by the same write. The public aggregates can be eventually consistent because the user has no expectation of seeing their specific change. The high stakes operations like payments use strong consistency. The low stakes operations can use eventual.

The teams that get this right ship products that feel consistent even though the underlying systems are not. The teams that get it wrong ship products with subtle bugs that customers report as confusion. The fix in either case is to design for the window deliberately.

The patterns to use

PatternUse
Read your writesUser sees their own changes immediately
Sticky sessionsUser stays on the same primary for a period
Synchronous replica catch upWait for replica before responding
Write through cacheUpdate the cache as part of the write
Optimistic UIUpdate the UI before server confirms
Polling with TTLAccept the window for non critical data
Server sent eventsPush updates when state changes

How much does this cost

The cost is engineering discipline. A few hours per feature to think about consistency. The infrastructure cost depends on the choice. Read your writes through primary routing is free. Synchronous replica catch up has a latency cost. Strong consistency on every operation has a scale cost.

Features the consistency design must have

  • A documented consistency model per surface.
  • Read your writes for the user's own data.
  • Optimistic UI where appropriate.
  • Cache invalidation that matches the model.
  • Monitoring on replica lag.
  • A clear policy on what is strong and what is eventual.
  • Tests that exercise the consistency window.
  • Documentation for engineers joining the team.

Expert opinion

Eventual consistency is one of those topics that engineers learn the hard way through bugs. The fix is to recognize where the window matters and where it does not, and design for both deliberately. The product manager who understands this designs flows that feel consistent. The engineer who understands this writes code that handles the window correctly. The discipline is small once it is built into the team's thinking.

>

Yashveer Singh, founder of Yashveer Labs

How this played out on a real project

A client SaaS was getting customer complaints that their plan upgrade had not gone through. The customer would pay, get a confirmation, and then see the old plan features for the next minute. The pattern was confusing customers and producing support tickets.

The investigation showed the application was reading the customer's plan from a read replica. The replica lagged the primary by a few seconds. The customer refreshed during the lag and saw the old plan.

We changed the read after write to the primary for the customer's own data. The bug disappeared. The customer experience matched the customer's mental model. The fix was hours of engineering work. The savings was the support load and the customer trust.

For more on the related work, see ACID vs BASE when each belongs in your architecture and read replicas when they save you and when they lie to you.

Common mistakes teams make

  1. Treating the read replica as strongly consistent.
  2. No read your writes pattern for the user's own data.
  3. Cache invalidation that does not match the consistency model.
  4. Strong consistency demanded everywhere. Scales poorly.
  5. Eventual consistency for high stakes operations like payments.
  6. No monitoring on replica lag.
  7. No documentation of the consistency model.
  8. Treating the consistency window as an academic concept.

A 30 day audit and fix

  1. Week one. Identify where the system reads from replicas or caches.
  2. Week two. Classify each surface. Read your writes, eventual, or strong required.
  3. Week three. Fix the surfaces that have the wrong model.
  4. Week four. Document the model. Add tests.

For more on the related work, read ACID vs BASE when each belongs in your architecture and read replicas when they save you and when they lie to you. On the broader architecture side, the hidden cost of eventual consistency a SaaS postmortem is the natural next read.

FAQ

Frequently asked

Author

Why this work lands with me

I am Yashveer Singh. Founder of Yashveer Labs. I take this kind of project because I have done enough of them to know what kills them. The version of me that writes a post like this is the same one who builds the system afterward. There is no handoff to a junior, no agency middleman, no surprise scope. That is the bet I am making on my own brand.

Related reading