When you migrate a critical data dependency to a new system, how do you know the new system is telling the truth? We recently went through this in Coop's Identity Provider, and this post covers the technique we used to check consistency between the old and new systems during the switchover, and what we learned from it.
Two sources, one truth
The Identity Provider authenticates millions of Coop customers. Most customer data reaches us as events, but sometimes we need to fetch up-to-date information about a customer on the fly. For years we relied on a single well-established system for this. We were introducing a newer system meant to replace it eventually.
Both systems receive updates, but not always at the same time, in the same order, or through the same data model. Before trusting the new system as our primary source, we needed proof that its data actually matched what the legacy system reported. Switching over and hoping for it to work out wasn't an option: if the new system returned stale or incorrect data at the wrong moment, a customer could end up unable to log in.
The facade
Our solution is a Facade sitting in front of both systems. All lookup
logic goes through it instead of calling either system directly.
The facade does two things: it decides which system to call for production traffic, and it silently checks whether the two systems agree.
Routing works via a progressive rollout feature, letting traffic shift from the legacy system to the new one gradually rather than with a hard cutover. Sources are tried in order, and the first one that returns a result wins. If a source fails with a transient error, the facade falls back to the next one and logs a warning.
Comparing without slowing anything down
The comparison logic runs in a background goroutine, outside the hot path:
go f.compareNewWithLegacy(context.WithoutCancel(ctx), coopID, newSystemCustomer, err)
Two details matter here. We use context.WithoutCancel to create a
context that survives after the parent gets cancelled, which happens the moment the
response goes back to the caller. Without it, the goroutine would die almost
immediately. And we don't wait for the comparison to finish. It's fire-and-forget.
For each lookup, there are three possible outcomes:
- both systems report "not found" and agree
- one system finds a record while the other doesn't
- both systems return a record and we run a field-by-field comparison of the customer data
Each outcome gets recorded as a metric counter with tags describing the kind of match or mismatch it produced:
metrics.Incr("comparison.match", metrics.WithTag("exists", "false"))
metrics.Incr("comparison.match", metrics.WithTag("exists", "true"))
metrics.Incr("comparison.mismatch", metrics.WithTag("fault", "different-data-between-systems"))
We track three mismatch types: missing-in-new-system (the record
exists in the legacy system but not the new one),
missing-in-legacy-system (the reverse), and
different-data-between-systems (both systems have a record, but the
fields don't match). Missing records and data discrepancies point to different
fixes, so keeping them separate in the metrics made triage easier.
What we found
After running this for a month across all customer lookups, about 0.2% of calls reported a mismatch between the two systems.
0.2% sounds small until you remember these are identity lookups in a live login flow: a fraction of a percent is still real customers. It gave us a concrete bar to clear before trusting the new system as primary, rather than a vague sense that things seemed fine.
Most of the mismatches fell into the different-data-between-systems
bucket rather than missing records outright, which pointed us toward digging into
the actual cases rather than just the aggregate number.
Digging into the mismatches
0.2% isn't a number you just accept, so we looked at individual cases to understand what was happening.
Most mismatches turned out to be caused by a change landing in the new system first, with the legacy system simply not caught up yet. The new system wasn't wrong, it was ahead. For the smaller set of cases where the legacy system was ahead instead, we could reason about why: the new system is event-driven, and once it processes the relevant event it converges to the same state. Those mismatches are temporary by design.
That understanding, more than the raw percentage, is what gave us the confidence to cut over to the new system fully. The facade and its comparison logic turned that decision from a guess into something we could actually measure.
Why this worked
Several properties of the approach mattered in practice. The comparison is non-blocking, so it adds zero latency for the customer. Because it only writes metrics and never affects the response, a bug in the comparison code can't break production, which meant we could iterate on it freely. It also runs against real production traffic rather than synthetic test data, so the mismatch rate reflects the actual variety of customer records rather than whatever edge cases we thought to write tests for. Progressive rollout let us shift traffic gradually and check the mismatch rate at each step. We also handle transient errors separately: if either system has a network blip mid-comparison, we skip that comparison instead of logging a false mismatch, which keeps the signal clean.
Reusing this pattern elsewhere
Eventually consistent systems are a fact of life in distributed architectures: replication lag, transformations that lose precision, schema mappings that don't quite line up. The usual advice is to design for this: idempotent writes, event sourcing, reconciliation jobs. That's all good practice, but on its own it doesn't tell you how consistent your systems actually are right now.
What this pattern adds is a way to measure that, continuously, at production scale, instead of assuming two systems agree. Any time two systems are supposed to hold the same data - a cache and a database, a replica and a primary, a new service replacing an old one - you can wire up a non-blocking comparison like this and start getting real numbers instead of guesses. It's cheap to build and gives you a lot of insight.