Why schema drift goes undetected

Sun, 26 Jul 2026 00:00:00 GMT

Here is a scenario from a project in its third year of operation. Migrations are managed properly. CI is green. validate passes.

Then an error shows up in production only. The production table has a column that doesn’t exist anywhere else. Nobody remembers who added it, when, or why. There is no migration file for it.

The important part is not the mystery column. It’s that this divergence passed every check the team had. Nothing was broken. The tooling worked exactly as designed — and, as designed, it does not detect this class of problem.

This post is about why.

Code and databases are not symmetric

Git made deployment reproducible. Check out any commit and you get exactly the same tree, every time. If CI is green, the contents of that commit are settled.

Migrations brought the same idea to databases: record every change as a timestamped file, apply them in order, and every environment converges on the same schema.

There is an asymmetry hiding in that analogy, and it’s rarely stated outright.

Replaying history is deterministic for code. The repository’s state is fully determined by its history. If someone hand-edits a file on a server, the next deploy overwrites it. The working tree is disposable; the truth lives in the history.

Replaying history is not deterministic for databases. A database is a persistent entity you cannot throw away and rebuild from scratch. And it changes through paths that never touch your migration files:

An emergency ALTER TABLE run directly in a production console during an incident
Changes made by another team or another tool with write access — GUI clients, BI tools, monitoring agents, batch jobs
Implicit behavioral differences between engine versions: default collations, implicit type coercions, index length limits
An index added by hand in production because the data volume demanded it

None of these leave a trace in migration history.

A migration directory is an append-only log of intent. It records what someone meant to change. It is not a representation of what the database currently is.

For code, history is the truth. For a database, the truth is only in the database.

Two questions migrations cannot answer

This asymmetry has practical consequences. There are two questions a migration-based workflow structurally cannot answer.

1. What does the schema look like right now?

Migration files are a sequence of imperative steps, not a declaration. If you want to know the current definition of a table, you’d have to mentally fold hundreds of files in order. Nobody does that. By year three, everyone runs SHOW CREATE TABLE or checks \d in psql instead — because the live database is more trustworthy than the files.

And that’s the problem. The moment “the database is more reliable than the migration files” becomes shared team knowledge, the files stop functioning as documentation. Files nobody reads decay. Decayed files get read even less.

2. When two environments differ, which one is correct?

There is no arbiter. “Whatever you get by replaying all migrations against an empty database” sounds like an answer, but it’s circular — the whole problem is that this result and production disagree.

`validate` is not looking at your database

A common misconception is worth clearing up precisely.

Flyway’s validate checks that the checksums of applied migrations still match the files on disk. It answers the question “has anyone edited a migration file after it was applied?” It does not answer “does the actual schema match what the migrations describe?”

Run a manual ALTER in production and validate stays green. The files didn’t change, so from its point of view, nothing happened. This is correct behavior — for what the tool is scoped to do.

To be fair: tooling for the other question does exist. Liquibase ships diff and diffChangeLog in its open-source tier. Flyway offers drift detection in its paid editions. But the former assumes a changelog-centric CLI workflow with real setup cost, and the latter sits behind a license. The practical result across most teams is the same: the capability exists somewhere, and nobody runs it routinely. “Technically possible” and “actually happening every day” are very different states.

While we’re being honest about tooling: most codebases carry down migrations that have never been executed even once. Untested code doesn’t work. “We can roll back if we need to” is, in most shops, a comforting fiction that costs writing effort on every change.

The discipline problem

Everything so far was mechanical. The deeper problem is organizational.

A migration workflow looks like a technical mechanism, but what actually holds it together is a rule: every schema change goes through a migration file. No exceptions, no direct changes, everyone, always.

Rules like this have three properties that determine their fate:

They can be broken.
Things keep working when they’re broken.
Breaking them is not detected.

Any process with all three properties degrades. It’s only a question of when.

And this particular rule has an extra cost: it must be transmitted to every future participant for the lifetime of the project. It doesn’t matter that the current team understands it perfectly. The engineer who joins in six months, the team that inherits the system in two years, the contractor called in for a 2 a.m. incident five years from now — the rule binds all of them, and any one of them can silently break it.

The higher your staff turnover, the faster the history rots.

This is the trade migrations quietly make: less friction today, paid for with a discipline debt assigned to people who weren’t in the room.

The assumption that all environments are identical

There’s one more assumption baked into migration workflows, and it’s worth dragging into the light: every environment should have the same schema. Apply the same files in the same order, get the same result. Any divergence is an accident.

Except some divergence is deliberate.

An index that exists only in production, because production has 400× the data. Locally it would just slow the tests down.
Partitioning applied only in production.
Read-optimized indexes that exist only on replicas.
Collation or encoding differences mid-migration.
Debug views and seed tables that exist only in staging.
Per-customer schema variations in on-premise or packaged deployments.
A foreign key constraint deliberately dropped in production for operational reasons.

Try to express these in migrations and you get conditional branches keyed on environment variables — which makes the files even harder to read, which means they get read even less.

So there is a category of real, legitimate schema state that migrations cannot express even in principle.

To be clear, none of this is an argument against migrations. Versioned migration history is the correct practice. It just has a smaller jurisdiction than we usually assume.

Only a human can classify a difference — and we’ve given humans nothing to work with

Once you accept that intentional drift exists, a task becomes unavoidable: when a difference between environments surfaces, someone has to decide — is this intentional, or is it an accident?

That decision cannot be automated. Intent lives in people’s heads. No tool can know whether an extra index is a performance fix someone deliberately applied to production, or something an engineer added during an investigation and forgot to remove.

I think it’s important to say this plainly, because it marks the hard boundary of every “fully automated schema management” pitch: classification requires human judgment, full stop.

The real problem is not that humans have to look. It’s that we’ve made looking nearly impossible.

Today your options are roughly:

Open two database clients side by side. Works for three tables. Does not work for eighty. This is a limitation of human attention, not effort.

Dump both schemas and diff the text.

pg_dump --schema-only -h host-a mydb > a.sql
pg_dump --schema-only -h host-b mydb > b.sql
diff a.sql b.sql

Reasonable in theory. In practice, most of what comes out is noise: object ordering differences, AUTO_INCREMENT counters, formatting variations in defaults, type spelling differences (serial vs. identity columns), ownership lines, version-dependent SET statements. You end up hunting for the one meaningful line buried in dozens of irrelevant ones — and humans who scan noisy output for long enough start missing the meaningful lines too.

That’s the structural gap: the judgment can only be made by a human, and the output was never designed for human judgment.

Here’s the part that makes it tractable. Intentional drift is stable. That production-only index shows up as the same difference every single time you compare. If the comparison output is structured and noise-free, stable differences become recognizable patterns — after the third look, it’s “ah, that one again,” and the real anomalies stand out against a familiar background.

People don’t miss differences because there are too many. They miss them because the same difference looks different every time.

Where this leaves us

Migration history records intent. It does not describe the present. The present exists only in the database itself.
The verification built into migration tools checks files, not databases. Nothing in the default workflow detects drift.
The workflow’s integrity rests on a convention that is breakable, silently breakable, and must be re-taught forever.
Some drift is intentional, which means classification is unavoidable, which means human eyes are unavoidable.
So the leverage point is not more automation of intent — it’s better material for judging results: a clean, structured, repeatable comparison of what your databases actually are.

Manage history with migrations. Verify the present with diffs. These are different jobs, and conflating them is how a column ends up in production that nobody can explain.

I build DiffyPick, a desktop tool for comparing and syncing database schemas — it exists because of everything above.

DiffyPick Blog — database-migration