We removed a column. EU went offline.

ALTER TABLE events DROP COLUMN legacy_source;

Reviewed. Approved. Merged. The read had already moved to source_metadata JSONB. Tests passed on a laptop. The migration looked like cleanup.

The database migrates before the pods do. For about 90 seconds both are true at once: schema is v2, live traffic is still v1.

The overlap is guaranteed

Kubernetes + a migration job looks like this:

T+0s    migration pod: DROP COLUMN legacy_source
T+3s    migration exits 0. schema is v2
T+5s    rollout starts. old pods still serving
T+35s   first new pod ready
T+90s   last old pod gone

Between T+3s and T+90s, every request that still selected legacy_source got 42703 undefined_column. That is not a race. Rolling deploys always overlap.

The window was long enough that Kubernetes marked the old pods unready and killed them before the new ones were in the load balancer. EU had zero healthy replicas for about four minutes.

Why it took the region down

Postgres rejects a missing column at parse time. No I/O. Fast. Relentless. Every read on that path failed.

Failed queries still held pool slots: ORM exception path, retries, request timeouts. The pool filled. New work queued. The health check did a cheap DB ping. With no free connections, the ping timed out. Pods went Unready.

Kubernetes replaced them. The rollout had not finished, so replacements were still the old image. They came up, hit the same SELECT, died again.

The v2 pods, the ones that no longer read the column, were still starting.

What I actually looked at

5xx spike. Then pods:

kubectl get pods -n production -l app=events-service
# NAME                    READY   STATUS             RESTARTS   AGE
# events-service-abc12    0/1     CrashLoopBackOff   5          3m
# events-service-def34    0/1     CrashLoopBackOff   4          3m
# events-service-ghi56    1/1     Running            0          45s

Previous logs on a crashing pod:

2024-03-14T09:23:01Z ERROR column "legacy_source" does not exist
2024-03-14T09:23:01Z ERROR column "legacy_source" does not exist

Hundreds of identical lines. First one at 09:22:58Z. That was the second the migration pod exited.

The database was bored:

SELECT count(*), state, wait_event_type
FROM pg_stat_activity
WHERE datname = 'production'
GROUP BY state, wait_event_type
ORDER BY count DESC;
 
--  count |  state  | wait_event_type
--    147 | idle    | Client
--      3 | active  |

147 idle connections held by application threads that had already crashed. Postgres was fine. The app was stuck on a column that no longer existed.

Adding the column back is not undo

Instinct: ALTER TABLE events ADD COLUMN legacy_source TEXT.

The column comes back empty. Anything written only to legacy_source during the window is gone. If some other code still assumed values there, you trade one outage for nulls.

We had been dual-writing, so add-column was safe. Confirming that while the region was dark took longer than I wanted.

Two deploys, not one

Same change, split.

Deploy N only grows the schema. Add source_metadata. Write both columns. Read the new one. Old pods still have legacy_source. New pods have the JSONB. The overlap window is boring.

ALTER TABLE events ADD COLUMN source_metadata JSONB;

Deploy N+1 drops the old column, after every pod has already stopped reading it.

ALTER TABLE events DROP COLUMN legacy_source;

An add is safe during a rolling deploy. A drop is only safe after the readers have already moved. Local docker compose up never shows you this, because there is no overlap.

We now fail CI if a PR drops a column and also deletes the code that read it. That was the whole bug: both halves in one merge.