THE ELEPHANT

A true infrastructure story, told as a restaurant.
How thirty kitchens were replaced mid-service, without a single guest noticing.

Seven cities. Thirty restaurants.
No service disruption.

Prologue

The Service Never Stopped

The obvious answer, when a kitchen needs replacing, is to close. Put up the sign. Lock the door. Gut it overnight and reopen in the morning. We know — we have done it that way. We know what the sign feels like, what the calls sound like, what the morning after costs.

Imagine a packed restaurant at full service. Every seat taken. Guests mid-meal, already thinking about their dessert order. A queue building at the counter — orders going directly to the kitchen, no one standing between the guests and the pass. The kitchen working at full pace.

Now imagine replacing the entire kitchen — every appliance, every station, every member of staff — while every guest keeps eating, mid-meal. Not after hours. Not with a closed sign on the door. Right now. During the rush.

Not once. Thirty times. Across seven cities and thirty restaurants. Each one full. Each one unaware that anything was happening at all.

This is that story.

Dramatis Personæ

The Cast

Applications

Guest

They order, eat, and keep coming back — reading from and writing to the database throughout service.

PgBouncer

Waiting Staff

Once in place, every order flows through them. The seamless coordination layer between guests and kitchen.

The Switchover Script

Manager

She stands at the pass — the point between the old kitchen and the new, between the order in flight and the moment it is fulfilled. She watches both. She checks the sync. She gives the signal.

PostgreSQL 13 · Aurora

Old Kitchen

Reliable. Proven. Years of service. Time to retire — gracefully, during full service.

PostgreSQL 17 · Aurora

New Kitchen

Built quietly in the back, receiving a live copy of everything. Waiting for its moment.

Chapter I

The Counter Problem

The counter, in this story, is the direct connection — the place orders went when there was no one to carry them to the guest. Guests came straight to the kitchen window, placing orders directly through the pass. No waiting staff. No coordination layer. Every order landing directly on the kitchen with nothing in between. You could hear it from the street.

It worked, until the day the kitchen needed replacing. Then the absence of that layer became impossible to ignore.

Thirty to sixty days before

The Reservation Book Opens

Everyone who relies on the kitchen has to be told something is coming. Not the details — just that there will be a night when the restaurant is dark. Multiple sign-offs. Calendars negotiated. Approvals from people several steps removed from the kitchen. The date gets written in, weeks out. Thirty days notice, minimum.

In the weeks that follow

Choosing the Night

A Saturday is chosen — after the last cover, after the staff have gone home, after the floor is empty. If something goes wrong on that quiet night, the staff are pulled back in on the one evening they thought they had off. The night was chosen to minimise the number of guests affected. It cannot, by policy, be moved without starting over.

Zero Hour

The Sign Goes Up

The front door is locked. Guests who arrive find nothing. Any order already in transit — gone. The counter goes dark. For the duration of the window, the restaurant does not exist.

The Black Box

Handed Off

Somewhere in the next five to ten minutes, the kitchen changes. Exactly when — no one can say. Sixty seconds. Maybe two minutes. The clock runs. There is no way to watch it happen. No way to intervene. The answer arrives when it arrives.

Reopening

The Reckoning

The sign comes down. Most guests find their way back. Some don't — they arrive at an address that no longer answers, or reconnect to the wrong kitchen. An incident review is opened. Notes filed. The next restaurant waits.

Per restaurant. Per city. Thirty times.

The Reckoning

The Bill

The old kitchen's lease had a clause buried in the fine print. Past a certain date, the landlord begins charging a premium — not for anything new, not for any improvement. Simply for the fact that you have not yet upgraded. The meter runs whether service is good or bad, busy or quiet. It just runs.

Across seven cities and thirty restaurants, with 888 burners (vCPUs) between them, that clause had a very specific number attached to it.

The premium — per burner, per hour, every hour the old lease ran
Years 1–2
0 / day
0 / month
0 / year
Year 3+
0 / day
0 / month
0 / year
Not for anything new. Not for any improvement. For the fact of staying.
† 888 vCPUs · Aurora Extended Support: $0.10/vCPU-hr yr 1–2 · $0.20/vCPU-hr yr 3+

The question was never whether to upgrade.
The question was how — without closing the restaurants.

Interlude

The Waiting Staff

The first move was not the kitchen swap. Before anything else could be touched, a coordination layer had to exist — something that could be redirected without the guests ever knowing. That layer was PgBouncer: the waiting staff. Putting them in place — across all thirty restaurants, before any kitchen work began — was itself a full operation, completed first.

Before
Guest
──────►
Kitchen
After
Guest
──►
Waiting Staff
──►
Kitchen

Every order now flows through them.

This matters more than it first appears. Without the waiting staff, there is no lever. No way to pause orders mid-service. No way to say: hold everything for ten seconds while the kitchen changes. No way to redirect traffic without the guests ever knowing a redirect happened. The dining room and the kitchen are the same room, and nothing can be done to one without disturbing the other.

With the waiting staff in place, the kitchen becomes a separate room for the first time. The dining room continues. The kitchen can be changed. The renovation becomes possible — not in theory, but in practice, at full service, with every seat taken. That separation is what the entire operation rests on.

For the first time, the kitchen could be touched without touching the guests.

Chapter II

Mise en Place

The new kitchen did not begin from nothing. It began as an exact copy of the old one — same recipes, same stock, same inventory — taken at a precise moment in time. From that moment forward, it received a live copy of every order the old kitchen fulfilled, every order placed by every guest. In real time. Always current. Nothing improvised. Everything ready.

While the new kitchen caught up — narrowing its lag from seconds to bytes to nothing — the script was built to handle every contingency.

But before any real restaurant was touched, the process ran against test kitchens — built to the same plan, standing in for the real ones in every structural way, with no real guests in their seats and no real orders coming in. This is staging. Every failure mode triggered deliberately, to confirm the recovery. Every guardrail tested against the thing it was meant to catch.

One test found something real. After the switch, the old kitchen is locked to everyone — by design. No guest, no member of staff, no stray order can reach it. The manager is the sole exception: she registers her own access card on the fly, as part of the process, before the lockdown begins. The card is not pre-issued. She whitelists herself. In the test kitchen, that registration produced the wrong credential, and the door did not recognise her. At the exact moment when getting in was critical, she would have been outside. She is the one who must enter after the switch: first to cut the supply line from old kitchen to new, then to route it in reverse — so the old kitchen stays current, and a return is possible if anything goes wrong. Without that access, neither step could happen. Not a delay. A permanent loss of control. Found in the test kitchen. Fixed before any real kitchen switchover.

Another test found that the step confirming the reverse supply line had been established — that the old kitchen was receiving what it needed to stay current — had been written to always report success, whether the line was open or not. If the connection failed to form, no one would know. Found in testing. Fixed before any real kitchen switchover.

The manager did not arrive at a real kitchen until she already knew how every edge of the story ended.

"What if the kitchens fall out of sync at the critical moment?"

↻ click to reveal

Before she gives the signal, the manager walks the pass between the two kitchens and checks the gap — the lag between what the old kitchen knows and what the new one has received. It must read zero. Not near zero. Not almost. She stands there until it does.

"What if a guest somehow reaches the old kitchen after the switch?"

↻ click to reveal

The moment the switch is made, the old kitchen is locked to everyone. A guest who still remembers the old kitchen's address finds a locked door. There is no way in. There is no mistake to make.

"What if something goes wrong mid-switchover?"

↻ click to reveal

The manager steps back. She signals the waiting staff to resume on the old kitchen, and service continues without a pause — the guests never knew she'd moved at all. She takes a breath, watches for the right moment, and tries again. Nothing broken. Nothing to explain.

"What about the hundreds of counters the kitchen copy process doesn't carry automatically?"

↻ click to reveal

Every kitchen tracks running totals — ticket numbers, dish tallies, stock counts. The copy process doesn't carry them. Get them wrong and the new kitchen issues duplicate tickets and mis-numbered orders. In the quiet window before resuming, the manager ran a single check across all of them — hundreds — and advanced each counter past where the old kitchen left off. No number could repeat.

"What if something fails after the new kitchen opens?"

↻ click to reveal

The moment the new kitchen took its first order, the old one began receiving a copy of everything — quietly, in the background, keeping pace. Receiving a copy of every new order, but no longer serving them. Current. Still ready. Not standing by out of fear — standing by because a plan that accounts for everything accounts for this too.

"Not courage under pressure. Just pressure, applied earlier."

Chapter III

The Manager's Rhythm

The manager does not force her way. She watches. She waits for the right moment. If the moment isn't right — she steps back, lets service continue, and tries again. Not because she is timid. Because she knows that a forced transition is a failed one.

switchover.sh — opportunistic pause
# Watching for the right moment Checking the gap between kitchens... gap: closing... closing... none ✓ ── Attempt 1 ────────────────────── Waiting staff: hold all orders. In-flight orders: 6... 5... 4... 4... Still 4... ⏱ running out of time. ⚠ Guests would feel the wait. Stepping back. Old kitchen resumes. Waiting staff resumed → old kitchen Service uninterrupted. Watching for a quieter moment... ── Attempt 2 ────────────────────── Waiting staff: hold all orders. In-flight orders: 3... 2... 1... In-flight orders: 0 ✓ New kitchen: fully current ✓ Yes, Chef. ✓ Window open. Proceeding.

She never compromises the customer experience to force the switch.

Chapter IV

The Switchover

When the moment came, it was not dramatic. It was the execution of a plan prepared in full. The guests were eating. The manager was watching. She said fire. The kitchen changed.

The waiting staff paused. The final orders drained. Hundreds of counters aligned. The old kitchen was isolated — its replication path kept open, its data still current. The waiting staff reloaded and resumed, now pointed at the new kitchen. The old kitchen was kept live and confirmed ready — a standby.

Every second counts. Under ten of them. Start to finish.

If any order reached the kitchen a moment later than usual, no guest knew it. No order was lost. No plate came back. The pass moved freely.

Tuesday afternoon. Service in full swing. Nobody noticed.
Chapter V

The Scale

It wasn't one restaurant.

0 Cities
0 Restaurants
0 Initiative
0 Errors Business as usual for every application

Not thirty moments of tension — thirty executions of the same careful plan. Each one ran the same way.

Not thirty restaurants at once. One by one — starting with the test kitchens, where no real guests were at risk. Then the smallest, quietest restaurant in the network. Then the next. Each successful switchover was proof. Each one made the next more certain.

By the time the largest, busiest flagship locations came — the ones with the most to lose — the process had already run across every smaller kitchen that preceded them. The path was well-worn by then.

Test kitchens
Smallest restaurants
Mid-sized restaurants
Larger restaurants
Flagship locations

What this approach avoided across 30 clusters

Up to 5 hrs Combined downtime avoided
The Reckoning

AWS Blue-Green vs. Seamless Upgrade

AWS Blue-Green Seamless Upgrade
Customer notice required 30–60 days (change management policy) None
Teams involved Many Infra-storage
Timing Weekend, off-hours Working hours
Downtime per cluster 60–120s (unpredictable within 5–10 min window) < 10 seconds
Switchover control AWS black box Full control
Retry on failure Manual, high-stakes Automatic, safe
Kitchen changes address Same name, new address — guests may lose their way until the maps catch up The waiting staff's address never changes — they handle the kitchen move themselves
Errors across 30 clusters Expected 0
Seven cities. Thirty restaurants. Every one of them mid-service. Every one of them seamless.
The guests finished their meals. No order was repeated. No dish arrived at the wrong seat.

Service continued.

The old kitchen: locked, current, and quietly ready — just in case.

PostgreSQL 13 → 17

7 environments · 30 clusters · 0 errors

2026