A Billion Row Versions From One Uncommitted Read Stalled a MySQL Replica

A read replica is three days behind and falling further. The source is serving 15,000 queries a second, exactly as it has every month for a year. Restarting the replica clears the lag, and within a day it is behind again. The replica's data directory has grown to four times the size of the source's. One connection is open against it, doing nothing, started by a server that no longer exists. What is that connection still holding?

A Query on a 3-Billion-Row Table Took 100 Minutes: One Added Line Made It 40ms

A report against a 3.2-billion-row table runs for 1 hour 43 minutes and returns 1,823 rows for one retail chain's 603 stores. Every join hits an index. The chain_id column that would make the whole thing trivial is sitting right there on the table. One added line takes it to 40 milliseconds with no schema change. Which line, and why couldn't the optimizer find it on its own?

Collation Drift: The Silent Index Killer

Two customer_ref VARCHARs, same name, same length, one on utf8mb4_general_ci and one on utf8mb4_0900_ai_ci because they were born three years apart on either side of a version upgrade. The join still runs. It just quietly stopped using the index.

Self-Healing Databases with a Health Check and a Watch Script

A disk fills with binlogs while the on-call sleeps. The fix is a single PURGE statement, and picking the wrong target file leaves a lagging replica unable to ever reconnect. A health check wired to a handler can pick the safe target in the moment, with every replica's position in hand, and file the ticket that keeps the burst from becoming invisible.

Before the Bad Write

Here is a drill for your next DR review: remove one bad write from six weeks ago without throwing away the six weeks of legitimate writes that landed on top of it. The full checklist fails it, backups, PITR, hot replica, delayed replica, all of it. The layer that passes is the one most teams never build.

Keep Your Skills Sharp the Hard Way

The endless stream of 'better prompts' and 'agentic loops' content is mostly a campaign by the people who bill you for the dependence. The thing it tells you to skip is the only thing that keeps the agent from shipping damage under your name.

Scale the Pattern, Not the Instance

Three engineers wrote three slightly different cleanup queries for the same orphaned-row bug across three quarters, each fixing the occurrence in front of them and none touching the class. The fix that holds lives one level up, where a single foreign key, pipeline, or batched handler covers every recurrence that hasn't happened yet.