How do you rehearse a cutover you can only do once?
Three rehearsals, and the useful one is always the second.
The first rehearsal is not really a rehearsal. It is the first honest reading of the plan, and it fails — reliably, in the first hour, usually on something nobody would have predicted and everybody recognizes immediately. A step in the wrong order. A dependency that only exists on Tuesdays. A credential that expired.
Everyone expects this, which is what makes it safe. Nobody is embarrassed by a first rehearsal going wrong, so nobody defends anything, and the sequence gets corrected quickly and without argument. Its output is a better runbook, and that is all it is for. A first rehearsal that passes cleanly is not good news; it means the rehearsal was not faithful enough to production to find anything.
The second rehearsal is where the plan survives, and that changes what people do with it. Once the steps hold, attention moves off the sequence and onto everything around it — and that is when somebody discovers they are not sure what they are allowed to decide.
It comes out sideways, usually as a question about something else. If the reconciliation is out by a rounding error at four in the morning, do we stop? And the room goes quiet, because two people believe they own that call and one of them is wrong, and nobody has ever had to find out.
That is the entire value of the second rehearsal, and it is why skipping it is so expensive. People will say the thing once. Having said it, and having had it answered, they never say it again — so if you go from the first rehearsal to the night, the question is asked for the first time at four in the morning by somebody who has been awake for nineteen hours.
The second rehearsal is the only one where somebody says out loud that they do not know whose call it is. After that they never say it again, so if you skip it you never hear it.
>
— Yusuf Bramwell, Principal, Migration and Cutover
The third rehearsal is theater and it is still necessary.
It changes almost nothing. The sequence is right, the authority is settled, the timings are known. What it does is put the people who could not attend the first two in the room once — the on-call DBA, the person from the downstream team, the executive who will be phoned if it goes wrong. They need to have stood in that room, seen who is at which desk, and heard what the escalation actually sounds like.
It is worth being straight that this is what it is for, rather than pretending it is a test. Calling it a dress rehearsal and expecting findings from it leads people to invent findings, which wastes the one rehearsal whose job is to be uneventful. It should be boring. If the third rehearsal produces a surprise, the honest response is to schedule a fourth, and to be glad the surprise arrived on a Wednesday.
