Operations

Friday Deploys and the Absent Operator

Shipped at 4:47pm. Worked in staging. Discovered the edge case at 6am Monday, when ten thousand customers found it first.

Friday Deploys and the Absent Operator
Illustration · Deimar Gutiérrez

A senior engineer merged a feature at 4:47pm on a Friday. It passed CI. It passed staging. It passed every manual check on the list. He closed his laptop and went into the weekend feeling like a professional. The Slack thread had three thumbs-up and one celebratory GIF.

At 6:03am Monday, the on-call paged him. The feature had failed silently for every customer in the EU. The test environment didn't include the timezone that exposed the bug. Ten thousand customers hit the error path over the weekend. Support tickets had been stacking up since 4am, and the on-call had been answering them alone. He spent Monday morning on the incident report and Monday afternoon on the fix he'd promised himself, on Friday, he'd have plenty of time to write.

The Friday push has a predictable cost shape. It works often enough, maybe eight times in ten, that the person shipping ends the week feeling sharp. The two times it breaks, it breaks at the worst hour, in front of the worst audience. And the bill goes to the on-call, not the deployer. The deployer learns nothing, because someone else absorbed the lesson. The on-call learns plenty, and rarely has the standing to change the practice.

The popular answer is a no-Friday-deploy rule. It holds for a quarter, then erodes. Engineers route around it. It's a tiny change. The flag is off, so it's not live yet. The customer needs it by Tuesday. A cherry-pick is a hotfix, not a deploy. The exceptions multiply. Within two quarters the rule is decorative, and the manager enforcing it ends up policing edge cases the team invented to dodge it.

The version that holds is operator-relative, not calendar-relative. Deploys happen in windows where a named, paid, on-call human has agreed to be paged for the fallout. Friday afternoon usually isn't that window. Not because the day is cursed. Because the operator pool shrinks for the next sixty hours. The question shifts from "can I ship this" to "who agreed to catch it if it breaks."

That reframing changes behavior. The engineer who wants to ship now has to find the on-call, ask directly, and explain the risk. Most won't run that conversation for a non-urgent change. The friction is the point. A team that internalizes the conversation stops needing the calendar rule, because the conversation becomes the deploy's natural precondition. It's the same failure as the runbook that lives in one person's head: the risk stays invisible until whoever carries it goes unreachable.

The culture point underneath is harder. Shipping ends the engineer's work and starts the operator's. When the two are the same person, deploy habits get healthier on their own. The Friday push costs differently when you're the one holding the pager. Split the roles, and you have to bridge the gap with policy, and policy decays. The on-call absorbs the cost without the authority to prevent it. The deployer creates the risk without feeling it. That misalignment is the cause. The calendar rule is a symptom.

The structural fix, where a company can afford it, is to rotate on-call across the deploying engineers. The week you're on-call is the week you think hardest about what you ship. Once you've been paged during someone else's bad Friday deploy, your own Friday deploys get more careful, more reversible, better tested. The lesson spreads through the team instead of through a policy doc. It's the same reason pulling your best engineer off the tools quietly raises everyone's risk: the person who understands the failure mode is no longer the one on the hook for it.

The deeper habit is to see the Friday deploy plainly. It's a decision about other people's weekends. Say it plainly and most engineers defer without being asked. The change you're about to ship will be carried by the on-call all weekend. They didn't agree to that. The sentence turns a personal finish line into a team consequence. Some deploys survive it. Those are the ones worth shipping.

The Friday push feels like closure. It isn't. Closure is the deploy that ran clean through the weekend and was proven by Monday morning. The one you shipped at 4:47pm is a hypothesis until then. Treat it like one. The teams that learn this after a silent quarter tend to learn it the expensive way.

Before your next Friday deploy, ask:

  • Does the named on-call know about this deploy, and have they agreed to carry its risk this weekend?
  • If it waited until Tuesday morning, what would the company lose?
  • What's the reversal plan if it breaks between 8pm Friday and 7am Monday?
  • If I were on-call this weekend, would I want this shipped Friday at 4:47pm?

The fourth question is the real one. Most Friday deploys don't survive it. The ones that do are already coordinated with the on-call and have a clean way back. Ship those. The rest can wait for Tuesday, and for an operator who was asked.