Operations

The billing system only a former contractor understands

Production revenue ran on a service nobody could fix. The contractor who built it had quit three years earlier and replied to email when he felt like it.

The billing system only a former contractor understands
Illustration · Deimar Gutiérrez

A senior engineer at a series-B company walked me through their billing system one afternoon. A contractor had built it in year two, when the team was small and nobody had the bandwidth to take it on. The contractor was good. The system worked. He'd taken a full-time job elsewhere three years earlier. At that one company, the billing system was still in production, still processing around $400,000 a month.

I asked who'd fix it if it broke at 3am. He thought for a while, then said they'd probably call the original contractor. When had he last spoken to him? About a year ago. How long had that last email taken to get a reply? Three weeks. The company's revenue infrastructure was running on a dependency whose emergency response time was measured in weeks.

This pattern is common in growth-stage engineering and almost never named as a risk in an operations review. At the time, the contractor was the right call. The team didn't have the capacity, the project had a deadline, and the contractor was available and competent. The system shipped. The engagement ended. The system kept running. What little knowledge the team had of it was thin to begin with, because an outsider built it, and it thinned further every quarter. By year three, nobody in-house understood the code well enough to fix it, and the contractor was gone.

The risk is built into how the system was made. The contractor made a hundred small decisions during the build, each one justified by the constraints of that week, none of them written down. Those decisions live in the code in non-obvious ways. The current team, trying to fix a bug or extend the system, hits those decisions as obstacles rather than choices. The work takes longer than anyone estimated, because the team is reverse-engineering reasoning from conversations they were never part of. This is the same failure mode as the runbook that lives in one person's head, except the person already left.

The cost stays invisible during normal operations. The system runs. Nobody thinks about it. The risk only shows up when something breaks: an upstream API changes, a credential expires, a dependency bumps a major version. At that moment the team needs someone who understands the system immediately, and the only such person answers email in three weeks. A thirty-minute fix becomes a week-long outage.

Remediation is expensive either way. The two real options are bringing the system in-house, rebuilt by engineers who'll own it, or paying the contractor a retainer for guaranteed response time. The rebuild costs more than the original, because the team now works without the deadline pressure that produced the lean first version and without the context behind the original design. The retainer is unreliable, because contractors at growth-stage companies have day jobs and your emergency isn't their priority. This is the same trap as shipping an integration nobody's assigned to own: the thing works right up until it needs a human, and then there isn't one.

The right answer follows the stakes. For anything that handles revenue, customer data, or core operations, in-house ownership is the only durable answer, and the cost is worth paying. For lower-stakes systems, a retainer can hold, as long as everyone accepts that response time will be slower than they'd like.

Run the audit. List every production system. Next to each, write the name of the engineer who'd fix it at three in the morning. Any system whose answer is a former contractor is a risk you haven't priced yet.