Twelve Permission Levels Became One
An admin tool shipped with twelve permission levels. By month twenty-eight every senior engineer had root, granted one exception at a time.
Every senior engineer at the company had root access. Nobody had granted it. It accrued one reasonable exception at a time, until the 12-level permission system meant nothing.
The internal admin tool shipped in 2024 with twelve permission levels: read-only, customer-data, billing-data, support-actions, billing-write, refunds-under-a-cap, refunds-over-a-cap, user-management, role-assignment, system-config, audit-log, and root. The hierarchy was designed with care. The point was to constrain access by function and tenure.
By month twenty-eight, the exceptions had eaten the design. A senior CSM got refunds-over-cap for one customer, and it stuck. An engineer got role-assignment for a migration and never lost it. A support agent got bumped to user-management to clear a backlog, and the access outlived the backlog. Each grant was approved by whoever was asked, usually the requester's own manager, for a task that had an end. None of the grants did.
It surfaced during the audit before a SOC 2 renewal, the AICPA attestation the company's larger customers asked for. The auditor pulled the access matrix. It didn't match the org chart or the written role definitions. He flagged thirty-six anomalies. Remediation ran six weeks of engineering time that was supposed to go to the roadmap, and meant clawing back access people had held for over a year. Say two engineers carried it at a loaded $3,500 a week each: that is $42,000 of roadmap spent undoing grants that were each approved in a single message. Each clawback hurt, because everyone had built workflows on top of access they weren't supposed to have.
The tempting lesson is that twelve tiers was too many. Cutting to four would have made the mess easier to read, and it would have arrived anyway. The tier count didn't grant refunds-over-cap to a CSM. A grant with no end date did, signed by someone who didn't own the refund budget. Four tiers with that same grant process drift as surely, onto a smaller chart.
So the repair goes into the grant. Every exception carries three fields: who it's for, which data owner approved it, and the date it ends. The data owner is whoever answers for that data, so refunds-over-cap gets signed by the person who owns the refund budget, not the CSM's manager. The default end date is thirty days out, and the tool revokes the access on that date by itself, with no reminder that someone can ignore. Renewal takes a fresh approval from the same owner. If a person needs the access permanently, the role definition changes, in writing, and everyone in that role gets it. Like a runbook that lives in one person's head, access with no expiry rots quietly, so the expiry has to be automatic.
The artifact that proves it works is still boring: a one-page access matrix, pulled on the first day of each quarter by the owner or whoever runs operations, compared against the written roles. Its job is no longer cleanup. It checks the expiries. If it finds a grant past its end date that is still live, the revocation job is broken. If it finds the same exception renewed four times, that role definition is wrong. Either one gets fixed that day, before an auditor prints the matrix for you.