The request that started it was two sentences long, and both were reasonable. "Can the token include which projects the user can see? We're making a call back to the API on every page load just to filter the project switcher, and it's slow."
The question arrived from outside counsel and it was one sentence long: between 1 March and 30 June, which of our employees accessed the records of these 340 customers? It landed on the identity team, because the identity platform was where the audit log lived. Everyone knew that. It was
The outage lasted ninety-four seconds and took out every API in the company. Not the login page. Every API. The mobile app, the partner integrations, the internal admin tools, the checkout flow, the webhook dispatcher — everything returned 401 for a minute and a half, and then everything came back on
The incident started at 09:12 on a Tuesday, and for the first forty minutes nobody on the call believed it was an identity problem. Logins were timing out. Not all of them — maybe one in six, and only for users in two of the company's business units.
A team I spoke to had a login endpoint with a p99 of 420ms and no idea why. They had done the obvious work first. Tenant config was cached. The password hash was Argon2id tuned to about 90ms, which they knew was deliberate and correct. They had a read replica.
Minute thirty-eight of a sixty-minute technical call. The commercial part went well. Two of their engineers have been quiet for twenty minutes, and then one of them unmutes and says the thing: "So we'll need to run a small script at login. It's not much
The deploy went out at 09:12 on a Tuesday. It changed one line in the assertion-consumer path: a normalization step so that a customer whose IdP emitted NameID with a trailing space would stop failing. Small, obviously correct, reviewed in four minutes. At 09:41 the on-call engineer opened
The ticket said: "Enrichment hook stopped populating department for tenant 4471 after 3.9.0." It had been open for nine days when it reached me, which tells you something about how long it takes to believe a report like that. The hook interface hadn't changed.
The first enterprise deal usually arrives as a spreadsheet. Your product has been signing people up for two years. Email and password, Google and GitHub buttons, a magic link for people who forget. Signup takes eleven seconds. You have measured it, because signup conversion is a number your team looks
The ticket had been reopened three times, which is what made it interesting. A finance user's account had been compromised in February. The response was textbook and fast: password reset, all sessions revoked, TOTP seed invalidated and re-enrolled, mailbox rules audited, forwarding removed. Two weeks later the same
The outage started nine minutes after the capacity increase. The team had done everything right, by the usual standards. A large customer was going live on Monday and their own estimate said traffic would roughly triple. So on Friday afternoon the platform team took the runtime from six instances to
The ticket said: "Login works, but only sometimes, and only for some people at Contoso." Three days of investigation produced the following evidence. The CDN's access log showed a 302 for every request in the flow, no 4xx, no 5xx. The gateway showed the same requests