Leading a Company-Wide Reliability Surge
A cross-team effort that took worst-case data processing from 39 days to under 12 hours.
Overview
Led the application platform side of a company-wide tiger team formed to close accumulated reliability gaps. Leadership had flagged product stability as a risk to the go-to-market.
Problem
Rapid platform scaling had left behind a set of reliability gaps with no single owner and no shared view of root cause: stuck operations, multi-week post-processing jobs, a hard cap on password audit scale, and degraded page load times across core product pages. Interim engineering leadership declared it a company-level priority and deprioritized other roadmap work to address it.
Constraints
- Roughly 80% of the team's engineering capacity was redirected from feature work for the duration of the surge.
- Fixes had to ship without regressing existing customer-facing behavior.
- Several root causes had no established owner or prior investigation.
- Work had to continue in parallel with normal on-call load.
Approach
Broke the problem into 10 discrete, owned problem classes instead of one undifferentiated backlog, assigned a sponsor to each, and tracked every fix against statistically validated before-and-after metrics. Authored the executive-facing reliability report shared with all of engineering at the close of the surge.
Key Decisions
Organize the surge as 10 owned problem classes rather than a single shared backlog
A single backlog would have buried root causes under ticket volume with no clear accountability. Assigning a sponsor to each problem class, stuck operations, post-op ETL, password audit scale, and others, made progress visible team by team and gave each area a clear decision-maker.
- Run one unified bug-bash against a shared ticket queue.
- Prioritize top-down from a single leadership-ranked list.
Redirect the large majority of engineering capacity to reliability for the surge window
Leadership had identified stability as a risk to how the market perceived the product. Partial capacity allocation would not have moved the underlying metrics fast enough to change that perception within the window that mattered.
- Hold feature work steady and allocate a fixed percentage of capacity to reliability.
Require statistically validated before-and-after metrics for every shipped fix
Prior fixes in this area hadn't always held up under real production traffic. Requiring sample sizes and significance testing on every change built confidence, internally and with leadership, that the reported improvements were real.
- Ship fixes and monitor informally based on support and CS feedback.
Tech Stack
- Datadog
- PostgreSQL
- AWS
- Kubernetes
- GraphQL
- Kafka
Result & Impact
- 39 days to under 12 hoursPost-op ETL duration (largest customers)
- ~2 days to ~4 hours (12x faster)Password audit duration (50k identities)
- 50,000 to 250,000 validated at scalePassword audit identity cap
- up to 95% faster across core product pagesKey page load times
- ~5,000 per week eliminatedAuth errors from a race condition
The surge turned a vague, company-level stability complaint into 10 tractable engineering projects with named owners. The password audit and post-op ETL fixes in particular unblocked enterprise deals that had been stalled for months on platform limitations, and the portal performance work grew into its own standalone initiative with weekly, data-driven release notes.
Learnings
- Assigning a single owner per problem class turned a diffuse stability complaint into 10 tractable projects with clear accountability.
- Requiring statistical significance on before-and-after metrics built trust in the surge's results, internally and with leadership.
- Reliability work compounds: removing a hard scaling cap on password audits directly unblocked enterprise deals that had been stuck on platform limitations for months.
The Breaking Point
By early 2026, reliability problems had accumulated faster than they were being fixed. Password audits capped out at 50,000 identities. The largest customers’ post-op processing could take weeks. Core product pages were loading slowly enough to generate regular support tickets. None of it had a single owner, and none of it had been prioritized over feature work, until interim engineering leadership called it what it was: a risk to how the market perceived the product.
Ten Problems, Ten Owners
Rather than run one shared backlog, the surge split into 10 discrete problem classes, stuck operations, post-op ETL, the insights pipeline, database performance, password audit reliability, and others, each with a named sponsor accountable for root cause and resolution. That structure made a genuinely fuzzy, company-wide complaint tractable: each sponsor could report concrete progress against their specific area instead of a shared pile of tickets nobody fully owned.
Making the Numbers Trustworthy
Every fix shipped against a validated before-and-after measurement, not an anecdotal read of whether things felt better. That discipline mattered most on the password audit work: the identity cap that had sat at 50,000 for years came out entirely, validated first at 250,000 identities and then at 1 million, while duration on the original 50,000-identity benchmark dropped 12x. The same rigor applied to the portal performance work, where individual fixes were tracked with sample sizes and significance testing before being reported as done.
Result
Post-op processing for the largest customers went from weeks to under 12 hours. Page loads across core product pages improved by as much as 95%. The surge closed with an executive-facing report summarizing the full body of work, and the portal performance thread it started continued on as its own standing initiative well past the original surge window.