The Black Friday That Took Down the Startup
Black Friday, a product launch, a press mention, or a social media moment: any predictable traffic spike can expose architectural weaknesses that steady-state load never triggered. The startups that go down under a spike usually had the same failure mode. The database connection pool saturated. The queue backed up. The cache was not there. The autopsy always shows the same three lines in the logs.
Written by Yashveer Singh, founder of Yashveer Labs.
What you actually need to know
- Database connection pool exhaustion is the most common cause of spike-driven outages. Fix it before the spike.
- Steady-state load tests do not reveal spike behavior. Run tests at 3x your peak with a ramp-up period.
- Latency percentiles are your early warning system. An alert on p95 at 2x baseline gives you response time before the outage.
- Scaling vertically first is faster than scaling horizontally. Have a vertical scaling runbook ready.
- The postmortem must be written within 24 hours and acted on within two weeks or it will not be acted on.
| Failure Mode | Detection | Mitigation | Prevention |
|---|---|---|---|
| DB connection pool exhaustion | High p95 latency | Scale vertically, add PgBouncer | Connection pooler in production |
| Cache miss storm | CPU spike on DB | Warm cache before spike | Cache warming runbook |
| Queue backup | Job lag metric | Scale workers | Auto-scaling queue workers |
| Server OOM | Instance crash | Restart, scale up | Memory profiling, load testing |
The core argument
The spike that takes down a startup almost always comes on the day that matters most. A press feature. The product launch. Black Friday. The spike is predictable. The failure is not, because the team never ran the system under spike conditions in a controlled environment to see what would break first.
The autopsy is consistent across the outages I have seen. The database connection pool hit its limit. Requests started queuing for database connections. Response times climbed from 200ms to 2 seconds to 30 seconds. Users refreshed. The load increased. The queue grew faster than it drained. The system stopped responding. The founder was on the phone with AWS support while their biggest opportunity of the year was going down.
The failure is almost never in the code. The logic is correct. The feature works in staging. The failure is in the configuration and the capacity assumptions. A connection pool sized for 100 concurrent users fails under 1000. A server sized for steady-state load runs out of memory under a ramp. A database that handles 200 queries per second at p50 hits 8 seconds at p99 when the query mix changes under load.
These failures are findable before they happen. A proper load test with a realistic traffic ramp reveals the connection pool limit, the memory ceiling, and the cache miss behavior. The load test is the most underinvested hour in every startup's engineering calendar.
What the load test should cover
A load test that is useful for spike preparation runs in three phases. The ramp-up phase, where traffic increases linearly from zero to 3x peak over 15 minutes. The steady-state phase, where traffic holds at 3x peak for 30 minutes. The ramp-down phase, where traffic decreases back to zero.
During the test, watch four metrics. Database connection pool utilization: if it hits 80 percent during steady state, the pool is too small. p95 response time: should stay under 2x baseline during steady state. Memory utilization on the application servers: should not exceed 85 percent. Queue depth for background jobs: should not grow continuously during steady state.
When any of these metrics breach their threshold during the load test, that is the finding. The load test result is not pass/fail. It is a list of the things that will break first, in order, so you can fix them before the spike arrives.
Common mistakes startups make before a traffic spike
- Not running a load test at all. Hoping is not a capacity plan.
- Running a load test at 1.5x instead of 3x. Spikes are not gentle. The test should be more stressful than the expected spike.
- Running the load test once in staging but not in production-equivalent infrastructure. Staging and production often have different database configurations, different connection pool settings, and different memory allocations.
- Not having a scaling runbook. When the spike arrives, there is no time to figure out how to scale. The runbook should be written and practiced before the day.
- Not alerting on latency percentiles. By the time errors appear, the outage is already in progress. The p95 alert catches it earlier.
Where to start: a 3-step spike preparation plan
Step 1: Run a load test this week. Use k6, Locust, or Artillery. Write a scenario that represents your most common user flows. Ramp from zero to 3x your current peak traffic over 15 minutes. Hold for 30 minutes. Watch the four metrics. Write down what broke first.
Step 2: Fix the first failure mode. Whatever hit its limit first in the load test is the thing most likely to cause an outage under a real spike. If it was the database connection pool, add PgBouncer or increase the pool size. If it was server memory, right-size the instances or look for the memory leak. Fix the first failure mode before the next load test.
Step 3: Write the scaling runbook. Define the steps to scale vertically (bigger instances), to scale horizontally (more instances), and to enable circuit breakers or graceful degradation if the load is unmanageable. The runbook should be runnable by any engineer on the team without the original author present.
Related reading
Frequently asked
Why this work lands with me
I am Yashveer Singh. Founder of Yashveer Labs. I take this kind of project because I have done enough of them to know what kills them. The version of me that writes a post like this is the same one who builds the system afterward. There is no handoff to a junior, no agency middleman, no surprise scope. That is the bet I am making on my own brand.
Posts that line up with this one.
- Startup Failure Postmortems and Fear
The Engineer Who Left a Year of Bug Fixes Behind
A postmortem on the silent damage an engineer carries when they leave without handing off what they know. What actually gets lost, why bus factor kills quietly, and how to build teams that survive a departure.
- Startup Failure Postmortems and Fear
The Vendor Outage That Tested Your Disaster Plan
A postmortem on a third-party vendor failure that exposed a startup's missing disaster recovery plan. What broke, who owned nothing, and how the business relationship with customers changed permanently.
- Startup Failure Postmortems and Fear
The Wrong Tech Stack Decision That Compounded for Three Years
A postmortem on a technology choice made at the beginning of a startup that became increasingly expensive to live with. What the team optimized for, what they should have optimized for, and how long the cost kept compounding.
- Startup Failure Postmortems and Fear
The Side Project That Became the Main Project (and the Reverse)
Two stories that look opposite and are actually the same. The thing you built on the side took off and ate your main thing. The main thing you built faded and the side thing carries you. Both are about paying attention to what is working.