I used to think retries were one of those features you added almost without thinking.
An API request fails, you wait a little, and then you try again. It feels so obvious that most of us never stop to question it. In fact, if someone asked me a year ago how to make an application more reliable, adding retries would've probably been one of the first suggestions I'd give. The logic seems impossible to argue against. Networks fail. Servers restart. Temporary failures happen. If the first attempt doesn't work, the second one probably will.
It wasn't until I read Amazon's Builders' Library article, "Timeouts, Retries, and Backoff with Jitter," that I realized I had been thinking about reliability from only one side of the connection.
I was thinking like a client.
AWS was thinking about an ecosystem.
That difference completely changed how I look at distributed systems.
This is why tech companies ship SDKs in the first place. They don't want every developer hand-writing raw HTTP calls, auth headers, request signing, pagination, timeouts, retries, error parsing, and rate-limit handling from scratch. The SDK wraps all of that messy protocol work into clean functions. Instead of learning every tiny rule of the API, you install a package, call something like client.payments.create(), and move on with the actual product you're building.
Most applications today don't talk to servers directly. They usually use SDKs. A payment SDK. An authentication SDK. A storage SDK. An AI SDK. Every mobile application, backend service, CLI, cron job, browser extension, or internal tool using that SDK inherits the exact same retry behavior written by its developers.
That sounds harmless until you remember that software almost never runs once.
It runs everywhere.
Imagine you've published an SDK that thousands of companies integrate into their products. At any given moment there might be ten thousand applications communicating with your API. Some belong to startups. Some belong to enterprise systems. Some are mobile apps sitting in people's pockets. Others are backend services processing millions of requests every hour.
Now imagine that one dependency inside your infrastructure becomes slow.
Not completely unavailable, just slow enough that requests begin timing out.
Every application using your SDK experiences the same failure at almost the same moment. None of those applications know anything about each other. They simply execute the code you wrote.
And your code says exactly the same thing for everyone.
Retry after one second.
In code, the risky version usually looks boring:
func requestWithRetry() error {
if err := callAPI(); err != nil {
time.Sleep(1 * time.Second)
return callAPI()
}
return nil
}
Nothing feels dangerous about that instruction until you stop looking at a single application and start looking at all of them together.
Look at the first diagram.
At time zero, the service becomes unavailable for a brief moment. Around ten thousand requests fail. Every SDK receives a timeout. Since every SDK follows the exact same retry strategy, every one of them waits for one second.
Then something unexpected happens.
Exactly one second later, ten thousand new requests arrive almost simultaneously.
It wasn't a sudden user spike, an attack, or normal traffic growth.
The traffic was created by your recovery logic.
The server is already trying to recover from the original failure, but before it has a chance to breathe, another perfectly synchronized wave arrives. Some of those requests fail as well, so every SDK waits again. Three seconds later another wave appears. Seven seconds later another one. Every retry creates another synchronized spike.
t=0s server dies -> 10,000 requests fail
t=1s ALL retry at once -> 10,000 hit simultaneously <- SPIKE
t=3s ALL retry at once -> 10,000 hit simultaneously <- SPIKE
t=7s ALL retry at once -> 10,000 hit simultaneously <- SPIKE
The requests themselves are not the real problem.
Their timing is.
That's the part I had never thought about before.
Individually, every client makes a perfectly reasonable decision. If your request failed, trying again is usually the correct thing to do. But distributed systems don't experience individual decisions. They experience the combined effect of millions of individual decisions happening at exactly the same time.
AWS describes retries as selfish, and I think that's one of the most accurate engineering descriptions I've ever read.
Every client is only trying to help itself succeed.
Collectively, those clients make recovery significantly harder.
So the obvious question becomes: if retries are still useful, how do companies like Amazon stop this synchronization from happening?
The answer turns out to be surprisingly simple.
Don't make every client wait the same amount of time.
Instead of telling every SDK to retry after exactly one second, let every SDK choose a random value somewhere between zero and one second. Some retry after 120 milliseconds. Others after 430 milliseconds. Some wait almost the entire second. Nobody knows exactly when the next client will retry, and that's precisely the point.
Now look at the second diagram.
The same ten thousand requests still exist.
The same outage still happened.
Nothing about the failure changed.
What changed was the shape of the traffic.
Instead of a single wall of requests hitting the server at one precise moment, the retries spread themselves naturally across the entire retry window. One server no longer has to absorb ten thousand requests in a single instant. It processes a steady stream instead of repeated explosions.
That tiny amount of randomness has a name.
Jitter.
AWS recommends something called Full Jitter, where every retry waits for a random amount of time between zero and the current backoff limit. If the backoff delay is one second, the SDK randomly chooses anywhere between zero and one second. If the next retry allows two seconds, it randomly chooses somewhere between zero and two seconds. Then zero to four. Then zero to eight.
That turns the retry logic into something closer to this:
func retryWithFullJitter(
ctx context.Context,
maxAttempts int,
baseDelay time.Duration,
maxDelay time.Duration,
call func(context.Context) error,
) error {
var err error
for attempt := 0; attempt < maxAttempts; attempt++ {
if err = call(ctx); err == nil {
return nil
}
if attempt == maxAttempts-1 || !isRetryable(err) {
return err
}
backoff := baseDelay << attempt
if backoff > maxDelay {
backoff = maxDelay
}
delay := time.Duration(rand.Int63n(int64(backoff) + 1))
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return err
}
This version does a few important things at once: it caps the number of attempts so the client cannot retry forever, doubles the retry window after every failure, limits that window with maxDelay, and then picks a random delay anywhere inside the current window instead of making every client sleep for the same duration. The context.Context part matters too, because if the original request is cancelled or times out, the retry loop stops immediately instead of quietly continuing in the background.
At first glance, introducing randomness into software feels wrong. We spend our careers trying to make systems predictable. We remove randomness because deterministic software is easier to reason about.
Distributed systems quietly teach the opposite lesson.
Sometimes the most predictable thing you can do is make sure everyone behaves a little differently.
That was my biggest takeaway from Amazon's article.
Reliability isn't only about recovering from failure.
It's about recovering without causing everyone else to fail with you.
And sometimes, the most dangerous line of code isn't a network call or a database query.
It's a harmless-looking retry().
References