While studying retry strategies, I noticed that many explanations begin with a reasonable rule: retry an operation when the failure is temporary.
The rule becomes harder to apply when the client cannot tell whether the operation failed.
Suppose a client sends a POST request to create an order. The request times out before a response arrives. Retrying seems like the obvious next step. But what if the server inserted the order and committed its transaction before the response was lost?
The client would see a timeout. The database would contain a successful order.
A second POST could then create another one.
I wanted to verify that failure window instead of treating it as a purely theoretical scenario. I used a small local HTTP server and SQLite database, delayed the server response until after the client timeout, and retried the same request.
This article follows that experiment and uses it to answer four questions:
- What does a client timeout actually tell us?
- Why can the server succeed when the client reports failure?
- Why is comparing request bodies not enough to detect a retry?
- What must an API remember before a repeated POST becomes safe?
The experiment is deliberately small. It does not model a production cluster, external payments, or concurrent requests arriving at exactly the same instant. Its purpose is to isolate one boundary that is easy to overlook: a business effect can become durable before the response becomes observable to the client.
The initial assumption
Application code often presents a remote call as a simple binary result.
response returned → success
exception thrown → failure
Inside one process, that model is often useful. If a local function throws before mutating state, the caller can usually reason about what happened from the call result and the language’s exception rules.
A network call has another boundary.
The caller observes messages crossing the network. It does not directly observe the server’s instruction pointer, transaction state, or durable storage.
When an HTTP client reports a timeout, it has established one fact:
No usable response arrived before the client’s deadline.
That fact does not identify where the request stopped.
The request might never have reached the server. The server might have rejected it before changing any state. The operation might still be running. The server might also have completed the operation, committed its database transaction, and failed only while returning the response.
Those paths are different inside the server, but they can be indistinguishable to the client.
That distinction matters because the correct next action is not the same in every case.
If the request never reached the server, retrying is necessary.
If the operation already committed, treating the retry as a new command can repeat the business effect.
Reproducing the failure window
To make the timing visible, I used a minimal local setup:
- an HTTP endpoint that accepts POST /orders;
- a SQLite table that stores each order;
- a server handler that commits the row immediately;
- a deliberate 1.5-second delay before returning the response;
- a client with a 0.4-second timeout;
- one retry after the first timeout.
Python and SQLite keep the experiment small, but neither is central to the result. The same ordering can occur with a Spring Boot service, PostgreSQL, a load balancer, and an HTTP client whenever the durable write completes before the response reaches the caller.
The important part of the server handler looked like this:
def do_POST(self):
body = self.rfile.read(content_length).decode("utf-8")
with sqlite3.connect(DB_PATH) as connection:
cursor = connection.execute(
"""
INSERT INTO orders (request_body, created_at)
VALUES (?, ?)
""",
(body, time.time()),
)
order_id = cursor.lastrowid
connection.commit()
log(f"server committed order id={order_id}")
# Deliberately slower than the client timeout.
time.sleep(1.5)
send_json_response(
status=201,
body={"orderId": order_id},
)
The delay is not intended as production code. It creates a deterministic interval between two events that are often discussed as though they happen together:
- the database transaction commits;
- the client receives the success response.
The client sent the request with a 0.4-second timeout and retried once after the timeout:
for attempt in (1, 2):
try:
send_post_request(
url="/orders",
body=order,
timeout=0.4,
)
except TimeoutError:
log(f"attempt {attempt} timed out")
The two attempts used exactly the same body.
{
"customerId": "customer-42",
"productId": "product-17",
"quantity": 1
}
What actually happened
The executed test produced this sequence:
00.008s client sends attempt 1
00.049s server committed order id=1
00.435s client attempt 1 ended with TimeoutError
00.535s client sends attempt 2
00.544s server committed order id=2
00.937s client attempt 2 ended with TimeoutError
01.550s server could not deliver response for order id=1
02.045s server could not deliver response for order id=2
02.939s database row count=2
The first row was durable roughly 386 milliseconds before the client reported the timeout.
That is the key observation.
The client-side exception did not roll back the server’s database transaction. The client and server had already crossed different boundaries:
server:
request received
→ row inserted
→ transaction committed
→ response delayed
client:
request sent
→ waiting
→ deadline reached
→ timeout reported
By the time the client entered its exception handler, the business effect already existed.
The retry did not “continue” the first HTTP request. It created a second request. The server had no information connecting the second delivery to the first operation, so it inserted another row.
From the HTTP server’s point of view, it had received two valid commands to create an order.
The duplicate was not caused by SQLite, Python, or the timeout itself. It was caused by the absence of a durable relationship between two request deliveries that represented one user intention.
What HTTP idempotency does—and does not—say
RFC 9110 defines an HTTP method as idempotent when multiple identical requests have the same intended effect on the server as one request. PUT, DELETE, and safe methods are idempotent under the HTTP specification. The property applies to the effect requested by the user; a server may still authenticate, log, measure, or internally process every delivery.
A PUT request often describes a target state.
PUT /profiles/42/preferences
Content-Type: application/json
{
"language": "en",
"darkMode": true
}
If the request is applied once, the preferences have those values.
If the same request is applied again, the intended state remains the same.
A typical resource-creation POST expresses something different.
POST /orders
Content-Type: application/json
{
"productId": "product-17",
"quantity": 1
}
Processing this request twice can reasonably mean “create two orders.” HTTP does not know that the second delivery came from a client retry rather than a second purchase.
For that reason, RFC 9110 says that a client should not automatically retry a non-idempotent request unless it has another way to know that the request semantics are idempotent or that the original request was never applied.
The method name is therefore only the beginning of the analysis.
POST does not make safe retries impossible. It means the application must define additional semantics that HTTP does not provide automatically.
The request body cannot identify the operation
The two requests in the experiment had identical JSON.
It may seem reasonable to hash the body and treat a matching hash as proof of a retry.
That would reject legitimate operations.
A customer may intentionally order the same item twice. A deployment system may intentionally create two workers from the same configuration. A payroll system may send the same amount to the same employee in different periods.
The payload describes the operation’s input. It does not always identify the user’s intention.
The opposite problem also occurs. A retry of the same logical operation may contain a different timestamp, trace field, client version, or metadata value.
This gives us two separate concepts:
request content
operation identity
Request content answers:
What data should the server use?
Operation identity answers:
Do these two deliveries represent the same intended action?
Trying to derive the second entirely from the first makes the API guess at business intent.
A better contract lets the client state that intent explicitly.
Adding an operation identity
A common design adds a client-generated request identifier.
POST /orders
Idempotency-Key: order-attempt-20260726-001
Content-Type: application/json
{
"customerId": "customer-42",
"productId": "product-17",
"quantity": 1
}
The client creates one identifier for one logical operation.
A retry reuses it.
A new order receives a new identifier, even if its body is identical.
AWS describes the same idea as a unique client request identifier. Google AIP-155 uses a request_id field and states that providing it must make the request idempotent: a detected duplicate should not create another effect and should normally return the result of the earlier successful request.
The literal name Idempotency-Key should not be confused with a finalized universal HTTP standard. An IETF working-group draft proposed such a header, but its latest revision expired on April 18, 2026 and is not an RFC. An API that supports the header must still publish its own contract.
For the second experiment, I stored a small idempotency record alongside the order.
idempotency_key
request_body
order_id
created_at
The server’s behavior was:
if the key has not been seen:
create the order
store the key and order ID
commit
if the key already exists:
verify that the request matches
return the stored order ID
do not create another order
The first request still delayed its response long enough for the client to time out.
The retry reused the same key.
The output changed:
00.008s client sends attempt 1
00.035s server committed new order id=1
00.429s client attempt 1 ended with TimeoutError
00.529s client sends attempt 2 with the same key
00.532s server replayed order id=1
00.534s client received:
{"orderId": 1, "replayed": true}
02.035s database order row count=1
The timeout still happened.
The first server response was still lost.
The network did not become more reliable.
What changed was the server’s interpretation of the second delivery.
Without an operation identity:
second delivery → new command → second order
With an operation identity:
second delivery → known operation → previous result
This is the practical value of idempotency in a retryable API.
The experiment proves less than a production design needs
The second test demonstrates safe sequential replay, but it does not prove that the implementation is safe under concurrency.
Consider this algorithm:
1. Check whether the key exists.
2. If it does not, create the order.
3. Save the key.
Two requests can both complete step 1 before either completes step 3.
Request A: key not found
Request B: key not found
Request A: create order 1
Request B: create order 2
The duplicate check is present, but it does not protect the gap between checking and writing.
A production design must make the ownership decision atomic. Depending on the datastore, this may use a unique constraint, conditional write, transaction, compare-and-set operation, or another concurrency-control mechanism.
AWS’s current Well-Architected guidance explicitly connects idempotency state with concurrency control. It recommends storing the token and operation state, and maintaining consistency and atomicity between recording the token and running the associated mutations.
The record also needs more than a boolean flag.
A realistic state model may include:
PROCESSING
SUCCEEDED
FAILED
EXPIRED
If a duplicate arrives while the first request is still running, the server needs a documented response. It might return an operation resource, wait for a bounded interval, or report that the request is already in progress.
If the original request completed, the server needs to decide what to replay.
Stripe stores the status code and body from the first executed request, including a stored 500 response, and returns the same result for subsequent uses of the key. It also compares incoming parameters and rejects incompatible reuse. Stripe may remove a key after it is at least 24 hours old. These are concrete product decisions, not universal rules.
Google’s guidance permits returning the current state of a resource when reproducing the exact historical success response is no longer practical.
Both are valid contracts if they are deliberate and documented.
The difficult part is not generating a UUID.
The difficult part is defining what the identifier means throughout the operation’s lifetime.
The database is not always the end of the operation
The reproduction wrote to one SQLite database. That gives it a clear local consistency boundary.
Real operations often include effects outside that database:
create order
charge payment provider
reserve inventory
publish event
send confirmation email
A local transaction cannot normally roll back an HTTP call that already succeeded in another service.
Even if a unique constraint prevents two order rows, it does not prove that the payment provider was charged once or that one event was published.
The idempotency strategy must follow the business effect across every boundary that matters.
That may require:
- propagating the same operation identifier downstream;
- a transactional outbox;
- idempotent message consumers;
- explicit workflow states;
- reconciliation with an external provider;
- or compensating actions.
Those mechanisms deserve separate treatment. The experiment in this article establishes the problem that makes them necessary, but it does not claim to solve distributed atomicity.
Retry policy remains a separate decision
An idempotent endpoint can make duplicate delivery safer. It does not make every retry useful or free.
Retries still consume connections, threads, CPU, database capacity, and network bandwidth. Aggressive clients can amplify an outage.
Microsoft recommends retrying only faults that are likely to be transient, limiting the number of attempts, coordinating timeouts with the overall deadline, and using backoff and jitter where appropriate. It also warns against uncoordinated retry layers: three attempts at one layer and three attempts at another can produce nine calls against the underlying service.
The design therefore has to answer two different questions.
Could another attempt succeed?
That is a retry-policy question.
Would another delivery repeat an effect that may already exist?
That is an idempotency question.
A reliable system needs both answers.
What this investigation changed
The original binary model was too simple:
response → success
exception → failure
The more accurate model includes an unknown state:
confirmed success
confirmed failure
outcome unknown
A timeout often belongs to the third category.
Once that category is visible, the rest of the design follows more clearly.
A retry is not a continuation of the original request. It is another delivery.
The server cannot infer business identity from a matching JSON body.
A client-generated operation identifier can connect the deliveries, but the server must reserve it atomically, validate its reuse, retain enough state to answer duplicates, and document when the guarantee expires.
The experiment also shows why “we added a retry loop” and “the operation is safely retryable” are very different statements.
The network may deliver the same intention more than once.
The API has to decide whether that produces another effect.
Limitations and follow-up questions
This experiment used a single process and one SQLite database. It demonstrated the timing boundary and sequential replay, not a complete distributed implementation.
The next questions are more difficult:
- How should two concurrent requests reserve the same key?
- Should failed results be cached and replayed?
- How long should the key remain valid?
- What happens if the process crashes after the key is stored but before the result is complete?
- How should the identifier cross payment, inventory, and messaging boundaries?
- When does idempotency provide at-most-once effects, and when is reconciliation still required?
Those questions begin where this experiment ends.
References
- RFC 9110: HTTP Semantics — RFC Editor / IETF
- Making Retries Safe with Idempotent APIs — Amazon Web Services
- AIP-155: Request Identification — Google
- Idempotent Requests — Stripe
- REL04-BP04: Make Mutating Operations Idempotent — Amazon Web Services
- Best Practices for Transient Fault Handling — Microsoft
- The Idempotency-Key HTTP Header Field, Draft 07 — IETF HTTPAPI Working Group
