← all insights

#distributed-systems

Idempotency is a feature, not a fix

2026-01-09

Retries are normal. Networks drop, timeouts fire, and a client that didn't get a response will try again. The problem isn't the retry; it's that most systems assume every request arrives exactly once. They don't.

The cost of a duplicate

A payment charged twice. An order created twice. A webhook that fires an email to a customer ten times. These aren't edge cases we've read about; they're Tuesdays. And they're almost always caused by the same thing: a write that isn't safe to repeat.

Idempotency keys

The fix is boring and effective. The client generates a unique key per logical operation and sends it along. The server records the key and the result; if the same key shows up again, it returns the original result instead of doing the work twice.

// Express-style handler backed by Redis
app.post('/payments', async (req, res) => {
  const key = req.header('Idempotency-Key');
  const cached = await redis.get('pay:' + key);
  if (cached) return res.json(JSON.parse(cached));

  const result = await charge(req.body);
  await redis.set('pay:' + key, JSON.stringify(result), 'EX', 86400);
  res.json(result);
});

A few things that matter

  • The key is generated by the client, not the server. The client controls what 'once' means.
  • Store it with a TTL long enough to cover the retry window. A day is usually plenty.
  • Scope it to the user and the operation, so one customer's key can't collide with another's.

Done well, idempotency stops being a fire drill and becomes a property of the system. That's the difference between a fix and a feature.