Webhook Idempotency: Why It Matters and How to Implement It

Your webhook endpoint processes a payment, then the same payment again—now your customer is charged twice. Idempotency prevents this scenario. It's critical for webhook architecture yet often overlooked until failures occur.

What Is Idempotency?

An operation is idempotent when performing it multiple times produces the same result as doing it once. For webhooks: processing the same event twice should have exactly the same effect as processing it once.

Examples:

Idempotent operations:

Setting a user's email to "user@example.com" (doing it twice still results in the same email)
Marking an order as "shipped" (it's still shipped after multiple updates)
Recording a payment with a specific transaction ID (the record already exists)

Non-idempotent operations:

Incrementing an account balance by $50 (each call adds another $50)
Sending a notification email (each call sends another email)
Creating a new order record (each call creates a duplicate)

The distinction matters because webhooks are delivered with "at-least-once" semantics, not "exactly-once."

Why Webhooks Need Idempotency

Webhook providers guarantee delivery with a caveat: they may deliver the same event multiple times due to:

Network uncertainty: Response gets lost so sender retries. Timeout handling: Slow responses trigger retries despite success. Infrastructure retries: Load balancers/CDNs retry automatically. Exponential backoff: Recovery during retries delivers both original and retry.

Studies show 2-5% of deliveries retry. Your code must handle duplicates gracefully.

Real damage from missing idempotency:

Double-charging customers, duplicate refunds, wrong balances
Shipping same order twice, inventory errors
Duplicate accounts, multiple welcome emails
Discounts applied multiple times, billing errors

These happen daily in systems without proper idempotency.

Idempotency Keys: Your First Line of Defense

An idempotency key is a unique identifier that represents a specific event or operation. By tracking which keys you've already processed, you can safely ignore duplicates.

Types of Idempotency Keys

Event IDs: Most webhook providers include a unique event ID in each payload. Stripe uses evt_, GitHub uses delivery GUIDs, and Shopify includes X-Shopify-Webhook-Id headers.

Natural keys: Sometimes the payload itself contains a natural unique identifier—a transaction ID, order number, or combination of fields that uniquely identifies the event.

Computed keys: When no unique identifier exists, you can generate one by hashing the payload contents or combining multiple fields.

Idempotency Key Headers

Hook Mesh includes an X-HookMesh-Idempotency-Key header with every webhook delivery. This key remains consistent across retries, making it trivial to implement deduplication:

app.post('/webhooks', (req, res) => {
  const idempotencyKey = req.headers['x-hookmesh-idempotency-key'];

  // Use this key for deduplication
  if (await hasProcessedKey(idempotencyKey)) {
    return res.status(200).json({ status: 'already_processed' });
  }

  // Process the webhook...
});

This works consistently regardless of the original webhook source.

Deduplication Strategies

Strategy 1: Database Unique Constraints

The simplest approach uses database constraints to prevent duplicate records. This works well when webhook processing creates new records.

Node.js with PostgreSQL:

const { Pool } = require('pg');
const pool = new Pool();

async function processPaymentWebhook(event) {
  const { id: eventId, data } = event;
  const { transaction_id, amount, customer_id } = data;

  try {
    await pool.query(`
      INSERT INTO processed_payments
        (event_id, transaction_id, amount, customer_id, processed_at)
      VALUES ($1, $2, $3, $4, NOW())
      ON CONFLICT (event_id) DO NOTHING
    `, [eventId, transaction_id, amount, customer_id]);

    // Check if the insert actually happened
    const result = await pool.query(
      'SELECT * FROM processed_payments WHERE event_id = $1',
      [eventId]
    );

    if (result.rows[0].processed_at < new Date(Date.now() - 1000)) {
      console.log(`Event ${eventId} was already processed, skipping`);
      return { duplicate: true };
    }

    // Continue with side effects (emails, fulfillment, etc.)
    await fulfillOrder(transaction_id);
    return { duplicate: false };

  } catch (error) {
    if (error.code === '23505') { // Unique violation
      console.log(`Duplicate event ${eventId} detected`);
      return { duplicate: true };
    }
    throw error;
  }
}

Python with SQLAlchemy:

from sqlalchemy.exc import IntegrityError
from datetime import datetime

def process_payment_webhook(event):
    event_id = event['id']
    data = event['data']

    try:
        payment = ProcessedPayment(
            event_id=event_id,
            transaction_id=data['transaction_id'],
            amount=data['amount'],
            customer_id=data['customer_id'],
            processed_at=datetime.utcnow()
        )
        db.session.add(payment)
        db.session.commit()

        # Successfully inserted, continue processing
        fulfill_order(data['transaction_id'])
        return {'duplicate': False}

    except IntegrityError:
        db.session.rollback()
        print(f'Duplicate event {event_id} detected')
        return {'duplicate': True}

Strategy 2: Redis-Based Deduplication

For high-throughput systems, checking a database for every webhook adds latency. Redis provides faster lookups with automatic expiration.

Node.js with Redis:

const Redis = require('ioredis');
const redis = new Redis();

async function processWithRedisDedup(event, processingFn) {
  const eventId = event.id;
  const lockKey = `webhook:processed:${eventId}`;
  const lockTTL = 86400; // 24 hours

  // Try to set the key only if it doesn't exist (NX)
  const acquired = await redis.set(lockKey, Date.now(), 'NX', 'EX', lockTTL);

  if (!acquired) {
    console.log(`Event ${eventId} already processed`);
    return { duplicate: true };
  }

  try {
    await processingFn(event);
    return { duplicate: false };
  } catch (error) {
    // Processing failed, remove the key to allow retry
    await redis.del(lockKey);
    throw error;
  }
}

// Usage
app.post('/webhooks', async (req, res) => {
  const event = req.body;

  const result = await processWithRedisDedup(event, async (evt) => {
    await handlePaymentEvent(evt);
  });

  res.json({ processed: !result.duplicate });
});

Python with Redis:

import redis
import time

redis_client = redis.Redis()

def process_with_redis_dedup(event, processing_fn):
    event_id = event['id']
    lock_key = f"webhook:processed:{event_id}"
    lock_ttl = 86400  # 24 hours

    # Try to acquire lock
    acquired = redis_client.set(lock_key, time.time(), nx=True, ex=lock_ttl)

    if not acquired:
        print(f'Event {event_id} already processed')
        return {'duplicate': True}

    try:
        processing_fn(event)
        return {'duplicate': False}
    except Exception as e:
        # Processing failed, allow retry
        redis_client.delete(lock_key)
        raise

Real-World Scenarios

Payment Webhooks

Payment processing demands bulletproof idempotency. A duplicate charge damages customer trust and triggers chargebacks.

async function handlePaymentSucceeded(paymentIntent) {
  const paymentId = paymentIntent.id;

  // Use transaction to ensure atomicity
  await db.transaction(async (trx) => {
    // Check if already processed
    const existing = await trx('payments')
      .where('stripe_payment_id', paymentId)
      .first();

    if (existing) {
      console.log(`Payment ${paymentId} already recorded`);
      return;
    }

    // Record the payment
    await trx('payments').insert({
      stripe_payment_id: paymentId,
      amount: paymentIntent.amount,
      customer_id: paymentIntent.customer,
      status: 'completed',
      created_at: new Date()
    });

    // Update account balance (idempotent because we checked first)
    await trx('accounts')
      .where('stripe_customer_id', paymentIntent.customer)
      .increment('balance', paymentIntent.amount);
  });

  // Side effects outside transaction
  await sendReceiptEmail(paymentIntent);
}

Order Processing

Order webhooks often trigger fulfillment, inventory updates, and notifications. Each must be idempotent.

def handle_order_created(order_data):
    order_id = order_data['id']

    # Idempotent order creation
    order, created = Order.objects.get_or_create(
        external_id=order_id,
        defaults={
            'customer_id': order_data['customer_id'],
            'total': order_data['total'],
            'status': 'pending'
        }
    )

    if not created:
        print(f'Order {order_id} already exists')
        return

    # Reserve inventory (check current state, not just decrement)
    for item in order_data['items']:
        Inventory.objects.filter(
            sku=item['sku'],
            reserved_for_order__isnull=True
        ).update(reserved_for_order=order.id)

    # Queue fulfillment (worker checks order status before processing)
    fulfillment_queue.enqueue(order.id)

User Creation

Duplicate user webhooks can create multiple accounts or send redundant welcome emails.

async function handleUserCreated(userData) {
  const externalId = userData.id;

  // Upsert pattern - idempotent by design
  const user = await db.users.upsert({
    where: { externalId },
    create: {
      externalId,
      email: userData.email,
      name: userData.name,
      createdAt: new Date(),
      welcomeEmailSent: false
    },
    update: {} // Don't update if exists
  });

  // Only send welcome email once
  if (!user.welcomeEmailSent) {
    await sendWelcomeEmail(user);
    await db.users.update({
      where: { id: user.id },
      data: { welcomeEmailSent: true }
    });
  }
}

Common Mistakes and How to Avoid Them

Mistake 1: Checking After Side Effects

// WRONG: Side effect happens before duplicate check
async function handleWebhook(event) {
  await sendNotificationEmail(event.user); // Oops, already sent!

  const existing = await db.events.findOne({ id: event.id });
  if (existing) return;

  await db.events.create({ id: event.id });
}

Fix: Always check for duplicates before any side effects.

Mistake 2: Non-Atomic Operations

// WRONG: Race condition between check and insert
async function handleWebhook(event) {
  const existing = await db.events.findOne({ id: event.id });
  if (existing) return;

  // Another request could insert here!

  await db.events.create({ id: event.id });
}

Fix: Use database transactions, unique constraints, or atomic Redis operations.

Mistake 3: Short TTLs on Deduplication Keys

If your Redis TTL is 1 hour but webhooks can be retried for 72 hours, late retries will be processed as new events. This is especially problematic when webhooks end up in a dead letter queue and are replayed much later.

Fix: Set TTLs longer than the maximum retry window of your webhook providers.

Mistake 4: Forgetting Side Effect Idempotency

Even with event deduplication, your side effects must be idempotent. Sending emails, updating inventory, and calling external APIs all need their own idempotency guarantees.

Fix: Track completion of each side effect separately, or use idempotent API calls where available.

Conclusion

Webhook idempotency is fundamental—not optional. At-least-once delivery means duplicates are inevitable, and your code must handle them. This connects to webhook ordering guarantees—both handle distributed system realities.

Identify the idempotency key for each webhook type. Implement deduplication using database constraints, Redis, or hybrid approaches. Ensure all side effects are idempotent. Test by deliberately sending duplicates.

Invest in idempotency today to prevent bugs, data inconsistencies, and late-night debugging tomorrow. Build it right and integrations scale reliably. See webhook reliability engineering for comprehensive patterns.

Webhook Idempotency: Why It Matters and How to Implement It

Webhook Idempotency: Why It Matters and How to Implement It

What Is Idempotency?

Why Webhooks Need Idempotency

Idempotency Keys: Your First Line of Defense

Types of Idempotency Keys

Idempotency Key Headers

Deduplication Strategies

Strategy 1: Database Unique Constraints

Strategy 2: Redis-Based Deduplication

Real-World Scenarios

Payment Webhooks

Order Processing

User Creation

Common Mistakes and How to Avoid Them

Mistake 1: Checking After Side Effects

Mistake 2: Non-Atomic Operations

Mistake 3: Short TTLs on Deduplication Keys

Mistake 4: Forgetting Side Effect Idempotency

Conclusion

Related Posts

Webhook Retry Strategies: Linear vs Exponential Backoff

Why You Shouldn't Rely on Webhook Ordering

Dead Letter Queues for Failed Webhooks: A Complete Technical Guide

How to Receive Stripe Webhooks Reliably

Webhook Reliability Engineering: The Complete Guide