Troubleshooting

1. Verify Endpoint URL

Ensure the endpoint URL is correct and publicly accessible.

• URL must start with https:// (HTTP is not allowed in production)
• Domain must be publicly resolvable (not localhost or private IP)
• Path must match your webhook handler route exactly
• Test URL accessibility with curl or similar tool

2. Check Endpoint Status

Verify the endpoint is active in the Hook Mesh dashboard.

• Navigate to Applications → Your App → Endpoints
• Check endpoint status is "active" (not "paused" or "disabled")
• Verify event type subscriptions are configured

3. Review Webhook Job Status

Check if the webhook job was created and what happened during delivery.

# Get webhook job details
curl https://api.hookmesh.com/v1/webhook-jobs/{job_id} \
  -H "Authorization: Bearer YOUR_API_KEY"

# Check delivery attempts
curl https://api.hookmesh.com/v1/webhook-jobs/{job_id}/attempts \
  -H "Authorization: Bearer YOUR_API_KEY"

4. Check Firewall and Security Groups

Ensure your infrastructure allows inbound HTTPS requests from Hook Mesh.

• Verify firewall rules allow inbound HTTPS (port 443)
• Check security groups allow traffic from Hook Mesh IP ranges
• Ensure no reverse proxy is blocking or filtering requests
• Review WAF (Web Application Firewall) rules for blocks

5. Review Application Logs

Check if your application is receiving the request but not logging it.

• Search application logs for webhook endpoint path
• Look for requests with User-Agent containing HookMesh
• Check for any middleware rejecting requests early

6. Check Circuit Breaker Status

The circuit breaker may have opened due to previous failures.

# Check endpoint circuit breaker status
curl https://api.hookmesh.com/v1/endpoints/{endpoint_id} \
  -H "Authorization: Bearer YOUR_API_KEY"

# Look for:
# - circuit_breaker_state: "open" (deliveries paused)
# - consecutive_failures: > 5

If the circuit is open, fix the underlying issue and reset the circuit breaker.

1. Verify You're Using the Correct Secret

Each endpoint has a unique secret. Make sure you're using the right one.

• Find secret in dashboard: Applications → Endpoints → "Reveal Secret"
• Secret format: whsec_...
• Check if secret was recently rotated (old secret may still be valid)
• Ensure environment variable is set correctly

2. Check Timestamp Tolerance

Webhooks older than 5 minutes are rejected to prevent replay attacks.

• Verify server clock is synchronized with NTP
• Check for timezone issues (use UTC timestamps)
• Ensure timestamp validation uses correct epoch format (seconds, not milliseconds)

3. Use Request Body As-Is

The signature is computed over the raw request body. Don't parse or reformat it.

// ✓ CORRECT - Use raw body for signature verification
app.post('/webhooks', express.raw({ type: 'application/json' }), (req, res) => {
  const rawBody = req.body.toString('utf8');
  const webhookId = req.headers['webhook-id'];
  const timestamp = req.headers['webhook-timestamp'];
  const signature = req.headers['webhook-signature'];

  verifySignature(rawBody, webhookId, timestamp, signature, secret);

  // Parse after verification
  const payload = JSON.parse(rawBody);
});

// ✗ INCORRECT - Don't parse first
app.post('/webhooks', express.json(), (req, res) => {
  // req.body is already parsed - signature will fail
  // JSON.stringify may produce different formatting than the original
  const signature = req.headers['webhook-signature'];
  verifySignature(JSON.stringify(req.body), signature, secret); // Wrong!
});

4. Verify Algorithm (HMAC-SHA256)

Ensure you're using the correct hashing algorithm.

// Correct signature verification algorithm
const webhookId = headers['webhook-id'];
const timestamp = headers['webhook-timestamp'];
const signature = headers['webhook-signature'];

// Extract signature (format: "v1,<signature>")
const [version, expectedSig] = signature.split(',');
if (version !== 'v1') {
  throw new Error('Invalid signature version');
}

// Construct signed content: {webhookId}.{timestamp}.{jsonPayload}
const signedContent = `${webhookId}.${timestamp}.${rawBody}`;

// Compute HMAC-SHA256 and encode as base64
const hmac = crypto.createHmac('sha256', secret);
hmac.update(signedContent);
const computedSig = hmac.digest('base64');

// Timing-safe comparison
if (!crypto.timingSafeEqual(Buffer.from(expectedSig), Buffer.from(computedSig))) {
  throw new Error('Invalid signature');
}

5. Debug: Log Signature Components

Add temporary logging to compare expected vs. computed signatures.

const webhookId = headers['webhook-id'];
const timestamp = headers['webhook-timestamp'];
const expectedSig = headers['webhook-signature'].split(',')[1];

console.log('Webhook ID:', webhookId);
console.log('Timestamp:', timestamp);
console.log('Body length:', rawBody.length);
console.log('Expected signature:', expectedSig);

const signedContent = `${webhookId}.${timestamp}.${rawBody}`;
const computedSig = crypto.createHmac('sha256', secret)
  .update(signedContent)
  .digest('base64');

console.log('Computed signature:', computedSig);
console.log('Signatures match:', expectedSig === computedSig);

1. Measure Response Time

Add timing instrumentation to identify slow operations.

app.post('/webhooks', async (req, res) => {
  const start = Date.now();

  try {
    await processWebhook(req.body);
    const duration = Date.now() - start;
    console.log(`Webhook processed in ${duration}ms`);

    if (duration > 5000) {
      console.warn('Webhook processing is slow - consider offloading to queue');
    }

    res.status(200).json({ received: true });
  } catch (error) {
    console.error('Processing failed:', error);
    res.status(500).json({ error: error.message });
  }
});

2. Offload Heavy Processing to Job Queue

Webhook handlers should acknowledge receipt quickly, then process asynchronously.

import { Queue } from 'bullmq';

const webhookQueue = new Queue('webhooks', {
  connection: { host: 'localhost', port: 6379 }
});

app.post('/webhooks', async (req, res) => {
  const { event_id, event_type, payload } = req.body;

  // 1. Verify signature (fast)
  verifySignature(req.body, req.headers, secret);

  // 2. Queue for processing (fast)
  await webhookQueue.add('process-webhook', {
    event_id,
    event_type,
    payload
  });

  // 3. Return immediately
  res.status(200).json({ received: true, queued: true });
});

// Process webhooks asynchronously in worker
const worker = new Worker('webhooks', async (job) => {
  const { event_id, event_type, payload } = job.data;

  // Heavy processing happens here (no timeout pressure)
  await processEvent(event_type, payload);
  await sendNotifications(payload);
  await updateAnalytics(event_id);
});

3. Check Database Connection Pool

Exhausted connection pools can cause delays.

• Verify pool size is adequate for concurrent webhooks
• Check for connection leaks (connections not released)
• Monitor connection wait times
• Use connection timeouts to fail fast

4. Monitor Server Resources

Resource exhaustion can slow down webhook processing.

• Check CPU usage (high CPU can delay processing)
• Monitor memory usage (swapping kills performance)
• Review disk I/O (slow disk can delay database operations)
• Check network latency to external services

5. Review Slow Queries

Database queries should complete in milliseconds, not seconds.

• Enable slow query logging
• Add indexes on frequently queried columns
• Optimize N+1 query problems
• Consider read replicas for heavy read operations

1. Check Response Status Code

Your endpoint must return a 2xx status code to indicate success.

// ✓ CORRECT - Return 200-299 for success
app.post('/webhooks', async (req, res) => {
  try {
    await processWebhook(req.body);
    res.status(200).json({ received: true }); // Hook Mesh stops retrying
  } catch (error) {
    console.error('Processing failed:', error);
    res.status(500).json({ error: error.message }); // Hook Mesh will retry
  }
});

// ✗ INCORRECT - Don't return 3xx, 4xx for transient errors
res.status(400).json({ error: 'Processing failed' }); // No retry!

2. Implement Idempotency

Ensure your endpoint can safely handle duplicate deliveries.

See Idempotency guide for implementation details.

3. Check Application Logs for Errors

Review error logs to identify the root cause of failures.

• Search logs for webhook endpoint errors
• Look for database constraint violations
• Check for exceptions in business logic
• Review external API call failures

4. Review Error Rates in Dashboard

The Hook Mesh dashboard shows delivery success/failure rates.

• Navigate to Applications → Endpoints → View metrics
• Check delivery success rate (should be near 100%)
• Review recent failed deliveries and error messages
• Look for patterns in failures (specific event types, time of day)

5. Check Circuit Breaker Threshold

If failures continue, the circuit breaker will open after 5 consecutive failures.

See Circuit Breaker guide for details on how this works.

1. Verify Event Subscriptions

Endpoints only receive events they're subscribed to.

• Check endpoint configuration for event type subscriptions
• Ensure event type matches exactly (case-sensitive)
• Verify endpoint is subscribed to the specific event (e.g., user.created)

2. Verify Application ID

Webhook jobs must be created for the correct application.

// Ensure you're using the correct application_id
const response = await fetch('https://api.hookmesh.com/v1/webhook-jobs', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.HOOKMESH_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    application_id: 'app_abc123', // Must match endpoint's application
    event_type: 'user.created',
    payload: { user_id: 'usr_456', email: 'alice@example.com' }
  })
});

3. Check Rate Limits

If you're hitting rate limits, webhook jobs may be rejected.

• Review API responses for 429 (Too Many Requests) status codes
• Check your plan's rate limit in the dashboard
• Implement exponential backoff for API calls
• Consider upgrading plan for higher limits

4. Review Webhook Job Creation Logs

Check if the webhook job was successfully created.

// Log webhook job creation
const response = await fetch('https://api.hookmesh.com/v1/webhook-jobs', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.HOOKMESH_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({ /* ... */ })
});

if (!response.ok) {
  const error = await response.json();
  console.error('Failed to create webhook job:', error);
  // Check error.message for details
} else {
  const job = await response.json();
  console.log('Webhook job created:', job.id);
}

5. Check for API Errors

API errors can prevent webhook job creation.

• Verify API key is valid and has correct permissions
• Check for validation errors in request payload
• Ensure application_id and event_type are valid
• Review API error responses for specific error messages

1. Implement Idempotency

Use the event_id to track processed webhooks.

See the Idempotency guide for complete implementation examples.

2. Check Retry Configuration

Review Hook Mesh's retry behavior in the dashboard.

See Retry Strategy for details on retry timing and backoff.

3. Review Delivery Attempts

Check how many times a webhook was delivered in the dashboard.

# View all delivery attempts for a webhook job
curl https://api.hookmesh.com/v1/webhook-jobs/{job_id}/attempts \
  -H "Authorization: Bearer YOUR_API_KEY"

# Response includes:
# - attempt_number
# - status_code
# - response_time_ms
# - error_message (if failed)

1. Check Status Page

Visit status.hookmesh.com to see if there are any ongoing incidents or maintenance affecting webhook delivery.

2. Review Documentation

Check related guides for more detailed information:

• Retry Strategy
• Circuit Breaker
• Verifying Signatures
• Idempotency

3. Contact Support

When contacting support, include the following information:

• Webhook job ID (format: job_...)
• Timestamp of the issue (include timezone)
• Endpoint URL (or endpoint ID)
• Error messages from your application logs
• Response status codes and error details from Hook Mesh dashboard
• Steps you've already taken to troubleshoot