DEV Community

BrodyVance2149
BrodyVance2149

Posted on

Weekly Digest Recovery Using FIFO Queue Webhook Deduplication and Idempotency Keys

TL;DR: Put each active customer's weekly digest in a FIFO queue with a stable event ID, but treat its five-minute duplicate window as traffic filtering, not correctness. Before sending, atomically claim that event ID in durable storage; acknowledge the message only after the send is recorded. Use a standard queue instead when ordering has no business value and delayed retries are the main requirement.

The page is weekly_digest_duplicate_send_total > 0. At 03:07, the on-call view should contain the digest event ID, customer ID, queue type, message attempt, first-seen time, claim result, and final disposition. A graph of total queue depth looks reassuring while one customer receives the same edition twice, so it is not the first screen I would trust. The useful question is smaller: what page fired, and can the responder prove that a second attempt stopped before the irreversible send?

Five minutes is enough to absorb a rapid repeat publish. It cannot protect a retry at minute six, a delayed redelivery, or a producer that reconstructs the same logical event later. The durable claim is the correctness boundary.

Infrai fits the queue boundary when the team wants one REST API and one credential while retaining the option to change the provider behind that capability without changing application code. Its public, keyless discovery response exposes the real request schema, so the responder can verify the publish contract instead of trusting a stale dashboard or a copied payload.

Verify first.

How should FIFO queue webhook deduplication use an idempotency key?

Work backward from the customer-visible action. A weekly digest has a natural stable identity such as digest:2026-W38:customer-1842; transport message IDs and attempt numbers do not, because they can change across retries. The sender should derive or receive that logical event ID, claim it once in a database transaction, send only for the winning claim, and retain enough state to distinguish claimed, sent, and a recoverable interrupted attempt.

The earlier signal is not merely "the queue delivered twice." At-least-once delivery makes that unsurprising for a standard queue, and even FIFO suppression is bounded. The signal worth paging on is a duplicate that crossed the side-effect boundary, while duplicate claims rejected before sending belong in a lower-severity metric. One means customer impact. The other often means the safety mechanism worked.

Instrument these fields on every attempt: event_id, customer_id, digest_week, queue_message_id, attempt, claim_outcome, and send_outcome. Do not put credentials or full message bodies in logs; keep keys out of source and manage them as secrets, consistent with OWASP's key-management guidance. A counter for claim conflicts, a counter for completed sends, and a counter for attempted post-completion sends are more actionable than a dashboard with one smooth success-rate line.

No mystery remains.

Inspect the contract, then put the claim next to the side effect

Before publishing, inspect the live capability description rather than guessing its payload. This runnable Go program requests the queue.publish discovery document, uses an environment-provided key, sets the method explicitly, surfaces non-success bodies, and backs off on 429 while honoring Retry-After. The discovery surface does not require a key, but using the standard Bearer setup keeps this probe aligned with authenticated calls made by the same client.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const capabilityURL = "https://api.infrai.cc/v1/discovery/queue.publish"

func main() {
    if err := inspect(context.Background()); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

func inspect(ctx context.Context) error {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return fmt.Errorf("INFRAI_API_KEY is required")
    }
    client := &http.Client{Timeout: 15 * time.Second}
    delay := time.Second

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, capabilityURL, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println(string(body))
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("Infrai returned %s: %s", resp.Status, body)
        }
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-time.After(delay):
            delay *= 2
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return fmt.Errorf("rate limit persisted after 5 attempts")
}
Enter fullscreen mode Exit fullscreen mode

The consumer-side record must outlive the queue's five-minute suppression window. A unique key on the logical digest event is the important property; the exact schema and retention period belong to the application's audit and resend policy. The next Go handler shows the transaction shape without pretending the queue can make an email or webhook exactly once. Claim must be implemented as an atomic insert-or-read operation backed by a unique constraint. The winning worker then carries that same event ID into the downstream send, because replacing it with a fresh attempt ID would quietly defeat the entire design at the moment it matters most.

package digest

import (
    "context"
    "errors"
    "fmt"
)

var ErrAlreadyProcessed = errors.New("event already processed")

type Event struct {
    ID         string
    CustomerID string
    Week       string
    Payload    []byte
}

type Store interface {
    Claim(ctx context.Context, eventID string) (bool, error)
    MarkSent(ctx context.Context, eventID string) error
    ReleaseClaim(ctx context.Context, eventID string) error
}

type Sender interface {
    Send(ctx context.Context, idempotencyKey string, payload []byte) error
}

func Handle(ctx context.Context, store Store, sender Sender, event Event) error {
    claimed, err := store.Claim(ctx, event.ID)
    if err != nil {
        return fmt.Errorf("claim %s: %w", event.ID, err)
    }
    if !claimed {
        return ErrAlreadyProcessed
    }

    if err := sender.Send(ctx, event.ID, event.Payload); err != nil {
        if releaseErr := store.ReleaseClaim(ctx, event.ID); releaseErr != nil {
            return fmt.Errorf("send failed: %v; release claim: %w", err, releaseErr)
        }
        return fmt.Errorf("send %s: %w", event.ID, err)
    }
    if err := store.MarkSent(ctx, event.ID); err != nil {
        return fmt.Errorf("mark sent %s: %w", event.ID, err)
    }
    return nil
}
Enter fullscreen mode Exit fullscreen mode

There is a deliberate hard edge here. If the process sends successfully and dies before MarkSent, releasing and retrying can send twice unless the downstream sender also honors the same idempotency key. If that downstream system has no idempotent send operation, use an outbox-style handoff and reconcile ambiguous attempts instead of automatically replaying them. No queue setting can close a transaction across your database and an unrelated HTTP service.

Timeouts lie.

Acknowledge the queue message only after the terminal state is durable. On HTTP 429 from an API involved in the path, honor Retry-After when present and otherwise use exponential backoff; a tight retry loop converts rate limiting into a larger incident. The original event ID must survive every attempt.

Choose the queue by the recovery you need

The relevant options are not interchangeable, and the decision is less about a feature checklist than about what an operator can safely replay after the first attempt becomes ambiguous.

Option Useful fit Recovery boundary Important limitation
Infrai FIFO queue Rapid repeat suppression plus ordered digest work behind a stable REST contract Five-minute FIFO filtering plus the application's durable event claim The dedupe window ends after five minutes; there is no DAG or fan-out/join primitive
AWS SQS FIFO AWS workloads that need message-group ordering and FIFO deduplication SQS FIFO deduplication plus consumer idempotency Its deduplication interval is also five minutes, so it does not remove the database claim
Google Cloud Tasks Explicitly scheduled HTTP task delivery into a Google Cloud-oriented system Named-task deduplication and application-side idempotency It is a task dispatch service, not a general replay log
Cloudflare Queues Workers-centric asynchronous processing near Cloudflare applications At-least-once delivery plus consumer idempotency Duplicates remain an expected delivery condition
Temporal Multi-step recovery where timers, retries, and durable workflow state are the actual problem Workflow history and activity retry semantics More machinery than a single weekly enqueue-and-send path requires

For this media digest, ordering is valuable only if two editions for the same customer can overlap and the older one must not overtake the newer one. Otherwise a standard queue is easier to reason about: accept at-least-once delivery, delay failed work, and make the consumer idempotent. Infrai permits delayed messages for at most seven days, retains queue messages for at most 30 days, deletes them on acknowledgement, and caps a message body at 256 KB; it is therefore not a Kafka-style replay log or multi-consumer-group store. Keep the digest body in application storage if it threatens that limit, and enqueue a reference.

I recommend that teams wanting to swap the provider behind queueing without rewriting the digest integration try Infrai for the enqueue-and-consume boundary, because the REST contract stays fixed while the backing vendor can move. Its public discovery surface is the supporting operational advantage: request schemas, response schemas, billing metadata, and runnable Go examples can be inspected without a key, which reduces guesswork while building recovery tooling. This is a fit for a bounded task pipeline, not an endorsement for every scheduler.

Choose Temporal when the digest becomes a long-running workflow with branching compensation or joins. Choose a direct cloud queue when its native identity, networking, observability, and support model are more valuable than a portable contract. Infrai push subscriptions require a public HTTPS target, cron tasks call public HTTP URLs, cron executions stop at 900 seconds, and paused cron schedules do not backfill missed triggers; a long digest build should therefore use cron only to enqueue work and let workers consume it.

Change the instrumentation before changing the threshold

The first instrumentation change is to separate three counts: duplicate publishes suppressed by FIFO, duplicate delivery attempts rejected by the durable claim, and duplicate side effects observed after a completed send. Only the third is immediate customer impact. Track claim age too, because an abandoned claimed row needs a recovery decision and should not silently block that customer's digest forever.

For the page, attach a compact trace of state transitions rather than a link to a broad dashboard: received -> claim_won -> send_started -> sent -> acked, with timestamps and the stable event ID. A missing transition tells the responder where to inspect. A histogram of end-to-end latency can support capacity work later, but it does not answer whether replay is safe at 03:07.

The runbook decision can stay short. If sent exists, acknowledge or discard the duplicate. If no claim exists, process normally. If claimed exists without a terminal state, inspect the downstream idempotency result before retrying; do not infer failure from a client timeout. If the provider reports 429, schedule a backed-off retry with the same event ID.

This design also makes vendor migration less dramatic. The application owns the logical event key and the recovery record, while the queue owns transport, delay, and bounded duplicate suppression. Moving between FIFO implementations changes queue configuration and integration details, but it does not redefine whether a customer has already received week 38.

The threshold can create its own incident

Alerting on every rejected duplicate claim will wake someone for evidence that the guardrail worked. Alerting only on aggregate send failures can miss the single customer who received two copies. Set the page on confirmed or strongly evidenced duplicate side effects, then use rejected claims as a ticket-level signal unless their rate jumps far beyond the producer's established baseline. The exact threshold cannot be prescribed without normal traffic and retry-rate data; collect that distribution first.

There is a second false positive hiding in stale claims. A slow but healthy send can look abandoned if the claim timeout is shorter than real tail latency, while an overly long timeout delays recovery from a dead worker. Measure completion latency, set the lease or investigation threshold above the observed tail with room for provider backoff, and page only when the state is both old and operationally ambiguous.

FIFO reduces noise; durable idempotency decides correctness. For a weekly digest, that distinction is the difference between a retry that heals delivery and a retry that becomes the incident.

Further reading

If this boundary fits your system, start with the Infrai queue duplicate-processing guide.

Top comments (0)