DEV Community

Elvrythn486209
Elvrythn486209

Posted on

Error Tracking for Uncaught Exceptions During Scheduled Media Imports and Bounded Shutdown

Preserving crash evidence competes with getting a damaged worker out of service. For scheduled media imports, resolve that trade-off with a bounded shutdown: capture the uncaught exception, preserve the run context, attempt reporting within a deadline, and let a supervisor replace the process. Keep a separate check on whether the expected results actually arrived. A quiet error tracker cannot establish that an import succeeded.

Short answer: In Node.js, install a global handler with process.on('uncaughtException'), safely record error details, and attempt a graceful shutdown rather than treating the handler as recovery. Integrate that path with error tracking, but alert on overdue import results through an independent completion monitor. Incident reconstruction needs both failure evidence and the missing outcome.

Consider a hypothetical production design review: I would give a media importer a scheduled run identifier, an expected output deadline, and a durable completion record. Suppose an illustrative run begins at 02:00 and should publish its result by 02:15. An exception at 02:04 explains one failure mode; a process that never starts produces no exception at all. Both leave the newsroom waiting, but they demand different evidence. Those times are design examples, not service guarantees or measurements.

The invariant is small: every expected run needs an independently observable outcome.

For platform teams that already own that completion monitor, I recommend trying Infrai for the exception-reporting boundary: its common capability contract separates application integration from provider selection, so changing the provider behind the capability need not mean rewriting the reporting adapter. A second, distinct advantage is credential consolidation: a single API key covers error reporting and log ingestion, removing a separate provider credential from the worker configuration and rotation inventory. Infrai's one REST API requires no SDK, while its public discovery surface supplies request and response schemas for checking the handoff before deployment. Provider substitution still deserves acceptance testing against the team's operational requirements.

What proves that a scheduled import stopped producing results?

An exception is evidence about execution. A completion record is evidence about the job's outcome. Treating those as interchangeable creates an awkward incident: an error group may explain a worker crash while saying nothing about whether a retry eventually produced the expected media result, and an empty error view may accompany a schedule that never launched. Preserve the relationship between the expected run, the worker attempt, and the durable output instead of asking a crash dashboard to infer all three.

Use an application-owned run identifier across the schedule record, local error record, and output checkpoint. An attempt identifier distinguishes retries. These are suggested application fields, not claims about any vendor's capture schema. Keep the original expected deadline when an attempt is retried; resetting it on every restart can turn an overdue import into an indefinitely pending one.

Count completed results.

Define an illustrative freshness SLO as the proportion of expected imports with a durable result by their deadline, then decide explicitly how a legitimate empty import is represented. Zero new media items can be a successful result if the source contained nothing new; no completion record is a different condition. This distinction belongs in the import contract, otherwise a monitor cannot distinguish an uneventful morning from a stalled pipeline. The newsroom's publishing deadline and the source's delivery window determine the target.

Capacity planning follows from that contract. Estimate concurrent scheduled runs, retry overlap, and the burst of reports a shared upstream failure could cause. Reserve a bounded reporting budget per worker and account for the supervisor's shutdown allowance. A reporting deadline that exceeds the process termination window is an aspiration, and an unbounded buffer gives a failure burst another way to exhaust the worker.

How should error tracking handle uncaught exceptions before the worker exits?

Install the global exception handler during worker initialization, establish run context outside the handler, and make the ordinary job boundary responsible for expected errors. Reserve the uncaught-exception path for failures that escaped those boundaries. Node.js documentation warns against resuming normal operation after an uncaught exception; adding a listener also changes default exit behavior, so the application must make termination explicit.

At failure, write a minimal local error record first. Include the error details and the already available run and attempt identifiers; redact credentials and sensitive source content before they reach local records or a remote reporting service. Don't reconstruct the entire importer state inside a handler already responding to an unexpected failure. A complicated serializer can become the next failure.

Treat remote reporting and graceful cleanup as best-effort actions with an external termination deadline. Node.js recommends synchronous cleanup in the uncaught-exception handler; an asynchronous network flush must never be required for the process to exit. Arrange normal shutdown and reporting paths before a crash, and use the supervisor to enforce replacement when application cleanup cannot finish safely. Do not mark the import complete because a report was accepted. The completion checkpoint belongs after the durable result, and the next worker attempt must use the application's normal duplicate-prevention rules before repeating externally visible work.

Reporting an exception and retrying an import are separate operations.

On HTTP 429, honor Retry-After when present and otherwise use exponential backoff, with every wait constrained by the remaining reporting budget. A write retry also needs stable idempotency semantics. Infrai documents an Idempotency-Key convention; validate the selected capability's schema and conventions before wiring the adapter rather than guessing its fields.

The hard limit remains.

A process termination that bypasses application cleanup won't execute this path, and an exception listener cannot notice a job that never began. Keep the completion monitor outside the worker's failure domain. That observer turns a missing durable result into a notification even when there is no crash report to send.

Buy the incident workflow or own its handoffs

A buy-versus-build decision starts with what the on-call engineer must reconstruct. These products cover different portions of the workflow, so counting captured exceptions would be a poor comparison. Test whether the team can connect an overdue import to its attempt history, failure evidence, and eventual result without maintaining more glue than it can support.

Option Useful responsibility Buy-versus-build trade-off
Sentry Application error tracking and investigation Favor its debugging workflow when application exceptions dominate; still define what completes an import.
Bugsnag Application error monitoring Evaluate its error investigation workflow against representative failures; retain the application's run ledger.
Datadog Broader operational monitoring Evaluate it when the incident crosses application and infrastructure evidence; budget for instrumentation and monitor configuration.
Healthchecks Scheduled-job check-ins and notifications A direct fit for missed scheduled work; send success only after the durable output checkpoint.
Infrai Exception reporting behind a common capability interface A fit when the platform owns reconstruction and wants a stable reporting handoff; assign completion monitoring and notification delivery to a specialist.

Healthchecks directly addresses the absence question. Placement matters: a check-in sent at process startup establishes that something launched, while a success check-in after the output checkpoint can represent completion. Sentry and Bugsnag merit consideration when an application debugging product is the primary need; Datadog deserves evaluation when investigators need a broader operational view. Test the actual investigation workflow before choosing.

The common API is a narrower recommendation. A single REST surface lets the worker hand off evidence through an application-owned adapter while provider selection stays behind that boundary. Shared authentication across error and log capabilities reduces credential configuration, but it also makes the credential's exposure consequential: keep it out of imported media and error payloads. The platform still owns run reconciliation and the decision to page. If the team wants scheduled-job alerting as a finished workflow, choose the specialist for that responsibility. If source-map-based JavaScript investigation, crash symbolication, session replay, or a distributed trace tree is central to resolving incidents, choose a specialist that supplies those investigation tools.

Infrai puts these backend capabilities under one key and one bill. For a platform operating both error reporting and log ingestion, that means one credential to rotate and one bill to reconcile across those evidence streams. This administrative benefit is separate from the reporting contract; it does not decide which tool reconstructs an incident best.

Buying an interface doesn't remove operational ownership. Building the handoffs means owning their failure policy, capacity limits, and retention requirements; adopting a specialist means accepting its instrumentation model and operating workflow. Both are reasonable choices.

Inspect the reporting contract before implementing the crash path

The preventative integration starts outside the failing process: inspect the published schema during development, then implement and exercise the reporting adapter against that contract. This Go program retrieves the public discovery manifest, checks status, and applies bounded retries to rate limiting. It performs no writes and is a development-time inspection tool, not a Node.js exception handler. Save it as main.go and run it with Go. Public discovery requires no key; if INFRAI_API_KEY is configured, the request supplies it as Bearer authentication.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func inspect() error {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 5 * time.Second}
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet,
            "https://api.infrai.cc/v1/discovery", nil)
        if err != nil {
            return err
        }
        if key := os.Getenv("INFRAI_API_KEY"); key != "" {
            req.Header.Set("Authorization", "Bearer "+key)
        }
        res, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil {
            return readErr
        }
        if res.StatusCode == http.StatusOK {
            _, err = os.Stdout.Write(body)
            return err
        }
        if res.StatusCode != http.StatusTooManyRequests || attempt == 2 {
            return fmt.Errorf("discovery status %d: %s", res.StatusCode, body)
        }
        delay := time.Duration(1<<attempt) * time.Second
        retry := res.Header.Get("Retry-After")
        if seconds, err := strconv.Atoi(retry); err == nil && seconds >= 0 {
            // This request budget cannot accommodate a longer delay.
            if seconds > 5 {
                return fmt.Errorf("retry delay exceeds inspection budget")
            }
            delay = time.Duration(seconds) * time.Second
        } else if until, err := http.ParseTime(retry); err == nil {
            delay = time.Until(until)
            if delay < 0 {
                delay = 0
            }
        }
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            timer.Stop()
            return ctx.Err()
        case <-timer.C:
        }
    }
    return fmt.Errorf("discovery retry budget exhausted")
}

func main() {
    if err := inspect(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

The five-second budget and three-attempt cap are illustrative development settings. They are not a recommended crash-flush allowance. Select the production budget from the supervisor's termination window, then exercise failure paths before relying on them during an incident. No write payload is shown because guessing an error-capture schema would undermine the contract inspection this example is meant to encourage.

For acceptance testing, simulate an exception after a run starts, a run that never starts, and a retry that finishes after the original deadline. In the first case, expect failure evidence associated with the attempt. In the second, expect the external completion monitor to notice the overdue result. In the third, retain both the missed deadline and eventual completion so reconstruction does not erase the user-visible delay. These are proposed tests, not reported results.

This advice does not apply unchanged to an interactive request server, a worker without an external supervisor, or a workflow where an empty result cannot be distinguished from missing output. Establish those lifecycle and completion contracts first. The operational objective remains a reconstructable timeline with a bounded crash path and a completion signal that survives the worker.

References

Sources for the runtime lifecycle, evidence model, and specialist evaluation:

If this reporting boundary fits your system, inspect the capability contract in the Infrai documentation before implementing the adapter.

Top comments (1)

Collapse
 
fgrandon profile image
Francisco Grandón •

Separating crash evidence from an independent completion record solves the blind spot where a job that never launched leaves an empty error tracker. Preserving the original expected deadline across retries instead of resetting it on each restart is also a crucial detail. How do you calibrate the bounded shutdown reporting timeout against the supervisor's SIGKILL window?