DEV Community

SullivanReed1247
SullivanReed1247

Posted on

Implementing Direct Browser Uploads in Next.js: Presigned URLs, CORS, Deletion Deadlines

Pick the upload path by file size and by how the deletion clock works, not by which client library drops into React fastest. A logistics document vault makes that concrete: proof-of-delivery scans arrive as 40–300 MB PDFs from a driver's tablet, they have to land in private object storage in Europe, and each one carries a destroy-by date that came from a customer contract. Large-file throughput and a hard retention deadline are the constraints that decide everything downstream — how signed URLs are minted, who writes the CORS rules, and what the Next.js route handler is allowed to touch.

The storage product is the last decision, not the first.

The two hard edges: 300 MB scans that must be destroyed on a date

A driver scans a signed delivery note at the tailgate. What arrives is a 40–300 MB PDF, or occasionally a multi-page TIFF from a depot scanner that nobody has reconfigured since it was installed. Traffic is bursty in a way that averages hide: a shift ends, and four hundred documents show up in twenty minutes over hotel-grade Wi-Fi and 4G tethering. The bytes are not the interesting part on their own — the interesting part is that a signature and a name make each file personal data, the contract says it is destroyed twenty-four months after delivery, and "destroyed" is not "moved to an archive tier and forgotten".

So there are two edges. Bytes per second on the way in, and a date on the way out. Everything in the middle is plumbing that must not lose either one.

I'd frame the retention side as a data model problem rather than a storage feature problem, because the date is per document and the delivery event — not the upload timestamp — starts the clock. A document uploaded late, after a dispute, may already be eleven months into its life when it first hits the bucket.

What actually breaks when the browser writes straight into private storage

The failure modes are boring, specific, and almost all of them show up only above a few hundred megabytes.

A presigned PUT is never a CORS-simple request, so every distinct part URL earns its own preflight, and the preflight result cache is keyed per URL. Sixty parts means sixty OPTIONS round trips before a single useful byte moves. You cannot cache your way out of it either: Chromium clamps Access-Control-Max-Age to 2 hours and Firefox to 24, and neither cap helps when the URLs are unique per upload anyway.

Then the header nobody remembers. Multipart completion needs the ETag of each part, and JavaScript cannot read a response header that the bucket policy does not expose. Leave ExposeHeaders out and every part uploads perfectly while the completion call fails with a list of empty tags — a bug that looks like a storage outage and is actually four lines of JSON.

[
  {
    "AllowedOrigins": ["https://vault.example.com"],
    "AllowedMethods": ["PUT", "GET", "HEAD"],
    "AllowedHeaders": ["content-type", "content-md5"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3600
  }
]
Enter fullscreen mode Exit fullscreen mode

Progress reporting is the next trap. fetch() still reports nothing about request upload progress, so a 300 MB upload with a spinner and no percentage gets cancelled by impatient drivers; XMLHttpRequest.upload has fired progress events for years and remains the pragmatic choice, unless you are willing to restrict yourself to streamed request bodies, which need HTTP/2 and duplex: "half" and are Chromium-only today.

Credential lifetime catches people out too. A SigV4 presigned URL is capped at seven days, but that ceiling is theoretical if you sign with the temporary role credentials of a serverless function: the URL dies when the credentials do, often within the hour. A truck parked in a yard with two bars of signal will resume an upload three hours later and find every part URL dead, so re-presigning has to be an endpoint, not an afterthought.

And the last one, which is a habit rather than a bug: never let the client's "upload finished" callback be the thing that marks a document received. Issue a HEAD on the object, compare the size and content type against what the browser declared, and only then write the ledger row. Truncated uploads over flaky links are common enough that this check pays for itself in the first month.

Should the browser upload directly to private storage, or should the Next.js server proxy the files?

Proxying through a route handler is genuinely simpler, and for files in the low megabytes it is the right amount of machinery. Above that it stops working for reasons that have nothing to do with your code: serverless platforms cap request bodies aggressively — Vercel functions reject anything over 4.5 MB — and even on a long-lived Node process you are paying for the same bytes twice, once inbound and once outbound, while holding a connection open for minutes.

Direct-to-storage for the bytes, server for the control plane. That split is what the rest of the design hangs on.

The server decides the object key, mints the part URLs, records the destroy-by date, and verifies the result. The bucket stays private, with no public read path at all; download links are short-lived signed GETs that the app issues after checking the ledger. The trade-off is that you now own CORS configuration, part-size arithmetic, and a resumable-upload state machine in the browser — roughly a week of work you do not spend if you proxy.

import os
from datetime import date, timedelta

import boto3
from botocore.config import Config
from fastapi import FastAPI
from pydantic import BaseModel

from vault import ledger

MIN_PART = 5 * 1024 * 1024      # floor for every part except the last
MAX_PARTS = 10_000
RETENTION = timedelta(days=730)  # 24 months from the delivery event

store = boto3.client(
    "s3",
    endpoint_url=os.environ["OBJECT_STORE_ENDPOINT"],
    region_name=os.environ.get("OBJECT_STORE_REGION", "eu-central-1"),
    config=Config(signature_version="s3v4"),
)
BUCKET = os.environ["DOCS_BUCKET"]


def part_size(total_bytes: int) -> int:
    """Smallest legal part size that still fits under the part-count ceiling."""
    size = MIN_PART
    while total_bytes / size > MAX_PARTS:
        size *= 2
    return size


class StartUpload(BaseModel):
    shipment_id: str
    filename: str
    size_bytes: int
    delivered_on: date


app = FastAPI()


@app.post("/uploads/start")
def start_upload(req: StartUpload):
    destroy_by = req.delivered_on + RETENTION
    key = f"pod/destroy={destroy_by.isoformat()}/{req.shipment_id}/{req.filename}"
    created = store.create_multipart_upload(
        Bucket=BUCKET,
        Key=key,
        ContentType="application/pdf",
        Tagging=f"destroy_by={destroy_by.isoformat()}",
    )
    chunk = part_size(req.size_bytes)
    count = (req.size_bytes + chunk - 1) // chunk
    parts = [
        {
            "part_number": n,
            "url": store.generate_presigned_url(
                "upload_part",
                Params={
                    "Bucket": BUCKET,
                    "Key": key,
                    "UploadId": created["UploadId"],
                    "PartNumber": n,
                },
                ExpiresIn=3600,  # the browser retries a part, never the whole file
            ),
        }
        for n in range(1, count + 1)
    ]
    # The ledger row is what the deletion job trusts later — not a bucket listing.
    ledger.record(key=key, upload_id=created["UploadId"], destroy_by=destroy_by)
    return {"key": key, "upload_id": created["UploadId"], "part_size": chunk, "parts": parts}
Enter fullscreen mode Exit fullscreen mode

The key carries the cohort date and a tag repeats it, which gives two independent ways to find a document when someone asks the uncomfortable question about what is still retained.

Making the deletion deadline a property of the object

Lifecycle engines are day-granular and asynchronous. Objects are queued for removal once a rule matches, and the rule evaluation runs on a daily cadence, so "expires on the 31st" means "gone shortly after the 31st" — fine for a contract, not fine for a promise you made about an exact hour.

Three mechanisms, with real limitations between them. Prefix-based rules are easy to reason about but you get a bounded number of rules per bucket — a thousand in S3 — so one rule per calendar day of expiry runs out after roughly three years. Tag-based rules scale better and let a single rule cover a retention class, at the cost of being invisible in a listing. Per-object expiry metadata is the cleanest of the three where it exists: Google Cloud Storage lets you stamp a customTime on each object and write one lifecycle rule with daysSinceCustomTime, which expresses "delete 730 days after the delivery event" exactly, without cohort prefixes.

Versioning is where compliance stories quietly fall apart. A delete in a versioned bucket writes a delete marker; the bytes stay as a noncurrent version until a second rule expires those too, and an auditor who knows this will ask. Object Lock is the mirror-image mistake — retention there is a floor that forbids early deletion, not a ceiling that guarantees eventual deletion, and in compliance mode it will happily block the destruction you contractually owe.

So the bucket rules are a backstop, and a job you own is the primary enforcement.

def sweep(today: date) -> int:
    destroyed = 0
    for row in ledger.due(today):            # the ledger is authoritative
        store.delete_object(Bucket=BUCKET, Key=row.key)
        for version in noncurrent_versions(row.key):
            store.delete_object(Bucket=BUCKET, Key=row.key, VersionId=version)
        ledger.mark_destroyed(row.key, today)
        destroyed += 1
    return destroyed
Enter fullscreen mode Exit fullscreen mode

One metric matters more than the rest: the count of ledger rows whose destroy-by date has passed and whose object still answers a HEAD. That number should be zero, it should be on a dashboard, and it should page someone when it is not.

There is a delivery-side edge case here that anyone who has shipped one-time links by email already knows, and it applies directly to signed URLs. The email that tells a shipper their document is ready may sit in a greylist queue for fifteen minutes, and corporate mail scanners follow every link in a message before a human ever sees it. A signed URL embedded in that email is therefore both stale and already fetched by the time it is clicked. Link to your own app instead, check the ledger, and mint a signed GET valid for five minutes at click time — the retention deadline stays enforceable because expired documents simply stop producing links.

Four upload paths, compared on throughput and on what expiry costs

Path How the browser writes Large-file behaviour Who configures CORS Deletion deadline
Managed storage SDK (Supabase Storage) SDK call or a signed upload URL Resumable TUS endpoint; per-project size limits Platform, per project A job you write and schedule
S3 presigned multipart PUT per presigned part URL 5 MiB minimum part, 10,000 part ceiling You, in the bucket CORS policy Lifecycle rules by prefix or tag, 1,000 per bucket
R2 presigned multipart (S3-compatible) PUT per presigned part URL All parts except the last must be equal size You, in the bucket CORS policy Lifecycle rules with expiration and multipart abort
Google Cloud Storage resumable One session URI, chunked PUTs Single upload session, byte-range resume You, in the bucket CORS config daysSinceCustomTime per object

Read that table as four different answers to "who owns the hard parts", because the SDK ergonomics converge once you are past the first afternoon. A managed storage layer with a resumable endpoint removes the part-size arithmetic and the CORS policy from your plate, which is a real saving, and it charges you back in expiry work you script yourself. The S3-compatible paths give you a lifecycle engine but hand you the part accounting. R2's equal-part-size requirement is stricter than the S3 rule it otherwise mirrors, so a client that varies part size mid-upload is portable in theory and broken in practice.

European residency is a bucket-placement question on all four, and it is worth confirming what "in Europe" means for backups and for whatever CDN sits in front of downloads, which is where residency assumptions usually leak.

Rolling it out on a vault that is already taking traffic

Ship the control plane first, behind a flag, for one depot. Keep the proxy route alive for anything under 8 MB — it will keep working, and it gives you a fallback when a browser does something unexpected. Backfill cohort prefixes or tags onto existing objects before you enable any lifecycle rule, then run the sweeper in report-only mode for a full week and diff its intentions against the ledger.

Test the expiry path in a staging bucket with a one-day rule, so you observe the real deletion latency instead of assuming it.

Honestly, the storage layer here is the least durable part of the decision — buckets are portable, and a week of client work moves with you. The deletion ledger, the destroy-by dates, and the evidence that both were honoured are what you will still be maintaining in five years, and no vendor will own that for you.

References

Top comments (0)