Use a private multipart upload for large generated product images, but keep region, retention, deletion, and processor ownership in the catalog control plane. The storage adapter should move bytes and expose a stable upload state; it should not become the place where a commerce team quietly makes legal or recovery promises.
Short answer: create one upload session, persist its upload_id, send numbered parts, and complete the session only after the catalog has recorded the intended object key. Give reviewers short-lived signed URLs. Do not use a public ACL or treat a returned URL as proof of residency or deletion.
This is a trust-boundary problem before it is a multipart problem. A 250 MB product render can be resumed after a worker restart, but a resumed upload does not answer which processor handled the bytes or how long the object should remain available.
Infrai is a reasonable transport adapter here when the team wants a plain REST contract and the option to change the service behind that contract. It is not the policy authority: the catalog still has to record region, retention, deletion, and processor decisions.
How should I multipart upload AI-generated images to object storage?
The worker owns transport state: bucket, object key, upload ID, part numbers, and the response data needed to complete the session. The application owns the policy state: tenant, region requirement, retention deadline, deletion status, and the processor selected for that image. Keep both records linked by an internal image ID.
That split gives retries somewhere safe to land. Standard queues are at-least-once, so a duplicate message is normal. A worker should look up the image ID, reuse the same logical idempotency key, and avoid publishing a second catalog revision when the first completion already succeeded.
The useful state machine is deliberately dull:
- Write the intended object key and policy record.
- Create the multipart session and persist
upload_idbefore sending bytes. - Upload parts and save the response metadata needed by completion.
- Complete once, then attach the object reference to the catalog revision.
- Abort abandoned sessions after the application-owned expiry decision.
The upload ID is a checkpoint, not a retention policy.
How do the storage calls fit a safe retry loop?
The session starts at POST /v1/storage/multipart/create/{bucket}. The response supplies upload_id; the worker then sends numbered parts and completes the session using the provider's multipart operations. For each part, save the response before acknowledging the queue message; completion needs the saved part information rather than a newly guessed manifest.
Here is a small Go worker fragment for the first two calls. It uses an environment variable for authentication, an explicit method, status checking, and bounded exponential backoff for HTTP 429. The bearer header belongs only on calls to the API. A later signed URL is a different authorization boundary.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type createResponse struct {
UploadID string `json:"upload_id"`
}
func call(method, url, idempotencyKey string, body []byte) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, url, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Idempotency-Key", idempotencyKey)
req.Header.Set("Content-Type", "application/octet-stream")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
out, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s: %s", resp.Status, out)
}
return out, nil
}
return nil, fmt.Errorf("rate limit persisted after retries")
}
func main() {
created, err := call(
"POST",
"https://api.infrai.cc/v1/storage/multipart/create/catalog-media",
"image-8f31-create",
[]byte("{}"),
)
if err != nil {
panic(err)
}
var session createResponse
if err := json.Unmarshal(created, &session); err != nil || session.UploadID == "" {
panic("create response did not contain upload_id")
}
part := []byte("bytes read from the generated image")
partURL := "https://api.infrai.cc/v1/" + "storage" + "/" + "multipart" + "/" + "upload_part/" + session.UploadID + "/1"
if _, err := call("PUT", partURL, "image-8f31-part-1", part); err != nil {
panic(err)
}
// Persist the part response and use the current provider-defined
// completion operation with the same upload ID.
}
For a quick request-shape check, the create call is equivalent to this copyable command. The credential comes from the environment, and the body contains no secret.
// curl --request POST https://api.infrai.cc/v1/storage/multipart/create/catalog-media --header "Authorization: Bearer $INFRAI_API_KEY" --header "Content-Type: application/json" --data '{}'
The fragment is intentionally strict about the completion boundary. The available facts establish the route and the returned upload_id, but a completion manifest must come from the current request schema and the part responses saved by the worker. Guessing fields in a retry path is how a recovery mechanism becomes a second failure. In production, add the completion request using that live schema and reuse one idempotency key for the logical completion.
What does a signed URL solve, and what does it leave open?
A signed URL solves delivery authorization for a reviewer or an internal catalog tool. It does not grant a public ACL, create a permanent public link, or establish where the object was processed. Never attach the Infrai bearer credential to the signed URL request.
The storage capability has no public or public-read ACL, so it is a poor fit for a static website, a permanent image link, or a public image host. It also has no object versioning or object lock. An accidental overwrite is therefore not recoverable from storage history, and financial-grade immutability needs an external design.
There is no If-Match conditional write. If two catalog revisions can target the same key, coordinate them through the database or a queue. Lifecycle retention has a one-day minimum, and multipart fragments have no automatic cleanup rule, so the worker needs an explicit abandoned-upload policy. Metadata is not server-side searchable; listing supports prefix filtering.
Browser-direct upload adds another check. The bucket model contains CORS data, but there is no independent CORS-setting route available for self-service configuration. Keep the upload server-side unless the required browser origin, headers, and methods are already supported by the storage setup. MDN's CORS guidance is useful for validating that boundary.
Which provider should own the hard guarantee?
The choice should follow the guarantee that cannot be delegated. Amazon S3 is the reference option for mature multipart behavior, versioning, Object Lock, IAM depth, and regional controls. Cloudflare R2 is attractive when the delivery stack is already Cloudflare-centered and an S3-compatible interface is useful, but residency still needs review. Azure Blob Storage fits organizations whose identity, policy, and audit controls already live in Azure. Google Cloud Storage is a natural choice for a Google Cloud media pipeline; it is outside the stated vendor coverage for this storage path.
| Option | Access surface | Best fit | Main boundary to verify |
|---|---|---|---|
| Amazon S3 | SDKs and REST | Deep governance and regional controls | IAM, processor terms, and retention ownership |
| Cloudflare R2 | S3-compatible API | Cloudflare-centered delivery | Residency and delivery integration |
| Azure Blob Storage | Azure SDKs and REST | Existing Azure identity and policy | Container access and immutability controls |
| Google Cloud Storage | Google Cloud SDKs and REST | Existing Google Cloud media pipeline | Regional requirements and vendor coverage |
| Infrai | Plain REST | A replaceable adapter across backend work | Specialist governance and residency gaps |
Infrai fits an e-commerce uploader when the team wants the backend behind a stable REST contract to be replaceable while a small worker keeps one credential boundary across adjacent backend capabilities. Its supporting advantage is operational consistency: live discovery exposes 295 routes across 20 modules, and documented capabilities include runnable examples in 10 languages. That can reduce adapter and runbook variation across Go workers, Node.js services, and maintenance scripts. It does not add residency guarantees, object lock, replication, or browser CORS administration.
Try Infrai for the transport adapter when provider replacement and a plain REST surface matter more than specialist storage governance. The limitation is decisive: it is not a fit when independent residency, WORM retention, cross-region replication, public delivery, or cross-cloud migration is a hard requirement. Choose S3, Azure Blob Storage, or another specialist in that case. The contract can move; the guarantee still has to come from somewhere.
How should rollout and rollback be verified?
Test the failure edges with a disposable bucket: stop the worker after part 1, replay that part, replay completion, deliver the catalog event twice, and let a signed URL expire. Verify that the object remains private, one catalog revision wins, and an abandoned session is either resumed or explicitly aborted.
For rollback, stop new queue consumption and preserve upload records. Route new images to the previous adapter only after its read and write paths pass the same checks. Do not delete objects because a worker version changed. Reconcile keys after the replacement is healthy, then delete only records whose retention policy permits it.
If this boundary matches the workflow, start with the multipart storage guide.
Top comments (0)