Short answer: for e-commerce product search, choose a retrieval architecture that records the decision inputs, candidate identities, policy version, and cited evidence in one append-only audit event; don't treat the final answer or a general application log as an adequate record. In a B2B SaaS internal knowledge-base bot, that boundary lets an operator explain which catalog material was eligible without retaining every product description or employee question forever.
The key trade-off is evidence versus exposure. Logging too little makes a grounded answer impossible to investigate. Logging raw queries, chunks, and generated text by default creates a second content store with its own access and retention burden. My default is therefore a compact decision record with stable document revisions and keyed content fingerprints, plus separately controlled payload capture for cases that genuinely require it.
How should e-commerce product search retrieval architecture use audit logs?
An audit event should answer four questions: what request entered retrieval, what rules constrained it, what evidence was considered, and what evidence crossed the citation boundary. Those are separate facts. A trace tells engineers where time went; an audit event explains why a product claim became eligible for an answer.
For an internal knowledge-base bot, picture a support engineer asking whether a component is compatible with a particular product revision. The retriever searches catalog specifications, approved merchandising copy, and internal support policy. Each source has a revision, an access scope, and an effective time. The audit event needs the opaque query ID and a protected query fingerprint; the index snapshot or corpus watermark; filter, ranking, and embedding policy versions; an ordered list of candidate revision IDs with their scores; the revision IDs admitted to generation; and the citation IDs emitted with the answer. It also needs one end-to-end trace ID so an operator can correlate timing and audit evidence without pretending the two records serve the same purpose.
That shape follows the central RAG distinction in the original paper: generation is conditioned on retrieved external evidence rather than model parameters alone. The paper doesn't prescribe an operations schema. The architecture choice here is to preserve the boundary between retrieval and generation because that is where a grounded answer gains, loses, or misattributes evidence.
Don't log every intermediate artifact merely because it exists. The useful invariant is narrower: given the same authorized corpus revision and policy version, an investigator can reconstruct the candidate set and see exactly which revisions were supplied to generation. If the corpus has since changed, the event must say so rather than silently replaying against today's index.
Small records win here.
The incident lesson: delivery semantics apply to evidence, too
I've been paged for missed scheduled jobs and duplicate deliveries. That history changes how I review retrieval logging: a request ID is not proof that one durable audit record exists, and a successful answer is not proof that its evidence record committed. The same retry that protects an answer path can duplicate audit rows; a queue handoff can also leave the answer visible while its asynchronously written evidence is absent. This is not an exotic search problem. It is a delivery-semantics problem wearing search vocabulary.
The invariant I want is one immutable audit event per logical retrieval attempt, keyed by a server-issued event ID. A repeated delivery may attempt the same insert, but it must not create a second event. A distinct retry that actually performs retrieval gets a new attempt ID and points to the same parent request. That distinction matters during review: two deliveries of one event are noise, while two retrieval attempts may have observed different corpus watermarks and are operationally meaningful.
Three common storage choices expose different failure surfaces. An application log is easy to correlate and useful for short-lived diagnosis, but rotation, sampling, and free-form fields make it a weak audit ledger. A dedicated append-only table gives explicit uniqueness, retention, and access policy, though the team must own partitioning and deletion. An event stream decouples producers from archival consumers, but replay and at-least-once delivery mean the consumer still needs an idempotency key. None is universally best. The selection question is who must query the evidence, how quickly, and under which deletion policy.
The preventative write path can stay boring. This Go example deliberately records references and fingerprints rather than raw query or chunk text. event_id is generated before a queue handoff, while attempt_id distinguishes a fresh retrieval execution. The database enforces the delivery invariant.
package audit
import (
"context"
"crypto/hmac"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"fmt"
"time"
)
type Candidate struct {
RevisionID string `json:"revision_id"`
Rank int `json:"rank"`
Score float64 `json:"score"`
Admitted bool `json:"admitted"`
}
type Event struct {
EventID string
AttemptID string
RequestID string
TraceID string
QueryDigest string
CorpusRevision string
PolicyVersion string
Candidates []Candidate
CitationIDs []string
OccurredAt time.Time
}
func Digest(secret []byte, value string) string {
mac := hmac.New(sha256.New, secret)
_, _ = mac.Write([]byte(value))
return hex.EncodeToString(mac.Sum(nil))
}
func Append(ctx context.Context, db *sql.DB, e Event) error {
candidates, err := json.Marshal(e.Candidates)
if err != nil {
return fmt.Errorf("encode candidates: %w", err)
}
citations, err := json.Marshal(e.CitationIDs)
if err != nil {
return fmt.Errorf("encode citations: %w", err)
}
_, err = db.ExecContext(ctx, `
INSERT INTO retrieval_audit (
event_id, attempt_id, request_id, trace_id, query_digest,
corpus_revision, policy_version, candidates, citation_ids, occurred_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
ON CONFLICT (event_id) DO NOTHING`,
e.EventID, e.AttemptID, e.RequestID, e.TraceID, e.QueryDigest,
e.CorpusRevision, e.PolicyVersion, candidates, citations, e.OccurredAt.UTC(),
)
if err != nil {
return fmt.Errorf("append retrieval audit event: %w", err)
}
return nil
}
This path assumes event_id has a unique constraint and rows cannot be updated by the application role. HMAC is used because a plain hash of a predictable query can be guessed by testing candidate strings; the key belongs outside the audit table and should rotate under an explicit policy. I'm not sure one retention period fits both support investigations and privacy obligations. The data owner and security team have to resolve that with the actual classifications and deletion duties, not with a value copied from another system's runbook.
Design the record around a replay contract
The schema should be reviewed as a contract, not as a pile of convenient fields. A practical review starts with a replay question: “Can an authorized operator identify the exact evidence revisions and policy that shaped this answer?” If the answer depends on mutable document IDs, a current alias, or an unversioned ranking configuration, the record is incomplete even when it contains plenty of text.
| Record element | Why retain it | Avoid by default |
|---|---|---|
| Event, request, attempt, and trace IDs | Deduplicate delivery and correlate one logical search across systems | Hostnames and incidental process metadata |
| Corpus watermark and document revision IDs | Identify the evidence state used at decision time | A copy of every retrieved chunk |
| Filter and ranking policy versions | Explain eligibility and ordering | Unversioned configuration blobs |
| Candidate ranks, scores, and admission flags | Separate “considered” from “sent to generation” | Claims that scores are comparable across policy versions |
| Emitted citation IDs | Check the answer-to-evidence join | Generated answer text unless review policy requires it |
| Actor and authorization scope IDs | Explain what the requester was allowed to retrieve | Raw credentials or authorization tokens |
Scores require special care. They are observations within one configured retrieval attempt, not timeless measures of relevance. Store the policy version beside them and compare like with like. A rank change after an embedding, filter, or index revision is expected evidence of a changed decision context; it is not automatically a regression.
Access deserves the same precision. The bot's online role needs append permission, an investigation role needs bounded read permission, and routine application operators don't need broad query access to audit payloads. The OWASP logging guidance explicitly warns against recording secrets and sensitive personal data directly and recommends verification of access controls around logs. That makes redaction and authorization part of the schema review, not cleanup work after launch.
Trace Context gives traceparent and tracestate a standard propagation model. Use that shared trace identity for correlation, but keep the immutable decision fields in the audit store. Sampling an operational trace should not erase the durable evidence record; retaining an audit record should not force full-fidelity traces forever.
Replay, detection, and the operator runbook
Replay begins by freezing authority. Confirm the investigator may view the request, then load the original audit event and resolve its document revision IDs from the retained catalog history. Do not run the old query against the current index and call the result a reproduction. If an original revision has expired under policy, report “evidence unavailable under retention policy” and preserve that distinction in the review output.
Consider a synthetic review for an internal support answer about whether part-1842 fits device-77. The event says attempt a-03 ran against corpus watermark catalog-2026-08-31T22:00Z under eligibility policy eligible-us-support-v12; it considered 20 revisions, admitted three, and emitted two citation IDs. The current catalog now points the same product slug to a newer revision, so searching today would answer a different question. The operator first resolves all 20 recorded revision IDs from historical storage, checks that the requester's recorded scope allowed each candidate at the event time, and verifies that the three admitted IDs are a subset of those candidates. Only then does the operator compare the two emitted citations with the admitted set. Suppose one admitted revision describes the base device while a cited revision describes a later accessory bundle. The audit trail has already narrowed the review: candidate generation and authorization can be checked from stable IDs, while the citation join deserves scrutiny. If, instead, one of the 20 revisions is unavailable because its retention period elapsed, the runbook stops claiming full replay and records the missing boundary. It doesn't quietly substitute the new slug target, because that would produce a plausible answer to a different corpus state. This example uses invented identifiers, but the sequence is the important part: freeze authority, recover immutable revisions, compare each decision boundary, and label unavailable evidence honestly.
Next, compare boundaries in order: corpus watermark, authorization scope, eligibility filters, ordered candidates, admitted evidence, and emitted citations. This sequence keeps the investigation causal. If the candidate set differs, inspect ingestion and policy versions before looking at generation. If candidates match but admitted evidence differs, inspect reranking and token-budget policy. If admitted evidence matches but citations do not, the fault domain is downstream of retrieval. Stop guessing.
I would alert on invariants, not on search taste. Useful signals include an answered request with no committed event, two events with the same event ID, a citation whose revision ID is absent from the admitted set, or an audit write lag beyond the service's stated evidence-availability objective. Thresholds belong to the local service objective; your mileage may vary because a synchronous support workflow and an overnight catalog review have different urgency.
Make absence visible.
Deployment needs a shadow period. Emit the new event beside the existing diagnostics, verify completeness and cardinality, test duplicate delivery deliberately, and run deletion against a non-production partition before making the record authoritative. A rollout check should also confirm that new policy versions are registered before traffic uses them. Otherwise the first incident teaches the team that “policy v17” is only a label.
Where this architecture is the wrong choice
The catch is retention cost and governance, not the insert statement. Recording every candidate can be unsuitable for very broad retrieval, high-volume anonymous browsing, or catalogs whose candidate IDs themselves disclose sensitive assortment decisions. In those cases, retain the admitted evidence plus a bounded candidate sample, aggregate operational metrics separately, and document that full candidate replay is unavailable. Don't imply a stronger audit guarantee than the stored data supports.
An append-only relational ledger is also a poor fit when independent parties require tamper evidence beyond database access controls. Choose a cryptographically verifiable log or externally anchored digest design when that is an actual requirement, and have the threat model reviewed by someone accountable for it. Conversely, stick with ordinary structured application logs when the bot is experimental, processes no sensitive material, and needs only short-lived debugging; adding a permanent audit store before ownership and deletion rules exist can make the system harder to operate responsibly.
For the B2B SaaS knowledge bot, the decision rule is plain: retain the smallest record that proves which authorized catalog revisions shaped each cited answer, make retries idempotent, and state where replay ends. Grounding is an operational property only when someone on call can inspect it without reconstructing history from whatever logs survived.
Top comments (0)