A Change Data Capture (CDC) consumer that restarts from the wrong position either drops Opportunity updates or replays three days of them. This failure shows up after every deploy, crash, or network drop in services that mirror Salesforce records into a database. The fix is a durable checkpoint. If you want background on how Salesforce change events work in production, read that overview first. This article then shows how to store replay IDs, resume a gRPC subscription, and make replays harmless, using Node.js, the Pub/Sub API, and Postgres.
Context and Setup
You need a stream with a durable position, and CDC provides one through the replay ID. The scenario: a worker subscribes to /data/OpportunityChangeEvent and mirrors each change into Postgres for reporting.
Prerequisites:
CDC enabled for Opportunity in Setup.
An OAuth access token, instance URL, and org ID.
pubsub_api.proto from the forcedotcom/pub-sub-api repository.
@grpc/grpc-js, @grpc/proto-loader, avsc, and pg.
The time limit matters. The Streaming API documentation on message durability states that events are retained for 72 hours and that events outside the window are discarded. A Salesforce help article (2026) adds that CometD clients can end up with an invalid replay ID after events age out, and it recommends the Pub/Sub API.
Building a Salesforce CDC Consumer with Replay Checkpoints
The consumer needs three parts: a checkpoint table, a resumable subscription, and an idempotent apply step.
Step 1: Store the checkpoint beside the data
Keep the replay ID in the same database as the mirrored rows, so one transaction covers both. their calls replay IDs opaque, so store them as bytes.
Create one checkpoint row per channel.
Create a table of applied change keys.
sql
-- Why: the checkpoint lives next to the data it describes
CREATE TABLE cdc_checkpoint (channel text PRIMARY KEY, replay_id bytea NOT NULL);
-- Why: a replayed event hits this key and becomes a no-op
CREATE TABLE cdc_applied (tx_key text, seq int, PRIMARY KEY (tx_key, seq));
Step 2: Subscribe from the saved position
Resume with the CUSTOM replay preset and the stored bytes. Use LATEST only on the very first run.
js
const grpc = require('@grpc/grpc-js');
const loader = require('@grpc/proto-loader');
const def = loader.loadSync('pubsub_api.proto', { keepCase: true, longs: String, enums: String, defaults: true });
const { PubSub } = grpc.loadPackageDefinition(def).eventbus.v1;
function subscribe(auth, channel, saved, onBatch) {
const client = new PubSub('api.pubsub.salesforce.com:7443', grpc.credentials.createSsl());
const meta = new grpc.Metadata();
meta.add('accesstoken', auth.token);
meta.add('instanceurl', auth.instanceUrl);
meta.add('tenantid', auth.orgId);
const call = client.Subscribe(meta);
const N = 50; // Why: caps in-flight events so memory stays flat
const start = saved ? { replay_preset: 'CUSTOM', replay_id: saved } : { replay_preset: 'LATEST' };
call.write({ topic_name: channel, num_requested: N, ...start });
call.on('data', async (res) => {
call.pause(); // Why: keeps batches in order while we await the database
await onBatch(client, meta, res.events, res.latest_replay_id);
call.resume();
// Why: the server stops sending once the requested count is used up
if (res.pending_num_requested === 0) call.write({ topic_name: channel, num_requested: N });
});
return call;
}
Step 3: Apply changes idempotently and checkpoint atomically
At-least-once delivery plus a dedupe key is simpler than chasing exactly-once. The trade-off is one extra table, which you prune after 72 hours because nothing older can replay.
js
const avro = require('avsc');
const types = new Map(); // Why: schemas rarely change, so fetch each ID once
async function onBatch(client, meta, events, latest) {
const db = await pool.connect(); // pool is a pg.Pool created at startup
try {
await db.query('BEGIN');
for (const ev of events) {
let type = types.get(ev.event.schema_id);
if (!type) {
const s = await new Promise((ok, no) =>
client.GetSchema({ schema_id: ev.event.schema_id }, meta, (e, r) => (e ? no(e) : ok(r)))
);
type = avro.Type.forSchema(JSON.parse(s.schema_json));
types.set(ev.event.schema_id, type);
}
const rec = type.fromBuffer(ev.event.payload); // Avro bytes to object
const h = rec.ChangeEventHeader;
const fresh = await db.query(
'INSERT INTO cdc_applied VALUES ($1,$2) ON CONFLICT DO NOTHING', [h.transactionKey, h.sequenceNumber]);
if (fresh.rowCount === 1) await applyChange(db, h, rec); // your upsert logic
}
// Why: runs even for empty keep-alive replies, so quiet channels never age out
await db.query(
'INSERT INTO cdc_checkpoint VALUES ($1,$2) ON CONFLICT (channel) DO UPDATE SET replay_id = EXCLUDED.replay_id',
[CHANNEL, latest]);
await db.query('COMMIT'); // Why: data and checkpoint succeed or fail together
} catch (e) { await db.query('ROLLBACK'); throw e; } finally { db.release(); }
}
Against the older CometD route, this costs you gRPC setup and Avro decoding. In return, It's documents replay tightening for the Pub/Sub API, which advances your replay ID past irrelevant events on reconnect.
Real-World Application
In one of our Crm projects at Oodles, a client from real estate mirrored Opportunities into Postgres through a CometD listener. Each deploy left gaps, so a nightly reconciliation job ran to repair them. We moved the listener to the Pub/Sub API with the checkpoint design above.
Key Takeaways
Write the replay ID in the same transaction as the data, or a crash will split them.
Treat replay IDs as opaque bytes, never as numbers you can increment.
Persist the latest replay ID on empty replies to stay inside the 72-hour window.
Dedupe on transactionKey and sequenceNumber so replays do no harm.
Pause the stream while awaiting the database to keep event order.
Have a replay edge case or a gap you could not explain? Share it in the comments, or send your Salesforce architecture questions to our team directly.
What is Salesforce Change Data Capture?
It publishes a change event whenever a record is created, updated, deleted, or undeleted. Subscribers receive field values and a header describing the change, which lets external systems stay in sync without polling the API.
How long are change events stored?
It retains change events for 72 hours. Events older than that are purged, and Salesforce does not guarantee storage beyond the window. A consumer that stays offline longer must rebuild its state from a full data export.
Should I use CometD or the Pub/Sub API?
Choose the Pub/Sub API for new work. It uses gRPC, supports flow control, and adds replay tightening. It recommends migrating CometD subscribers, which can see long silent periods or invalid replay IDs.
What do the replay options mean?
LATEST delivers only new events, matching CometD’s -1. EARLIEST returns everything in the retention window, matching -2. CUSTOM resumes after a saved replay ID, which is the right choice after a restart.
How do I avoid processing an event twice?
Assume at-least-once delivery. Store the transactionKey and sequenceNumber from the change header with a unique constraint, and skip the write when the insert conflicts. Prune those keys after 72 hours.
Top comments (0)