Corrected, November 2026. The first version of this post told you to mint the new key under the address your installed copies already read, and publish it into that WKD bucket. Do that and they refuse every further update, because a tool on the default
bothkey source needs its embedded keys and its WKD keys to match exactly. That was my mistake. It also described the escape hatch as though it worked, and it isn't built yet. The steps below are the ones that work. If you've already published that way, put that bucket back to what your shipped copies embed, and they'll verify again as long as you're still signing with the old key. Anything you built with the bigger set against that address will refuse until it's reinstalled.
There are plenty of guides for starting to sign your releases. I've written some of them.
There are almost none for the day you have to replace the key you have been signing with, and that's the harder day, because every already-installed copy of your software is out there holding the old public key and checking each update against it.
Get the order wrong and self-update stops working for the people who haven't updated yet, and they're the group you can't reach by shipping an update (a special kind of irony when you're holding the fix). They're stuck on a version that refuses the only thing that would unstick it, and your fix is on the other side of the gate.
I did this in July, forced rather than scheduled, slap bang in the middle of rebuilding the AWS organisation the old key lived in (not a week I'd choose to repeat). This is the public version of the runbook that came out of it.
Rule one: trust a set, not a key
The rest follows from this. If your verifier holds the public key, you get a moment where the old key is wrong and the new key is not yet installed, and that moment is an outage you can't patch.
So the trust anchor is a set, and verification passes if any key in it validates the signature. In go-tool-base every .asc file dropped into cli/pkg/trustkeys/keys/ is embedded at build time, and that directory is the set a release can be signed by. Right now that's two files, the old key and the new one (a third key is embedded alongside them, which I'll come to, and it lives in a directory of its own for a reason).
The loader has one detail I'd happily see copied everywhere (it's two lines of Go and I'd have missed it the first time round): it panics rather than return a partial set.
A walk or read failure over the embedded filesystem is a build-time corruption of the binary's trust anchors, not a recoverable runtime condition — Keys panics rather than silently returning a partial or empty trust set, which would let verification fall open.
Think about what an empty trust set would actually do... there'd be nothing to check a signature against, and a verifier with nothing to check against has nothing to object to either, so everything passes. For a security gate that's about the worst way to fall over, which is why it refuses to run at all.
Rule two: two sources of truth, and they must agree
One set is compiled into the binary. The other is published to WKD on a domain the code host doesn't control.
If the only place to find your key is the same forge that serves your releases, an attacker who owns the forge owns both the artefact and the thing that vouches for it. A second anchor, published from somewhere the forge can't write to, means owning the forge isn't enough to publish a key on its own.
It only helps if the two agree, and "they agree" is checkable rather than assumed. Ask the live endpoint what it serves:
gpg --auto-key-locate clear,wkd --locate-keys [email protected]
Today that returns three certificates, and they match the three the binary embeds:
2B26658409047ED08B56CEBDCF5B8DBB5D9F19C2 rotation authority
6E2072BBF83DFAAF006300C495DDAC333C37AA35 signing key v1
E969F9AB167897D876B6A93AA44A36F8D7ACF36D signing key v2
Two signing keys and the rotation authority, which is what the binary holds. Now ask for the address the older copies read:
gpg --auto-key-locate clear,wkd --locate-keys [email protected]
2B26658409047ED08B56CEBDCF5B8DBB5D9F19C2 rotation authority
6E2072BBF83DFAAF006300C495DDAC333C37AA35 signing key v1
No v2 in that one, and that's the bit I got wrong the first time I wrote this up.
Agree means agree. On the default both key source the tool fetches the WKD keys for the address compiled into it and compares the two sets, and if the fingerprints differ at all it aborts. A bucket holding the embedded keys plus one more isn't close enough, and the check can't be relaxed. To the verifier a mismatch looks like somebody has tampered with one of the anchors.
A shipped binary's embedded set can't change either, it's compiled in. So the bucket it reads can't change, and from the day you publish it that bucket is frozen. Add a new key to it and the copies reading it refuse updates from then on (which is about the worst thing you can do to the people you're trying to look after). So a new generation of keys needs a new address. Older copies keep reading release@, newer ones read release-v2@, and that's why the two lookups above give different answers.
The sequence
Get the order right and the rest is mostly typing. Each step is there so that installed copies keep seeing the set they expect while new copies pick up a bigger one, until, as far as you can tell, the field has stopped needing the old key.
I ran all of this again for this post with throwaway keys on disk, so the output below is real (the address is example.com and the keys are long gone). The commands are sigillum's, which signs most of my releases these days. It's a single binary, so a release pipeline doesn't need gtb, or even Go, to use it. For a real key, swap --backend local for --backend aws-kms and point --key-id at a KMS alias.
1. Give the old key the new address. A key only lands in a WKD bucket if one of its user IDs names that address, so the old key needs minting again under release-v2@. The fingerprint is worked out partly from the creation time, so pass the original one and it stays the same key:
sigillum keys mint --backend local --key-id old.pem \
--name "Demo Release" --email [email protected] \
--created 2026-06-02T00:00:00Z --output old.asc
INFO Minted OpenPGP key backend=local key_id=old.pem output=old.asc creation_time=2026-06-02T00:00:00Z fingerprint=F3249017DCE1CEAC9A324FE5C6E63DB028BFD454
That's the fingerprint it had under release@. If yours comes back different the creation time is wrong, and you haven't published anything yet, so it's cheap to fix.
2. Mint the new key under the new address. With KMS the private half is generated inside it and never leaves, so you've no secret to copy off a laptop:
sigillum keys mint --backend local --key-id new.pem \
--name "Demo Release" --email [email protected] --output new.asc
INFO Minted OpenPGP key backend=local key_id=new.pem output=new.asc creation_time=2026-11-07T09:10:14Z fingerprint=0E052971E2786572365A8047256121957F71B5D6
For a key you're going to live with I'd provision it as code. Mine come from the signing-kms module, which creates the key alongside a signer role that only a release pipeline can assume (through OIDC), so me, the soft squishy chair-to-keyboard interface, can't accidentally sign anything with it. The catch is that minting makes one signature of its own, the certificate's self-signature, so with the module the mint has to happen in the pipeline as well.
3. Give the rotation authority the new address too. I'll come to what this key is for. It's in the embedded set, so it has to be in the new bucket, and it's the one step that needs a private half out of the safe. Do it in a throwaway keyring on a machine you trust:
gpg --import authority.priv.asc
gpg --quick-add-uid <authority-fingerprint> "Demo Rotation Authority <[email protected]>"
gpg --armor --export <authority-fingerprint> > authority.asc
gpg --show-keys authority.asc
CAA76565D9E8F2563203CD0F46C9F91CB172C889
uid Demo Rotation Authority <[email protected]>
uid Demo Rotation Authority <[email protected]>
Two user IDs, one fingerprint.
4. Publish the new bucket, before anything is signed with the new key, and leave the old one alone. Naming only the new address means the frozen bucket can't get rebuilt by accident:
sigillum keys wkd --domain example.com --email [email protected] \
--output ./wkd-new old.asc new.asc authority.asc
INFO WKD bucket [email protected] hash=yxidni38ndxos3irk9hcm9bjnnemdx8r keys=3
INFO wrote path=wkd-new/.well-known/openpgpkey/example.com/policy
INFO wrote path=wkd-new/.well-known/openpgpkey/example.com/hu/yxidni38ndxos3irk9hcm9bjnnemdx8r
INFO WKD tree complete output=./wkd-new method=advanced emails=1 files=2
Copy that one hu/ file into your staging copy of the live tree with cp --update=none, so it can't overwrite anything, and compare sha256sum on the old bucket before and after you deploy. A client on an old copy shouldn't see any difference, and the checksum is how you know that rather than hope it.
5. Ship a release that trusts the new set. The two signing keys go in internal/trustkeys/keys/, the authority in internal/trustkeys/authority/, and the tool points at the new address (gtb enable signing --email [email protected] from gtb v0.50.4 or later, if your tool came out of gtb, and run it before you copy the keys because it creates that directory on an older scaffold). If an earlier version of my tutorial had you put the authority in keys/, take it out of there. A key left in both directories counts twice, the embedded set stops matching the bucket, and copies of that release refuse all later updates. Keep signing with the old key. It's the only one the copies already out there trust, and they have to be able to verify the release that brings them the new one.
6. Dual-sign releases through the overlap. Sign with the old key, then --append a signature from the new one into that file:
sigillum sign --backend local --key-id old.pem \
--public-key old.asc --output checksums.txt.sig checksums.txt
sigillum sign --backend local --key-id new.pem \
--public-key new.asc --output checksums.txt.sig --append checksums.txt
That's one armoured block with two signatures in it, and a client holding either key can verify. go-tool-base's goreleaser config is the reference implementation: it signs the checksum manifest through a script, and during a rotation that script produces a signature under each key.
Checking that with plain gpg, from a keyring holding only the frozen bucket, gives you this:
gpg: Good signature from "Demo Release <[email protected]>" [unknown]
gpg: Can't check signature: No public key
The good signature is there, and gpg still exits 2, because it couldn't check the other one. go-tool-base's verifier passes if any key in the set validates (rule one again), so self-update sails through. Anyone verifying your releases with their own gpg --verify && install script doesn't, and won't for the whole overlap. I'd put a line in the release notes asking them to import the new key first.
7. Run both for a fixed window. Weeks rather than hours, depending on how often your users run it, with a reminder actually set, not "until it feels done". The window's job is to let consumers that update on their own pick up a binary carrying the new key.
8. Only then stop the old one. Stop signing with it, then retire the KMS key. Anything still on an old copy at that point can't verify a new release and has to reinstall once. Its public half stays embedded and the frozen bucket stays published, indefinitely. Old copies still read that bucket, and anything the old key signed should go on verifying.
If I had to fit it on a sticky note on the monitor, it'd say new key, new address, leave the old bucket alone, and sign both before you retire either.
The escape hatch isn't finished
There's a third key embedded, and it signs nothing... or at least it isn't allowed to.
It's the rotation authority: generated once on an offline workstation, no expiry, private half in a safe as an encrypted USB stick and a paper backup produced with paperkey. Two copies: a USB stick bit-rots and paper burns, and those are different failure modes.
Its job is meant to be the emergency the ordinary rotation can't handle. The sequence above assumes you still control the outgoing key. When you don't, because it's been lost or compromised, there's no overlap to arrange, and the plan is for the authority to sign a document naming the replacement so installed copies can adopt it.
The plan being the operative word. It's a draft spec, and nothing in go-tool-base or go/signing (v0.11.2, as I write this) reads such a document. So today, lose a signing key early and you're on your own... you mint a new key, give it a new address and a new bucket, sign with it alone, and tell your users to reinstall once from the release page, because the copies in the field will refuse until they do. That's what the overlap protects you from, and it stays that way for now, until the emergency path gets built.
The key went into the binary on the day the signing keys did regardless, doing nothing, waiting. A binary can only ever be rescued by a key it already trusts, though that's necessary rather than sufficient: it also needs code that knows what to do with it, and copies shipped today don't have that.
It lives in a directory of its own (authority/), and from gtb v0.50.4 a generated tool's self-update will refuse a release signed by it, so you can't bridge the gap by signing with it either. A key that's only meant to rotate the others has no business vouching for a download.
The step I won't automate
The WKD tree is deployed by hand. Built locally, pushed to Cloudflare Pages with wrangler, using an API token scoped to Pages: Edit alone, on an account administered with a different email and a different MFA factor from the forge and the cloud account.
That path has no CI in it, no webhook and no Git integration.
That's an automation gap I put there and intend to keep. A second trust anchor is only any use if it doesn't lean on the same things as the first, and wiring its deploy into the pipeline that builds the releases would hand both to whoever gets into the runners. An attacker who owns the build system could then publish a key that vouches for what they built.
The site isn't the whole path, though. People find it by name, and DNS here is managed as code like everything else, so the name is the part that sits closest to CI. Pointing the key's hostnames somewhere else needs code on a protected branch and a deploy run from a release tag (or my own token), and a check with no credentials asks those names every six hours what they're serving. That's narrower than the clean separation I first had in my head, and the Cloudflare post is how I found out.
Two minutes of typing, a few times a year, to keep the publishing step out of the pipeline (and I'm not about to automate two minutes away).
What it cost
The key work turned out to be the easy bit. What took the time was the consumers downstream catching up one at a time, each with its own release cycle and its own opinion about when to cut one... it's a coordination job with a bit of cryptography attached, and I'd rather have worked that out before I started than on day four!
Originally published at phpboyscout.uk on 30 October 2026.
Top comments (0)