I used to think my recovery plan was complete because every application had a backup.
The database dumps were stored off-site. Volumes were archived. Retention rules were configured. I could point to a list of successful backup jobs and feel reasonably safe.
Then I asked a different question:
What happens if the machine running the management panel disappears?
The applications may still be running on their VPSs, but the panel contains the map that explains how everything fits together:
- which servers exist and how to reach them;
- encrypted SSH keys and credentials;
- application environment variables;
- domains and routes;
- backup destinations and schedules;
- monitoring and notification settings;
- team membership and permissions;
- job and audit history.
Losing an application is one kind of failure. Losing the system that knows how every application is managed is another.
This is the problem behind Nitrobase self-backup and control-plane recovery.
Nitrobase has only recently launched and is still in Beta. The recovery workflow does not remove the need to test backups or document infrastructure. Its goal is narrower: make the panel itself a recoverable component rather than an undocumented single point of failure.
Two different things need protection
An application backup and a control-plane backup answer different questions.
| Backup | The question it answers |
|---|---|
| Application backup | Can I recover this application's data and configuration? |
| Nitrobase self-backup | Can I recover the panel that knows about all servers and applications? |
The relationship looks like this:
┌────────────────── Nitrobase panel ──────────────────┐
│ Servers · Apps · Secrets · Domains · Schedules │
│ Monitoring · Teams · Jobs · Backup destinations │
└────────────────────────┬─────────────────────────────┘
│ manages
┌───────────┴───────────┐
▼ ▼
Application VPS A Application VPS B
database + volumes database + volumes
│ │
▼ ▼
app backup A app backup B
Panel state ─────► self-backup
If I keep only the two application backups, I still have the data. What I lose is the operational state required to manage it in the same way.
What goes into the self-backup?
Nitrobase stores its desired state in PostgreSQL. The self-backup creates:
- a
pg_dumpof the panel database; - a manifest describing the archive;
- a compressed tar archive;
- encryption of that archive using
ENCRYPTION_KEY.
PostgreSQL + manifest
│
▼
tar.gz archive
│
▼ encrypt with ENCRYPTION_KEY
*.tar.gz.enc + SHA-256 checksum
The encrypted file can be stored in the same backup destinations used by application backups, including S3-compatible storage, Cloudflare R2, Backblaze B2, or a server folder.
For disaster recovery, off-site storage matters. A backup located only on the failed panel server is not a recovery plan for that server.
The backup file is deliberately not enough
The panel database contains encrypted SSH credentials, application secrets, and destination credentials. Nitrobase encrypts those values at rest using ENCRYPTION_KEY, and the self-backup archive is also protected by that key.
This creates an important recovery rule:
encrypted backup + encryption key = recoverable state
encrypted backup − encryption key = unreadable archive
The key must not be stored only on the same machine as the backup source.
My minimum recovery kit therefore contains:
- the latest encrypted self-backup;
- its SHA-256 checksum;
- the corresponding
ENCRYPTION_KEY; - the installation instructions or release version;
- access to the backup destination;
- a separate way to access the managed VPSs if recovery fails.
The self-backup protects the database. A separate copy of the key protects the ability to read it.
A recovery timeline
Imagine the VPS running Nitrobase becomes unavailable, while the application servers remain online.
The recovery path is:
Panel VPS fails
│
▼
Install a fresh Nitrobase instance
│
▼
Choose “Restore from a backup” during setup
│
▼
Upload archive + checksum + old ENCRYPTION_KEY
│
▼
Verify → decrypt → import → migrate → re-encrypt
│
▼
Restored panel starts PAUSED
│
▼
Stop the old panel permanently, verify the new one
│
▼
Confirm “This database was moved — resume here”
The restore process verifies the checksum before trusting the archive. It loads the SQL through an unprivileged database role, applies the migrations required by the installed Nitrobase version, and re-encrypts stored secrets for the new installation's key.
That last step is useful: restoring the old state does not require the new panel to continue using the old encryption key forever. The old key can be used to unlock the backup, then the imported secrets can be protected with the new installation's key.
Why does the restored panel start paused?
This was the part of the design I initially underestimated.
A control panel is not only a database viewer. It actively runs jobs:
- scheduled backups;
- monitoring checks;
- automatic updates;
- deployment and recovery jobs;
- notification delivery.
If the original panel is still alive and a restored copy starts running the same schedules, two control planes may act on the same servers.
┌────► backup job
Old panel ───────┤
└────► deploy/update
▲
│ same VPS
┌────► deploy/update
New panel ───────┤
└────► monitoring repair
This is a split-brain problem. Even if both panels contain identical data, their in-memory locks and schedulers are independent.
Nitrobase stores an instance identifier in the database and keeps a matching marker in the panel's data volume. After restoring the database on another machine, the identifiers do not match. The new panel therefore starts in a paused state:
- no scheduled jobs;
- no monitoring actions;
- no new deployment jobs;
- no attempt to manage the connected servers.
Only after the administrator stops the original installation and explicitly confirms the move does the restored panel resume control.
The pause is not an inconvenience in the recovery flow. It is the safety boundary that prevents recovery itself from creating a second incident.
What happens to the running applications?
The applications live on their managed VPSs, not inside the Nitrobase panel container.
If only the panel server fails, existing application containers can continue running. What stops temporarily is centralized management, monitoring, and scheduled operations.
Panel unavailable ❌ deploys, schedules, monitoring
Application VPS ✅ existing containers may keep running
Public application ✅ remains available if its own server is healthy
This separation buys time, but it should not create complacency. A missed backup window, an unreported disk alert, or an application failure will remain unnoticed until the panel returns or another monitoring system detects it.
Key rotation adds one more responsibility
Encryption keys should be rotatable, but old backups remain encrypted with the key that existed when they were created.
Suppose the key changes from K1 to K2:
older backups ── encrypted with K1
new panel data ─ encrypted with K2
newer backups ── encrypted with K2
Deleting K1 immediately makes the older recovery points unreadable.
Nitrobase supports previous encryption keys so old application archives and self-backups can still be decrypted during the retention period. The old key should remain available until every backup created with it has expired or been replaced.
Rotation therefore has two completion conditions:
- current secrets have been re-encrypted successfully;
- every retained backup is either readable with an available previous key or has aged out.
Changing a key is an event. Finishing a key rotation is a lifecycle.
Backing up before upgrading the backup manager
Nitrobase also requires a self-backup before upgrading its own panel.
The upgrade sequence is:
wait for active jobs
↓
mandatory self-backup
↓
pull and start the new panel version
↓
health check
↙ ↘
healthy failed
keep it return to previous image
Returning to the previous image does not necessarily reverse a database migration. That is why the pre-upgrade self-backup remains necessary even when the container image can be rolled back.
Application version rollback and database recovery are different tools. The same is true for the control plane itself.
A backup is not verified until recovery has been rehearsed
A green “backup completed” badge proves that a file was written. It does not prove that I possess the key, can access the destination from a new machine, understand the takeover step, or can restore within an acceptable time.
My practical test looks like this:
- create a manual self-backup;
- download the encrypted archive and record its checksum;
- verify that the encryption key exists outside the panel server;
- install a temporary Nitrobase instance;
- restore the archive without resuming server control;
- confirm that servers, applications, domains, and schedules are present;
- remove the temporary recovery instance.
The paused state makes this kind of inspection safer because the restored panel does not immediately begin operating the real infrastructure.
The problem this feature is trying to solve
Self-hosting often concentrates knowledge in a collection of files, secrets, and one administrator's memory. A management panel makes that knowledge easier to use, but it also becomes state that needs its own recovery path.
The Nitrobase self-backup is meant to preserve that operational map. The checksum detects a damaged or incorrect archive. Encryption protects the stored credentials. The separate key requirement prevents the archive from becoming a plaintext bundle of infrastructure secrets. The paused restore prevents two panels from managing the same servers.
None of those mechanisms replaces a recovery rehearsal.
But together they answer the question I had originally forgotten to ask:
If the tool responsible for restoring everything else disappears, how do I restore the tool?
Nitrobase source code: https://github.com/AydenGroup/nitrobase
Top comments (9)
Great framing of the difference between losing an application and losing the system that knows how the applications are managed. One thing that helped us after the same realisation: the orchestration config itself (cron entries, retention settings) now lives in version control alongside the scripts it calls, so a rebuild of the control plane can start from a known-good state. Thanks for writing this up — it is a kind reminder to check the map, not just the destinations.
That's a great approach! Keeping orchestration configs in version control makes recovery much more reliable. Having a known-good state to rebuild from is definitely valuable. Thanks for sharing your experience!
Treating the control plane as its own recovery target is an underrated point. A test we've found useful: restore the panel onto a fresh machine using only what's in the backup plus a printed recovery doc, with no access to the old box.
Anything you had to look up elsewhere (a DNS login, a 2FA seed, an SSH key passphrase) is what the backup is missing.
Thank you for sharing this—restoring onto a fresh machine with no access to the old box is a simple but powerful test. I especially like the rule that anything you had to look up elsewhere is something the recovery plan is missing. I’ll add this to my recovery checklist.
The question "what happens if the machine running the panel disappears" is the right one, and xmrlist's follow-up is the trap I would worry about most. If the destination credentials and the encryption key both live inside the thing you are trying to restore, the backup exists but cannot be opened. Wbcom's test, restoring onto a fresh machine with only the backup and a printed doc, is a cheap way to find that out before an incident does.
The other half of control-plane safety is what the tool does when it is about to destroy something. At Ascendra Ventures we built CleanSync, a self-hosted tool that keeps Google Drive in two-way sync with your own server and keeps a copy of every Gmail message. rclone moves the files, and CleanSync's job is mostly deciding when not to. Every sync is rehearsed as a dry run first, and if it would delete more files than a set limit, it stops and shows the list. Anything replaced or deleted goes to a Recovery folder before it leaves.
Does the Nitrobase self-backup run its own restore check, or is the SHA-256 checksum the only verification for now?
That is exactly the distinction I would make.
For Nitrobase self-backups, the scheduled/manual “Verify archive” action currently downloads the encrypted archive and verifies its recorded SHA-256. It does not automatically perform a full test restore.
The actual first-run restore goes further: it verifies the checksum, decrypts the archive with the old ENCRYPTION_KEY, validates the archive layout and manifest, rejects unexpected SQL content, imports into PostgreSQL, runs migrations, and starts the restored panel paused to avoid split-brain.
So today the answer is: checksum verification is automatic; restore rehearsal is still a separate manual test. A useful next step would be restoring into an isolated temporary database, validating the resulting state, then destroying it without ever enabling jobs or contacting managed servers.
I like the CleanSync model of rehearsing destructive work and stopping above a deletion threshold. “The tool’s job is mostly deciding when not to” captures an important part of safe automation. Thanks for raising this.
This is the gap most people find only during a real incident. A few things that tend to bite when you actually restore a control plane:
Good write-up. Does the self-backup include the SSH host keys of the managed servers, or does the restored panel have to re-trust them?
Those are excellent additions—especially version pinning and testing recovery on a machine with no access to the original panel.
Nitrobase does not copy the servers’ private SSH host keys; those remain on the managed servers. It stores each server’s trusted host-key fingerprint in the panel database, so the fingerprint is included in the self-backup.
After restoration, the panel keeps verifying that saved fingerprint on every SSH connection. If the server presents the same host key, no re-trust is required. If it changed, Nitrobase refuses the connection and requires an administrator to verify the new fingerprint explicitly before trusting it.
The SSH credentials used by Nitrobase to authenticate to the servers are also stored encrypted in the panel database and restored with the control-plane backup. That makes protecting the external ENCRYPTION_KEY essential—the backup may contain the encrypted credentials and trusted fingerprints, but neither is useful if the key needed to decrypt them disappeared with the old panel.
The backup manifest also records the Nitrobase version. Restoration currently migrates the recovered database to the version installed on the fresh machine, so recording and initially restoring with the exact original version is still the more conservative recovery procedure.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.