Self-hosts struggle to test data restores after failures
12 posts across 5 communities in the last 30 days; 67% from r/selfhosted.
“Well, in reality, my servers and my data live in separate places so it's highly unlikely that I would ever need both steps at the same time. Either the data gets corrupted, in which case the given process running in docker can't access it but is itself fully functional, at which…”
“Good point, a restore that brings files back but leaves the app broken doesn't prove much. What kind of app is it, and how do you check today that it still works after a restore, if you do??”
“How does additional equipment solves the problem of a bad backup concept? For example the Synology NASs can be made invisible, and most backup software can connect to the share before the backup and only for the time of the backup, so it is very very unlikely to be compromised…”
Multiple users acknowledge the importance of testing data restores but find it challenging. One user experienced a failed Immich update and struggled with restoring from backups, highlighting difficulties in recovering data and configurations. Another emphasizes thoroughness beyond basic file access during restore drills.
SOURCES (60)
“How often are you restoring from backup or migrating hosts? Maybe stop doing that. [re: How many containers is your media stack at now?]”
“My concern is that several copies can still share one failure domain if they all replicate the same deletion or corruption. I'd run a quarterly restore drill: recover a random date into an empty folder, compare file counts and checksums, then open a few videos and check their metadata. At least one copy should be versioned or offline. Until you've tested a restore, you don't know whether the backups work. [re: How I Organized, Culled, and Backed Up 60,000 Family Photos from iCloud]”
“Glad the rescan brought the pool back, but I'd want to know what state those 7 disks are really in before trusting the array again. Run a full scrub plus SMART long tests on all of them. Test your offsite replication now while things are calm. A pool that survived once isn't the same as a pool that's healthy. [re: My first TrueNAS Scale failure - oof]”
“Why didn't you just wipe and restore your backups? [re: My first TrueNAS Scale failure - oof]”
“Thanks for clarifying — so manual meals stay for recall but aren't in the backup/search DB yet. If meals landed in the same export as the logs, would that cover the restore case you're thinking about, or do you also need them searchable in-app? [re: I built a 12MB, no-account calorie counter with free barcode scanning (Open Food Facts)]”
“The restore test is the part everyone skips until the day it becomes the whole plan.”
“Well, in reality, my servers and my data live in separate places so it's highly unlikely that I would ever need both steps at the same time. Either the data gets corrupted, in which case the given process running in docker can't access it but is itself fully functional, at which point I need to restore the data with rsync. Or, the process in docker gets corrupted, at which point it required a rebuild or restart, but the data itself is completely unaffected.”
“You left out that second part in your original comment. That’s why I wasn’t sure if you were joking or not. Just doing compose up would not get you to the same point you were before a backup.”
“AI helped me tighten the wording. The setup is mine and runs on my own server: nightly pg_dump on a systemd timer, the last-success time as a textfile metric, the weekly restore drill, and the missing-metric alert I added after node-exporter came back without its textfile mount. No AI-generated project or code here, I'm just asking what people use.”
“Thanks, this is really helpful. Makes sense to keep it small and simple. Quick question: when you say check that the app starts, what would that look like for you? Like a website loading, or a database coming up? And roughly how much per month would feel fair for something like this?”
“Good point, a restore that brings files back but leaves the app broken doesn't prove much. What kind of app is it, and how do you check today that it still works after a restore, if you do??”
“I’ve been working in backup and storage for 9+ years, mostly with Commvault, Cohesity and Isilon, and I keep seeing the same problem. A backup job can show as “successful”, but when someone asks, “Can you prove the restore actually works?”, the answer isn’t always easy. Restore testing is often manual, not done regularly, and the immutable/offsite copies can be especially difficult to test. The results also aren’t always documented in a way that’s useful for audits. I’m thinking about building a”
“honestly the version mismatch between app and database has bitten me more times than I care to admit I do basically what you described but I also fire off a few API calls against the restored instance just to make sure the endpoints aren't returning garbage. caught a silent data corruption once where the web UI looked fine but the underlying data was toast”
“Expand the replies to this comment to learn how AI was used in this post/project.”
“A successful backup job seems like a different milestone from a usable restore. I’m interested in a small, repeatable check for self-hosted apps that doesn’t turn into a whole weekend project. My proposed minimum would be: restore into an isolated instance, restore the necessary configuration and secrets, sign in, open a known record and attachment, and check that user permissions still behave correctly. I would also keep outgoing email and webhooks disabled during the test so the restored insta”
“I've learned a lot about how to recover data from drives in this thread lol. I'll have to remember to come back here and read it next time I have a dead drive”
“One thing I would add: do the restore test once, on a fresh VM or another machine, before you trust the setup. I ran backups for almost a year before discovering my restic repo was missing half my postgres data because I excluded the volume wrong. The backup ran green every night. Worthless when I actually needed it. Syncthing plus a versioned tool is the right combo though, sync alone will happily replicate corruption.”
“Hi 👋 It's all in the title… Do you really test your backup restores regularly ? If so, how often ? If not, why ? What's the biggest thing holding you back ? submitted by /u/Benjamin-7933 [link] [comments]”
“I run FreeBSD with ZFS on the root disk drive. It automatically takes a snapshot before upgrade and can rollback to that (bectl feature).”
“you can. but you have to write a script to do it yourself. i.e. dump db backup using native tool to folder use pbs command line client to backup folder. that will work. on restore you do the inverse”
“My off-site is an old Dell Optiplex 390 with an i5-5th gen something. Mid-sized desktop case with a normal power supply, and BIOS set to boot after power loss. It's on a UPS and shuts down via apcupsd after 5 minutes. Two 8TB drives in software RAID 1, and boots from a small SSD. I run Borgbackup on it.”
“I run arcane in an LXC and back up the LXC. I should look at what arcane has too.”
“Yes but this would be just a sync, not a backup in the sense of: If I accidentally delete the wrong files, I have a backup.”
“I used to use duplicati, but the databases would get corrupted and do I stopped trusting it. I switched to using duplicacy. I have separate backups set up for different parts of my setup. One for next cloud, one for Plex, etc. There's a web UI you can use, but instead I wrote a wrapper for the CLI and have it parse the logs to write status updates to mqtt, which I input into a home assistant dashboard. My jobs are run by cron jobs, managed by the OS Web UI. For 3-2-1, I have a second compute”
“Get a large TB drive. Write an rsync script to copy home folders and data drives. AI can write the whole thing for you with logging.”
“I had ZFS set up like that but dismissed it. Mostly because I have to understand how ZFS works but since I only used it for the backup scripts and on no other machine, and since managing ZFS is not trivial, I did not trust this solution enough to keep it.”
I have a bash script that does a for loop on each folder and run rsync
“ZFS dataset snapshots and send them to another machine. I've been using this for years + Backblaze B2 for off site backup for the critical data”
“I’ve been looking at Home Assistant backups and noticed something I’d like to validate with people who actually rely on them. Home Assistant can create backups, but there doesn’t appear to be a built-in dry-run or pre-r…”
“That does sound confusing. Maybe they have a different setup going on or are using some third-party sync solution.”
“Not sure how old your synology is, but if anyone configured immutable snapshots you can get your data back. When customers refuse to pay for security products and follow best practices I always request they sign something stating so and agreeing that they read and understood the risks I outlined for them. Now is when you sell them as much as you can. Good luck.”
“How does additional equipment solves the problem of a bad backup concept? For example the Synology NASs can be made invisible, and most backup software can connect to the share before the backup and only for the time of the backup, so it is very very unlikely to be compromised. The networks I support survived this kind of attack. And a third tier backup with exchangable HDDs is a must, too.”
“We were hit about 8 years ago (an ancient Windows 2000 accounts box, and SMB1). Our planning saves us and we were able to restore everything in about 3 days ( one of the toughest 72 hour sessions I ever worked ). 3 layer backups with a true gapped off-site every night. That's what you need. Silver lining, we got to kill the accounts box for good and enforce compliance on across the whole accounts dept (and all their crappy legacy apps!)”
“A backup-age or last-restore-test widget would be genuinely useful. That’s the quiet failure I worry about most.”
“You know what, I need to test this more because my current test restore site is a local IP on a seperate, dedicated, WordPress container... Not a shared host... Give me a few days and I'll get this thoroughly tested on one of my own shared hosting accounts.. Thanks for pushing this point. These are the types of gotchas I really need to identify and iron out. I've been playing with this problem for situations where a migration is happening and DNS has not yet been cutover, but not tested”
“Bug Clicking "Create Backup Now" in Settings fails end to end. Confirmed live: \ \ \ tar: backup ...: Cannot stat: No such file or directory tar: Error exit delayed from previous errors. Root causes in \ lib/backup/snapshot.ts\ : 1. \ createBackup()\ runs \ tar czf "\${tarPath}" C "\${phoenixDir}" "\${backupId}"\ but the actual staged directory is \ \${phoenixDir}/backups/\${backupId}\ the \ C\ should be \ backupsDir\ , not \ phoenixDir\ . This alone causes the immediate crash. 2. Even once that”
“Yes you can roll back to a previous snapshot but you cannot restore from one. Learned this when I uninstalled a production app from the server instead of a users workstation cause I had the wrong window up. Luckally we had backups but it took 5 hours to restore it because unitrends”
Been there. I went through a similar cleanup a while back and this is the baseline that worked for me: scheduled backups you've actually tested a restore from (don't trust the "backup…
“One thing worth adding to the Veeam/snapshot/chaos-monkey answers already here: a restore job that reports "success" isn't the same claim as "the file is usable." I've seen backups that completed cleanly by every metric the tool exposes, but the restored data was corrupt, truncated, or from an app that needs a specific shutdown sequence to be consistent — none of which shows up as a job failure. The only way I've found to catch that class of problem is to actually”
“I used rsnapshot (built on top of rsync) literally decades ago and it looks like that is still a viable option. I just got Backrest (web UI for restic) set up within the past week, backing up to cloud storage I already had available, and it seems pretty solid so far.”
“If you haven't tested backup restoration, you don't have backups. This is a bizarre question. It depends entirely on your infrastructure and your environment. How long is a piece of string? How fast are you able to do it? Well, you decide when you build the backup system, based on the RPO/RTO objectives defined by the business. These are scary questions to asking about critical infrastructure.”
“We use everpure for our backups. We only use the snapshots that it takes of the volumes on the pure array for restoration. Hypothetically and in practice. I can roll back an entire volume to a previous point in time and it happens instantly. It’s all just pointers on the array. Some applications don’t like that or sometimes we just need one server out of a volume that could have 50 or even just 3 and you want to roll one back. Just copy the snapshot to make it another volume. Mount it in vsphere”
Veeam super easy. I have restored systems in 10-15 min consistently.
“It's unfortunately way too common, even at moderately sized organization that they don't do real cloud backup. I had a university client who had it as their strategy to use public cloud because then "they could save the backup cost". It's wild out there.”
“Wait, so your backup and your file stores are the same thing? I'm confused how you delete a real backup with a terminal command”
“After I had a disk failure and needed to get a restore, followed by a few config tweaks, etc, I noticed some errors in the supervisor log about backup.json missing from some of the backups, as per below messages. How can…”
“I think about the one guy whose backup plan was to do a nightly dd to another hard drive, and how he lost everything when the first drive corrupted. Given most people first starting up have no backup plan, that may be the reason it's not great to put the keys to all your castles on the self hosted That said, setting up Nextcloud with Keepass was the second thing I self hosted, after Homeassitant.”
“Small addendum on the restore point: I'd watch service UID/GID changes and share permissions closely. A flake can recreate every service correctly while the restored files still belong to the wrong identities. That's a very Nix-shaped way to end up with a successful rebuild and a broken server. Spell out persistent paths and ownership to avoid chasing the wrong problem.”
“The cadence and trigger halves are already answered above better than I would put them, so taking the two that are still open: who owns the test, and what a sandbox can actually validate. Ownership first, because it decides the rest. The one person who cannot prove a runbook is its author - they fill gaps from memory without noticing the gaps are there, which is the other side of navlio's point about the suspiciously clean drill. Give the test to whoever carries the recovery obligation for t”
“I'm convinced that doing restore tests manually on schedule is mandatory. Small one and really big ones. like actual disaster recovery. You never know if your emergency plans work until you've actually tested them If shit hits the fan you shouldn't run into problems you're not familiar dealing with”
“I'm on this ride too now. I tried a few of the other well known backup solutions but found them all so finicky to use in comparison when i started trying to do restore tests. My main pc is proxmox with 2 vms (ubunutu and win11), and i know i can get either back in minutes if I do something stupid.”
“Proxmxo makes it easy to schedule balckups. One on a different machine, one locally..”
“I’d like to be able to say that isn’t my strategy but I’ve accepted my single points of failure”
Use a boot schedule to boot at like 1am, then shutdown when done.