this post was submitted on 28 Jun 2025
90 points (97.9% liked)

Selfhosted

46672 readers
618 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
90
Docker Backup Stratagy (lemmy.world)
submitted 2 days ago* (last edited 2 days ago) by njordomir@lemmy.world to c/selfhosted@lemmy.world
28 comments fedilink hide all child comments
 

Hello Self-Hosters,

What is the best practice for backing up data from docker as a self-hoster looking for ease of maintenance and foolproof backups? (pick only one :D )

Assume directories with user data are mapped to a NAS share via NFS and backups are handled separately.

My bigger concern here is how do you handle all the other stuff that is stored locally on the server, like caches, databases, etc. The backup target will eventually be the NAS and then from there it'll be double-backed up to externals.

  1. Is it better to run #cp /var/lib/docker/volumes/* /backupLocation every once in a while, or is it preferable to define mountpoints for everything inside of /home/user/Containers and then use a script to sync it to wherever you keep backups? What pros and cons have you seen or experienced with these approaches?

  2. How do you test your backups? I'm thinking about digging up an old PC to use to test backups. I assume I can just edit the ip addresses in the docker compose, mount my NFS dirs, and failover to see if it runs.

  3. I started documenting my system in my notes and making a checklist for what I need to backup and where it's stored. Currently trying to figure out if I want to move some directories for consistency. Can I just do docker-compose down edit the mountpoints in docker-compose.yml and run docker-compose up to get a working system?

top 28 comments
sorted by: hot top controversial new old
[–] MangoPenguin@lemmy.blahaj.zone 1 points 6 hours ago* (last edited 6 hours ago)

I back up all the directories and docker-compose files using Restic (via Backrest) stored on Backblaze B2, and also the whole Docker LXC via Proxmox's backup function to a local HDD.

There's a chance some databases could be backed up in an unusable state, but I keep like 30-50 snapshots going back months, so I figure if the latest one has a bad DB backup, I could go back another day and try that one.

I also don't really have irreplaceable data stored in DBs, stuff like Immich has data in a DB that would be annoying to lose, but the photos themselves are just on the filesystem.

For testing Restic I pull a backup and just go through and check some of the important files.

Proxmox backup is really easy to test, as it just restores the whole LXC with a new ID and IP that I can check.

[–] glizzyguzzler@piefed.blahaj.zone 8 points 2 days ago (1 children)

You want your docker container’s persistent data mounted to real locations. I use the volumes for non-persistent stuff.

You want your real locations to have a file system that can snapshot (ZFS, BTRFS).

Then you can dump the superior Postgres databases and for all other databases and data, you stop the containers, snapshot, start the containers (limits downtime!), and then back up that snapshot. Thanks to snapshot, you don’t need to wait until the backup is done to bring the containers back up for data consistency. For backup I use restic, it does seem to work well, and has self-check functions so that’s nice. I chose restic instead of just sending snapshots because of its coupled encryption and checks, which allow for reliable data integrity on unreliable mediums (anyone, even giant providers, could blackhole bits of your backup!). I copy over the restic binary that made the backup using encrypted rclone, the encryption there prevents anyone (the baddies? Idk who’d target me but it doesn’t matter now!) from mucking with the binary if you did need that exact version to restore from for some reason.

Note I do not dump SQL or the like, they’re offline and get snapshotted in a stable state. The SQL dump scene was nasty, esp compared to Postgres’ amazingly straightforward way (while running!). I didn’t bother figuring out SQL dump or restore.

All of your containers should have specific users for them, specify the UID/GID so they’re easily recreatable in a restore scenario. (The database containers get their own users too)

Addendum for the specific users: Make an LXC container run by a specific user and put the docker container in it if the docker container is coded by F tier security peeps and hard requires root. Or use podman, it is competent and can successfully lie to those containers.

I don’t test my backups because the time to do so is stupid high thanks to my super low internet speeds. I tested restoring specific files with restic when setting it up and now I rely on the integrity checks (2GB check a day) to spot check everything is reliable. I have a local backup as well as a remote, the local is said snapshot used to make the restic remote backup. The snapshot is directly traversable and I don’t need to scrutinize it hard. If I had faster internet, I’d test restoring remote restic once a year probably. For now I try to restore a random file or small directory once a year.

Hope the rant helps

[–] njordomir@lemmy.world 1 points 2 days ago (1 children)

Interesting username. Are you a fellow student of Internet Comment Etiquette?

I know at least some of my containers use Postgres. Glad to know I inadvertently might have made it easier on myself. I'll have to look into the users for the db and db containers. I'm a bit lost on that. I know my db has a username and pass I set in the docker compose file and that the process is running with a particular GID UID or whatever. Is that what your talking about?

[–] glizzyguzzler@piefed.blahaj.zone 1 points 1 day ago

I do not know of Internet Comment Etiquette, sorry to disappoint! It’s a username that’s humorous to me and fits a core tenant of mine

Do remember (or put in the .env) the user/pass for your db’s, but they don’t matter much if you know them.

I’m talking about the process, the ‘user: 6969:6969’ in the docker.compose file. If it’s not there the container runs as the user running docker, and unless you’ve got non-root docker going it’ll run the containers as root. Which could be bad, so head that off if you can! Overall, I’d say it’s a low priority, but a real priority. Some naughty container could do bad things with root privilege and some docker vulnerabilities. I’ve never heard of it that kind of attack in the self hosted community, but as self hosting gains traction I worry a legit container will get an attack slipped in somehow and wreck (prob ransomware) root docker installations.

First priority is backup - then you can worry about removing root containers (if you haven’t already done so!).

[–] lazynooblet@lazysoci.al 25 points 2 days ago (1 children)

I'm lucky enough to run a business that needs a datacenter presence. So most my home-lab (including Lemmy) is actually hosted on a Dell PowerEdge R740xd in the DC. I can then use the small rack I have at home as off-site backups and some local services.

I treat the entirety of /var/lib/docker as expendable. When creating containers, I make sure any persistent data is mounted from a directory made just to host the persistent data. It means docker compose down --rmi all --volumes isn't destructive.

When a container needs a database, I make sure to add an extra read-only user. And all databases have their container and persistent volume directory named so scripts can identify them.

The backup strategy is then to backup all non-database persistent directories and dump all SQL databases, including permissions and user accounts. This gets run 4 times a day and the backup target is an NFS share elsewhere.

This is on top of daily backuppc backups of critical folders, automated Proxmox snapshots for docker hosts every 20 minutes, daily VM backups via Proxmox Backup Server and replication to another PBS at home.

I also try and use S3 where possible (seafile and lemmy are the 2 main uses) which is hosted in a container on a Synology RS2423RP+. Synology HyperBackup then performs a backup overnight to the Synology RS822+ I have at home.

Years ago I fucked up, didn't have backups, and lost all the photos of my sons early years. Backups are super important.

[–] hazl@lemmy.blahaj.zone 9 points 2 days ago (1 children)

That's a heart breaking way to learn your lesson about backups. It must have taken a long time to accept.

[–] pory@lemmy.world 9 points 2 days ago (2 children)

Everyone I know that actually keeps backups has the same kind of story. It's sad that no matter how many other people talk about keeping backups, it always takes a tragic loss like this to get people to buy hardware/subscriptions.

[–] lazynooblet@lazysoci.al 4 points 2 days ago

I hear you. I worked for an msp where some customers would refuse to invest in backup solutions and we either declined to renew their contract or they suffered an event and we were then setting up backups.

I was in the middle of a migration from OVH to Hetzner. I knew I had good backups at home so the plan was to blow away OVH and restore from backup to Hetzner. This was the mistake.

Mid migration I get an alert from the raid system that a drive has failed and had been marked as offline. I had a spare disk ready, as I planned for this type of event. So I swapped the disk. Mistake number 2.

I pulled the wrong disk. The Adaptec card shit a brick, kicked the whole array out. Couldn't bring it back together. I was too poor to afford recovery. This was my lesson.

Now I only use ZFS or MDRAID, and have multiple copies of data at all times.

[–] not_amm@lemmy.ml 1 points 1 day ago* (last edited 1 day ago) (1 children)

Damn, those stories scare me a lot because I can't afford good enough backups right now, so I rely on mirroring via Syncthing what I can and using an USB for my server (fortunately I don't have a lot of personal things). All the family memories are in multiple locations and devices so I don't have the time (nor storage) to backup those lol

Because of my situation, it takes me some time to save even for second hand things (I just bought a CPU and a new voltage regulator). I'm planning on buying some used HDDs as secondary storage and one as an external. Slowly but surely D:

[–] pory@lemmy.world 2 points 1 day ago

A second device on site is still infinitely more resilient than just letting it rock. Most use cases where a backup would help can be covered by an occasional one way sync or scheduled copy to a USB drive. Offsite is for catastrophes like your home burning down or flooding.

[–] greatley@ani.social 16 points 2 days ago* (last edited 2 days ago) (2 children)

I don't know if it's best practice but I mount everything to a specific folder instead of volumes. I also just stop all the containers before a backup instead of database dumping. Then just do an encrypted B2 backup using Restic.

So far I had to restore twice in six years of running this system and assuming the folder path is the same the only thing after downloading all the data I had to do was

docker-compose up -d
[–] dabe@lemmy.zip 1 points 7 hours ago* (last edited 7 hours ago)

yeah this is the way. as part of my borgmatic script I bring down all the stacks that have databases, let the backup run, then bring those stacks back up. as long as the containers aren’t running (and as long as the container properly closes itself down, not usually something to be worried about though), any method of data back up should be fine.

I do this with quadlets and systemd targets now but before I was doing it with a bunch of docker compose down commands.

It is quite convenient for restoration, as you say

[–] lorentz@feddit.it 4 points 2 days ago

I'd say that the most important takeover of this approach is to stop all the containers before the backup. Some applications (like databases) are extremely sensitive to data corruption. If you simply ´cp´ while they are running you may copy files of the same program at different point in time and get a corrupted backup. It is also important mentioning that a backup is good only if you verify that you can restore it. There are so many issues you can discover the first time you recover a backup, you want to be sure you discover them when you still have the original data.

[–] Sinirlan@lemmy.world 10 points 2 days ago (2 children)

I just took line of least effort, all my docker containers are hosted on dedicated VM in proxmox, so I just backup entire VM on weekly basis to my NAS. Already had to restore it once when I was replacing SSD in proxmox host, worked like a charm.

[–] dataprolet@lemmy.dbzer0.com 8 points 2 days ago (3 children)

While that's an easy solution it makes it impossible or rather difficult to restore single containers and/or files.

[–] MangoPenguin@lemmy.blahaj.zone 1 points 6 hours ago* (last edited 6 hours ago)

It's pretty easy, you can browse files in an LXC backup and restore specific parts. For VMs you can just restore the whole VM and copy out what you need.

[–] tofu@lemmy.nocturnal.garden 5 points 2 days ago

It's not that difficult. You can create a second VM from the backup with a few clicks and move the necessary data with scp.

[–] njordomir@lemmy.world 2 points 2 days ago

I miss this from cloud hosting. It's helpful to be able to save, clone, or do whatever with the current machine state and easily just flash back to where you were if you mess something up. Might be too much to set up for my current homelab though. My server does have btrfs snapshots of everything directly in grub which has let me roll back a few big screwups here and there.

[–] catloaf@lemm.ee 6 points 2 days ago (2 children)

Bind mounts. I've never bothered to figure out named volumes, since I often work with the contents outside Docker. Then I just back up the whole proxmox VM. (Yes I'm aware proxmox supports containers, no I don't plan to convert, that's more time and effort for no meaningful gain to me.)

You can restore that backup to a new VM. I just make sure it boots and I can access the files. Turn off networking before you boot it so that it doesn't cause conflicts.

[–] jacksilver@lemmy.world 3 points 2 days ago* (last edited 2 days ago)

Yep, bind mount the data and config directories and back those up. You can test a backup by spinning up a new container with the data/config directories.

This is both easy and generally the recommended thing I've seen for many services.

The only thing that could cause issues is breaking changes caused by the docker images themselves, but that's an issue regardless of backup strategy.

[–] ChapulinColorado@lemmy.world 1 points 2 days ago (1 children)

After getting a NAS to replace my raspberry pi 4 as a home server, I literally just SCPd the bind mounts and docker compose folder, adjusted a few env variables (and found out of a few I needed to add for things like the uid/guid the NAS used as default for the media user I created) and it took maybe 30 minutes total to be back and running. Highly agree with you from experience.

[–] catloaf@lemm.ee 2 points 1 day ago

Yup, it works great. I actually did it myself when migrating from a centos to debian host. Worked first try, no issues (except one thing that was already broken but I didn't know because I hadn't accessed it recently). Containers are great for this.

[–] plumbercraic@lemmy.sdf.org 9 points 2 days ago

I wouldn't backup the volumes directly. Better to use the mount points as you suggest then back up those mounted directories. If it's a database that usually needs to have its records exported into a backup friendly format. Typically I will do a db dump from a cron job in the host system to summon a script inside a container which writes to a mounted dir which is the thing that I back up.

[–] kewjo@lemmy.world 6 points 2 days ago* (last edited 2 days ago)

caches are never really a concern to me they will regen after the fact, from your description i would worry more about db, this is dependent though in what you're using and what you are storing. if the concern is having the same system intact then my primary concern would be backing up any config file you have. in cases of failure you mainly want to protect against data loss, if it takes time to regenerate cache/db that's time well spent for simplicity of actively maintaining your system

[–] bdonvr@thelemmy.club 3 points 2 days ago* (last edited 2 days ago)

BTRBK for easy BTRFS snapshots to an external drive.

Rsync to also upload those to B2 compatible storage (encrypted)

[–] dataprolet@lemmy.dbzer0.com 3 points 2 days ago

I use Borg to backup the default volume directory and my compose files. If you're interested I can share my backup script.

[–] confusedpuppy@lemmy.dbzer0.com 2 points 2 days ago

I created my own script/tool using rsync to handle backups and transferring data.

My needs are quite smaller with just a computer and two Raspberry Pi's but I found rsync to be really useful overall.

My backup strategy is to make a complete backup on the local device (Computer / RPi4 / RPi5) then copy all those backups to a Storage partition on my computer, then make a whole backup from the partition to an externally attached SSD.

The RPi's both use docker/podman containers so I make sure any persistent data is in mounted directories. I usually stop all containers before performing a backup, especially things with databases.

Everything in the docker containers is either hit or miss when it comes to restoring. The simple docker images restore as it they were untouched and will launch like nothing happened. I have a PieFed instance that must be rebuilt after restoring a backup. Since PieFed's persistent data is in mount points, everything works perfectly after a fresh build.

I can send a link to my rsync tool if that's any interest to anyone. I've found it super useful for backups and minimizes so much headache for myself when it comes to transferring files between different network connected devices.