I used to use rsnapshot, which is a thin wrapper around rsync to make it incremental, but moved to restic and never looked back. Much easier and encrypted by default.
Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
I think the there are better alternatives for backup like kopia and restic. Even seafile. Want protection against ransomware, storage compression, encryption, versioning, sync upon write and block deduplication.
rsync for backups? I guess it depends on what kind of backup
for redundant backups of my data and configs that I still have a live copy of, I use restic, it compresses extremely well
I have used rsync to permanently move something to another drive though
It's slow?!?
Grsync is great. Having a GUI can be helpful
Yeah it’s slow
What's slow about async? If you have a reasonably fast CPU and are merely syncing differences, it's pretty quick.
It's single thread, one file at a time.
I was planning to use rsync to ship several TB of stuff from my old NAS to my new one soon. Since we're already talking about rsync, I guess I may as well ask if this is right way to go?
It depends
rsync
is fine, but to clarify a little further...
If you think you'll stop the transfer and want it to resume (and some data might have changed), then yep, rsync
is best.
But, if you're just doing a 1-off bulk transfer in a single run, then you could use other tools like xcopy
/ scp
or - if you've mounted the remote NAS at a local mount point - just plain old cp
The reason for that is that rsync
has to work out what's at the other end for each file, so it's doing some back & forwards communications each time which as someone else pointed out can load the CPU and reduce throughput.
(From memory, I think Raspberry Pi don't handle large transfers over scp
well... I seem to recall a buffer gets saturated and the throughput drops off after a minute or so)
Also, on a local network, there's probably no point in using encryption or compression options - esp. for photos / videos / music... you're just loading the CPU again to work out that it can't compress any further.
I couldn't tell you if it's the right way but I used it on my Rpi4 to sync 4tb of stuff from my Plex drive to a backup and set a script up to have it check/mirror daily. Took a day and a half to copy and now it syncs in minutes tops when there's new data
yes, it's the right way to go.
rsync over ssh is the best, and works as long as rsync is installed on both systems.
On low end CPUs you can max out the CPU before maxing out network---if you want to get fancy, you can use rsync over an unencrypted remote shell like rsh
, but I would only do this if the computers were directly connected to each other by one Ethernet cable.
I would generally argue that rsync is not a backup solution. But it is one of the best transfer/archiving solutions.
Yes, it is INCREDIBLY powerful and is often 90% of what people actually want/need. But to be an actual backup solution you still need infrastructure around that. Bare minimum is a crontab. But if you are actually backing something up (not just copying it to a local directory) then you need some logging/retry logic on top of that.
At which point you are building your own borg, as it were. Which, to be clear, is a great thing to do. But... backups are incredibly important and it is very much important to understand what a backup actually needs to be.
I would generally argue that rsync is not a backup solution.
Yeah, if you want to use rsync specifically for backups, you're probably better-off using something like rdiff-backup
, which makes use of rsync to generate backups and store them efficiently, and drive it from something like backupninja
, which will run the task periodically and notify you if it fails.
rsync
: one-way synchronization
unison
: bidirectional synchronization
git
: synchronization of text files with good interactive merging.
rdiff-backup
: rsync
-based backups. I used to use this and moved to restic
, as the backupninja
target for rdiff-backup
has kind of fallen into disrepair.
That doesn't mean "don't use rsync
". I mean, rsync
's a fine tool. It's just...not really a backup program on its own.
Having a synced copy elsewhere is not an adequate backup and snapshots are pretty important. I recently had RAM go bad and my most recent backups had corrupt data, but having previous snapshots saved the day.
+1 for rdiff-backup. Been using it for 20 years or so, and I love it.
Use borg/borgmatic for your backups. Use rsync to send your differentials to your secondary & offsite backup storage.
Ive personally used rsync for backups for about....15 years or so? Its worked out great. An awesome video going over all the basics and what you can do with it.
It works fine if all you need is transfer, my issue with it it's just not efficient. If you want a "time travel" feature, your only option is to duplicate data. Differential backups, compression, and encryption for off-site ones is where other tools shine.
I have it add a backup suffix based on the date. It moves changed and deleted files to another directory adding the date to the filename.
It can also do hard-link copied so that you can have multiple full directory trees to avoid all that duplication.
No file deltas or compression, but it does mean that you can access the backups directly.
Agree. It's neat for file transfers and simple one-shot backups, but if you're looking for a proper backup solution then other tools/services have advanced virtually every aspect of backups so much it pretty much always makes sense to use one of those instead.
And I generally enjoy Veronica's presentation. Knowledgable and simple.
Her https://tinkerbetter.tube/w/ffhBwuXDg7ZuPPFcqR93Bd made me learn a new way of looking at data. There was some tricks I havent done before. She has such good videos.
Yep, I found her through YouTube. Her and action retro's content is always great.with some Adrian black on the side.
I use rsync for many of the reasons covered in the video. It's widely available and has a long history. To me that feels important because it's had time to become stable and reliable. Using Linux is a hobby for me so my needs are quite low. It's nice to have a tool that just works.
I use it for all my backups and moving my backups to off network locations as well as file/folder transfers on my own network.
I even made my own tool (https://codeberg.org/taters/rTransfer) to simplify all my rsync commands into readable files because rsync commands can get quite long and overwhelming. It's especially useful chaining multiple rsync commands together to run under a single command.
I've tried other backup and syncing programs and I've had bad experiences with all of them. Other backup programs have failed to restore my system. Syncing programs constantly stop working and I got tired of always troubleshooting. Rsync when set up properly has given me a lot less headaches.
rsnapshot is a script for the purpose of repeatedly creating deduplicated copies (hardlinks) for one or more directories. You can chose how many hourly, daily, weekly,... copies you'd like to keep and it removes outdated copies automatically. It wraps rsync and ssh (public key auth) which need to be configured before.
Hardlinks need to be on the same filesystem, don't they? I don't see how that would work with a remote backup...?
If you want rsync but shiny, check out rshiny
The thing I hate most about rsync is that I always fumble to get the right syntax and flags.
This is a problem because once it’s working I never have to touch it ever again because it just works and keeping working. There’s not enough time to memorize the usage.
I feel this too. I have a couple of "spells" that work wonders in a literal small notebook with other one liners over the years. Its my spell book lol.
This is why I still don't know sed
and awk
syntax lol. I eventually get the data in the shape I need and then move on, and never imprint how they actually work. Still feel like a script kiddie every time I use them (so once every few years).
I've been using borg because of the backend encryption and because the deduplication and snapshot features are really nice. It could be interesting to have cross-archive deduplication but maybe I can get something like that by reorganizing my backups. I do use rsync for mirroring and organizing downloads, but not really for backups. It's a synchronization program as the name implies, not really intended for backups.
slow
rsync
is pretty fast, frankly. Once it's run once, if you have -a
or -t
passed, it'll synchronize mtimes. If the modification time and filesize matches, by default, rsync
won't look at a file further, so subsequent runs will be pretty fast. You can't really beat that for speed unless you have some sort of monitoring system in place (like, filesystem-level support for identifying modifications).
yeah, more often than not I notice the bottleneck being the storage drive itself, not rsync.
Can also use fpsync to speed things up. Handles a lot for you
I need a breakdown like this for Rclone. I've got 1TB of OneDrive free and nothing to do with it.
I'd love to setup a home server and backup some stuff to it.
Maybe I am missing something but how does it handle snapshots?
I use rsync all the time but only for moving data around effectively. But not for backups as it doesn't (AFAIK) hanld snapshots
You get incremental backups (snapshots) by using
--link-dest=DIR hardlink to files in DIR when unchanged
To use this you pass in the previous snapshot location as DIR and use a new destination directory for the current snapshot. This creates hard links in the new snapshot to the files which were unchanged from the previous snapshot, so only the new files are transferred, and there is no duplication of data on disk (for whole-file matches).
This does of course require that all of the snapshots exist in the same filesystem, since you cannot hard-link across filesystems.
yeah, it doesn't, it's just for file transfer. It's only useful if transferring files somewhere else counts as a backup for you.
To me, the file transfer is just a small component of a backup tool.
Tangentially, I don’t see people talk about rclone a lot, which is like rsync for cloud storage.
It’s awesome for moving things from one provider to another, for example.
I use cp and an external hdd for backups
that's great until it's not.
If you're trying to back up Windows OS drives for some reason, robocopy works quite similarly to rsync.
Ah.. robocopy... that's a great tool