A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
The thing I hate most about rsync is that I always fumble to get the right syntax and flags.
This is a problem because once it’s working I never have to touch it ever again because it just works and keeping working. There’s not enough time to memorize the usage.
I feel this too. I have a couple of "spells" that work wonders in a literal small notebook with other one liners over the years. Its my spell book lol.
This is why I still don't know sed and awk syntax lol. I eventually get the data in the shape I need and then move on, and never imprint how they actually work. Still feel like a script kiddie every time I use them (so once every few years).
sed can do a bunch of things, but I overwhelmingly use it for a single operation in a pipeline: the s// operation. I think that that's worth knowing.
sed 's/foo/bar/'
will replace all the first text in each line matching the regex "foo" with "bar".
That'll already handle a lot of cases, but a few other helpful sub-uses:
sed 's/foo/bar/g'
will replace all text matching regex "foo" with "bar", even if there are more than one per line
sed 's/\([0-9a-f]*\)/0x\1/g
will take the text inside the backslash-escaped parens and put that matched text back in the replacement text, where one has '\1'. In the above example, that's finding all hexadecimal strings and prefixing them with '0x'
If you want to match a literal "/", the easiest way to do it is to just use a different separator; if you use something other than a "/" as separator after the "s", sed will expect that later in the expression too, like this:
sed 's%/%SLASH%g
will replace all instances of a "/" in the text with "SLASH".
One trick that one of my students taught me a decade or so ago is to actually make an alias to list the useful flags.
Yes, a lot of us think we are smart and set up aliases/functions and have a huge list of them that we never remember or, even worse, ONLY remember. What I noticed her doing was having something like goodman-rsync that would just echo out a list of the most useful flags and what they actually do.
So nine times out of 10 I just want rsync -azvh --progress ${SRC} ${DEST} but when I am doing something funky and am thinking "I vaguely recall how to do this"? dumbman rsync and I get a quick cheat sheet of what flags I have found REALLY useful in the past or even just explaining what azvh actually does without grepping past all the crap I don't care about in the man page. And I just keep that in the repo of dotfiles I copy to machines I work on regularly.
tldr and atuin have been my main way of remembering complex but frequent flag combinations
Yeah. There are a few useful websites I end up at that serve similar purposes.
My usual workflow is that I need to be able to work in an airgapped environment where it is a lot easier to get "my dotfiles" approved than to ask for utility packages like that. Especially since there will inevitably be some jackass who says "You don't know how to work without google? What are we paying you for?" because they mostly do the same task every day of their life.
And I do find that writing the cheat sheet myself goes a long way towards me actually learning them so I don't always need it. But I know that is very much how my brain works (I write probably hundreds of pages of notes a year... I look at maybe two pages a year).
Most Unix commands will show a short list of the most-helpful flags if you use --help or -h.
I've been using borg because of the backend encryption and because the deduplication and snapshot features are really nice. It could be interesting to have cross-archive deduplication but maybe I can get something like that by reorganizing my backups. I do use rsync for mirroring and organizing downloads, but not really for backups. It's a synchronization program as the name implies, not really intended for backups.
I think Arch wiki recommends rsync for backups
I need a breakdown like this for Rclone. I've got 1TB of OneDrive free and nothing to do with it.
I'd love to setup a home server and backup some stuff to it.
Tangentially, I don’t see people talk about rclone a lot, which is like rsync for cloud storage.
It’s awesome for moving things from one provider to another, for example.
@calliope It’s also great for local or remote backups over ssh, smb, etc.
It has been remarkably useful! I keep trying to tell people about it but apparently I am just their main use case or something.
I would have loved it when I was using Samba to share files on my local network decades ago. It’s like a Swiss Army knife!
I tried rclone once because I wanted to sync a single folder from documents and freaked out when it looked like it was going to purge all documents except for my targeted folder.
Then I just did it via the portal...
rsync can sometimes look similarly scary! I very clearly remember triple-checking what it’s doing.
rclone works amazingly well if you have hundreds of folders or thousands of files and you can’t be bothered to babysit a portal.
It's fine. But yes in the Linux space. We tend to want to host ourselves. Not have to trust some administrator of some cloud we don't know/trust.
rclone does support other protocols besides S3. You can also selfhost your own S3 storage.
slow
rsync is pretty fast, frankly. Once it's run once, if you have -a or -t passed, it'll synchronize mtimes. If the modification time and filesize matches, by default, rsync won't look at a file further, so subsequent runs will be pretty fast. You can't really beat that for speed unless you have some sort of monitoring system in place (like, filesystem-level support for identifying modifications).
yeah, more often than not I notice the bottleneck being the storage drive itself, not rsync.
Can also use fpsync to speed things up. Handles a lot for you
Maybe I am missing something but how does it handle snapshots?
I use rsync all the time but only for moving data around effectively. But not for backups as it doesn't (AFAIK) hanld snapshots
yeah, it doesn't, it's just for file transfer. It's only useful if transferring files somewhere else counts as a backup for you.
To me, the file transfer is just a small component of a backup tool.
If you want rsync but shiny, check out rshiny
I use cp and an external hdd for backups
that's great until it's not.
If you're trying to back up Windows OS drives for some reason, robocopy works quite similarly to rsync.
Rsync is great. I’ve been using it to back up my book library from my local Calibre collection to my NAS for years, it’s absurdly simple and convenient. Plus, -ruv lets me ignore unchanged files and backup recursively, and if I clean up locally and need that replicated, just need to add —delete.
Rsync depends on OpenSSH, but it definitely isn't SFTP. I've tried using it against an SFTPGo instance, and lost some files because it runs its own binary, bypassing SFTPGo's permission checks. Instead, I've opted for rclone with the SFTP backend, which does everything rsync do and is very well compliant.
In fact, while SFTPGo's main developer published a fix for this bug, he also expressed intention to drop support for the command entirely. I think I'm just commenting to give a heads up for any passerby.
Y’all don’t seem to know about rsbackup, which is a terrible shame for you.
(I mean the one on greenend.org.uk!)
Here’s how I approach old and slow: