Selfhosted

52965 readers

901 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz

Create a retention period for online backup storage (lemmy.world)

submitted 3 days ago by HereIAm@lemmy.world to c/selfhosted@lemmy.world

8 comments fedilink hide all child comments

I finally got some online storage to backup my home server to, and I can finally sleep easy when my house inevitably goes up in flames. My initial push has been with rclone using the sync command. I plan on running a daily or weekly push.

I've gone with Hetzner and they have a variety of options on how to connect to their storage box as they call it. They have an option for automatic snapshotting of, as I understand it, the entire storage box that you can then recreate.

So my thoughts are around rclones sync and copy commands, and the storage snapshots.
Copy) If I use copy, it won't clear up deleted files automatically that I actually want gone.
Sync) With sync I'm worried that say a drive failed (my set up is a bit wonky currently until I can build a proper RAID), it will do a sync, see an entire directory is gone, and go ahead and delete it.
Snapshots) Here I'm worried that any new data that's been added in-between the snap shot and a server failure would be lost. And I don't you can retrieve partial data from a snapshot.

What I have in mind is to use rclone's copy command, but then somehow have a retention period on files that no longer exist on my server. So a deleted file might exist in my storage bucket for a month before being removed from there as well.

How do you guys manage this? Are there any ready made solutions, or do I need to whip up a script that goes through and looks at the last modified dates and removes the old ones?

top 8 comments

sorted by: hot top controversial new old

[–] 123213217362131231@lemmy.world 2 points 1 day ago* (last edited 1 day ago)

If you wanna backup your mobile device as well, I can recommend Resticopia

It's based on Restic and you can choose retention rules 👍

[–] eager_eagle@lemmy.world 10 points 3 days ago* (last edited 3 days ago) (2 children)

I think you need something like restic with a retention policy

https://restic.readthedocs.io/en/stable/060_forget.html#removing-snapshots-according-to-a-policy

--keep-{hourly,daily,weekly,monthly,yearly}

other solutions that implement similar policies are kopia and rustic

the advantage of using an off the shelf solution is that it's almost certainly more reliable than what anyone can come up with in a few hours, and, it works with incremental backups, so your space requirements are drastically reduced depending how often you run it.

[–] olivier@lemmy.fait.ch 5 points 3 days ago

you might want to add borg, restic and - more recently - plakar to that list

[–] Jason2357@lemmy.ca 1 points 2 days ago

Restic is great, and the de-duplication between snapshots is amazingly good. Same content in different files (e.g. tar files of linux systems) take very little space like magic). Backrest is a nice web frontend for it.

Note that you should use some retention features of your provider to manage the risk of ransomware deleting your backups.

[–] pi3r8@lemmy.world 3 points 2 days ago (1 children)

I have been trying out duplicati for backups the last few weeks and so far it's doing what I need and I think would offer the features you are looking for.

[–] HereIAm@lemmy.world 1 points 2 days ago

Yeah that looks cool. I will give it a closer look tomorrow :)

[–] passenger@sopuli.xyz 1 points 2 days ago

Use duplicati, or if you want command line, use borg.

[–] non_burglar@lemmy.world 2 points 3 days ago

Scripting it isn't that tricky.

I'm old and use rsync, with mtime and not(!) operands and I'm able to keep 7 daily, 4 weekly, and 4 monthly backups. Runs every day, the rest of the backups are pruned in this rolling window.