Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
view the rest of the comments
It depends on the sync / backup software
Syncthing uses a stored list of hashes (which is why it takes a long time for the initial scan), then it can monitor filesystem activity for changes to know what to sync.
Rsync compares all source and destination files with some magical high speed algorithm
Then, backup software does... whatever.
Back in the day on FAT filesystems they used the archive bit on each file's metadata, which was (IIRC) set during a backup and reset with any writes to that file. The next backup could then just backup those files.
Your current strategy is ok - just doing an offline backup after a bulk update, maybe it's just making that more robust by automating it...?
I suspect you have quite a large archive as photos don't compress well, and +2TBs won't disappear with dedupe... so, it's mostly about long term archival rather than highly dynamic data changes.
So that +2TB... do you drop those files in amongst everything else, or do you have 2 separate locations ie, "My Photos" + "To Be Organised"?
Maybe only backup "MyPhotos" once a year / quarter (for example), but fully sync "To Be Organised"... then you've reduced risk, and volume of backup data...?
Ahh ok, that makes sense. Hah magical algorithm.
Yeah it’s about 30TB of photos/videos. I only recently got into videography which takes up a ton of space. About 25% of that is videos converted into an editing codec, but I don’t have those backed up to external drives. I also have some folders excluded that I know have duplicates. A winter project of mine will be to clear out some of the duplicates, and then cull the photos/videos I definitely don’t need. I got into a bad data hoarding habit and kept everything even after selecting the keepers.
I have an in progress folder where I dump everything, then folders by year/month for projects and keepers. I need to do better with culling as I go.
I like that idea, I will incorporate it into my strategy.
Thank you for taking the time to help me out with this, much appreciated!