3-2-1... Backup
Ever since I got into computers semi-seriously, the wisdom of keeping multiple backups and the 3-2-1 backup strategy has been in my mind. Easy, right? Just have 3 copies of data on 2 forms of media with at least 1 off-site. This could be as simple as 1. The copy on your machine, 2. A copy on a flash drive, and 3. A copy on your cloud provider (ex. Google Drive). I'm not one to be simple, though my intentions weren't to take over 5 years to deploy a working system that I was happy with. A single drive isn't something I'm comfortable with. I've seen my mom's single drive backup solution give out one day, leaving us with a paper weight and a loss of family photos. So at least one part of the chain needed to be a RAID 5 if not RAID 6.
I was happy enough with that. A single server with a few TB of storage utilizing a RAID 5. My other requirement, though, was to have a web inteface where I could easily access and share files to my friends and family. A self-hosted Google Drive. If all you do is live in *NIX land and would rather use rsync and never leave the terminal, that's fine. But that's not my use-case. Looking at you Ángel.
For a while I used Nextcloud running on TrueNAS. Why I thought that was a good idea I don't know. My poor Athlon X4 910e was very unhappy with the overhead. I suffered along like this for a while. Residential 10Mbps upload speeds were not helping any of the usability either, meaning that for anything other than regular text files took forever to access outside of my home network.
Fast forward to today, I now have a semi-competent solution that is much faster and simpler. Enter File Browser and rsync-prelude with rsync. Filebrowser is a web inteface to access your files. It supports upload/download and sharing among a few other things. It's written in Go and has very little overhead. All you do is run it and point it at a directory. For that RAID 5 I mentioned, I moved to FreeBSD with ZFS and set up a RAIDz1 across my three 2TB drives which allows for one drive failure before data loss. None of this solves the upload speed, though. For that, I decided to set up an identical server with a mirror of the data using a spare HP mini computer and an external 2TB drive. This will sit next to my computer at school, and I can use it as though it's the one back home. Each night, the two will sync taking as long as they like and I'll have two copies of my data. This does mean that there will be a full day in which I will not have three copies of the data, but I'm willing to live with that. Syncing the two servers became a bit of an issue, however. Rsync by default has no way of knowing if you've moved files around. See this superuser StackExchange post linking to a hack using hard links. This wasn't working for me, and after spending much too long trying to get another software, Syncthing, to work, I gave up as it was unable to handle the large number and size of the files well given my hardware. Luckily, I found rsync-prelude. Written in Python, it reorders files on the destination server ahead of time so that rsync will not re-transfer them just because they changed folders. Then you run rsync like normal. Easy(er). This will be run as a cronjob every day to sync up the backup server at home.
I'm very happy with this setup. The rsync process has yet to be vetted, but even if something goes awry, the single backup solution paired with File Browser is so much better than before. I'm able to play video out of the 10Mbps network where on Nextcloud it would stutter and hang (presumably because of system overhead). PDFs load and preview properly, where as on Nextcloud it just would error out. A great step up.
For anyone currious, here are the system specs:
-- Home Server -- CPU: AMD Athlon X4 910e RAM: 10GB DDR3 HDD: Seagate Barracuda 250GB 7200RPM (boot) HDD: 3x WD Purple 2TB 5400RPM (RAIDz1 storage) OS: FreeBSD 14.1 (ZFS) -- School Server -- CPU: Intel i5-4570T RAM: 8GB DDR3 SSD: Samsung 850 Pro 256GB (boot) HDD: WD MyPassport 2TB 5400RPM (storage) OS: FreeBSD 14.1 (ZFS)
I'm about to move into a new appartment, and I'm crossing my fingers for higher upload speeds given that it's a fiber connection and I'm in the city. This will make accessing my server much more useable when I'm away from the appartment. If you have any questions about setting up a system yourself, feel free to send an email to alex@scerba.org.
Created: 2024-08-16