The death and rebirth of my home server
27 points by steinuil
27 points by steinuil
I have a RpiZero in my electrical closet reading the serial trames I get from my power distributor, and I set it up AGES ago (probably >8 years). I set it up so that the entire FS is read only, and it has been working lile a charm. It's been handling many (many) reboots due to power outages, and has come up every time.
I can switch it to R/W in ssh to update packages (recently did it to upgrade node-red), but it runs 99.9% of the time in RO.
This is the way, having read-only filesystem makes sur the SD card is still read, but very rarely written.
*edit:
pi@linky(ro):/var/lib/dpkg/info$ ls -lrt *.list | head
-rw-r--r-- 1 root root 412 18 avril 2018 gcc-4.6-base:armhf.list
pi@linky(ro):~$ cat /sys/fs/ext4/mmcblk0p2/lifetime_write_kbytes
17702665
Coming on 8.5 years and only 17G written on an 8G SD card !
TIL you can that easily check the lifetime writes on a filesystem ... decided to check on my Homeassistant Pi:
root@applepie ~ # ls -lrt created /var/lib/dpkg/info/*.list | head -1
.rw-r--r-- 683 root 4 Jul 2024 /var/lib/dpkg/info/zstd.list
root@applepie ~ # cat /sys/fs/ext4/nvme0n1p2/lifetime_write_kbytes
2501628881
Luckily, I'm running on an NVMe. 2.5 TB probably would have killed an SD card already ..
How did you setup the Pi as readonly? Just use a normal image, install whatever you need and set the root mount to ro? There's probably a lot of things that start complaining with a readonly root? I've dabbled with buildroot and a squashfs filesystem before for a music streaming Pi but I feel that's way too cumbersome for regular updates.
This is the way, having read-only filesystem makes sur the SD card is still read, but very rarely written.
In 2018 I found an (already at the time) old blog describing the procedure, it looked a lot like this one: https://www.dzombak.com/blog/2024/03/running-a-raspberry-pi-with-a-read-only-root-filesystem/
I also fixed 1 or 2 things related to node-red. But I didn't document everything.. Honestly once the thing boots correctly, then it's a matter of fixing the odd service requiring R/W and iterating.
If I had to redo everything, I'd do it with an ESP8266/ESP32 and EspHome though. It's easier to maintain, way less moving parts (and easy HA integration)
I also love myself a nice library of Linux ISO. I should take some notes for my own home server, good ideas.
I've also killed an sd card by running nix on a pi. I switched to a high endurance card, tuned linux to reduce writes and switched to f2fs. f2fs failed at some point and I lost the install. I switched to ext4 and after a while, the high endurance SD card failed too.
So I gave up and bought a usb3-sata adapter and a cheap sata ssd. I've killed a handful of sd cards, but I've never had an ssd die (not even the ultra cheap ones). I saw people online warn about running linux over a usb3 adapter, but I've never had a problem with it. But, using up a usb3 on the root disk limits how many disks you can attach.
Coincidentally, my pi had the exact same hostname as yours.
Yeah I’ve always thought the SD card was by far the worst thing about a pi. It’s annoying to work around and makes it so you can’t count on one for anything you care about.
Oh, I completely forgot about f2fs. Maybe I'll have another look at it.
I posted this article on hacker news as well and lots of people commented about Pis killing their SD cards. At some point I think I read about heat being a factor in the lifetime of an SD card too, and Pis tend to run very hot unless you get a decent heatsink and maybe a fan too, so that might be one of the causes too?
Also, some people on the orange site suggested getting an M.2 hat for the Pi, which would probably be better than running the OS through USB.
Coincidentally, my pi had the exact same hostname as yours.
Aha, another follower of the priestess of the ancient ones I suppose :)
That was a joy to read, thanks! Also subtle accidental(?) Nix propaganda advertisement, though I’m still not convinced and will stick with Arch (btw)