The death of an SD Card
A couple of weekends ago I had a problem: I was bored. I needed to find something to tinker with, so I eyed the Raspberry Pi in the corner of our flat and SSH'd in.
The Raspberry Pi
I use this Raspberry Pi to run the Unifi Controller for the single
UAP-AC-LITE access point I use to blast WiFi all over the flat. I basically set
it up once, a (long) time ago and forgot about it, which also meant no one updated
it in a long while. The Pi was still running Raspbian Jessie, which came
installed on the official starter SD Cards at the time. I decided it was time to
upgrade to Stretch: step 1: upgrade everything that could be upgraded, step 2:
switch over the APT sources from
stretch and upgrade again.
The failed update
So I promptly ran
sudo apt update && sudo apt dist-upgrade and went to make a
cup of coffee. When I got back, the Pi was offline.
"Odd.. maybe it's restarting to install updates?" I thought. No Alex, this isn't
Windows. A hard reboot later and I'm back in an SSH session... when I notice (quite
bash couldn't do command or file autocompletion anymore.
The update had filled the tiny 16GB SD Card the Pi came in the box with.
"No problem", I think, "I'll just delete some of the .deb files I downloaded
rm-ing long filenames without file autocompletion is... a good way
to train one's patience. I knew the problem was bigger than first thought, when
after (successfully) deleting 100s of megabytes of files,
df -h still showed
the partition as being full.
fsck and a round of googling later: it looks like some SD cards go into
read-only mode when they fail (see this super helpful StackExchange answer).
The weirdest thing about it is that the SD Card's firmware still accepts writes,
it just commits them to disk.
Moral of the story
SD Cards will fail, at some point. Write-heavy workloads (like
are more likely to kill your Card.
I ended up buying a new SD Card and flashing Ubuntu 18.04 to it by following the official guide. The read-only mode on the old SD Card was a life-saver in the end as it allowed me to copy any old files I hadn't backed up (read: all of them!).