Yep, it was down due to a full disk.
Due to the number of spots that the RBN generates, there’s a lot of database activity, with rows being created and deleted every minute by the hundreds. By default, MariaDB/MySQL configures binary logging, which is used for two purposes: one, for replication of the database to another, and two, for redo logging (in the event something goes seriously wrong with the database and you have to restore from an older backup you can replay the binary logs to get to current state). It’s typically something that you don’t turn off because it’s a sensible default and disk is usually cheap.
For something like RBNHole, over the past three months, it’s generated about 10G of binary logs. I’d inadvertently cleaned out the logs around Christmas when troubleshooting another issue. 10G isn’t much, unless you’re running on a cloud hosting environment that charges according to RAM/storage size and you’re cheap, so you only went for the 512M/20G option.
Now, the RBNHole data is transient in the extreme - I’m not going to need to restore from backup because I can rebuild the server quicker than that, and the data in the database has a lifetime of minutes, not months. As for replication, there’s not much point: the thought of an RBNHole cluster makes me giggle more than slightly. So, I’ve disabled the binary logging, and this shouldn’t happen again.
Spots from RBN are flowing again into the database; there’s of course no CW activators out right now to verify, but I’ll take a look later in the day and check.