Recovering from a root 'rm' and why you need backups
Last week I was finally getting around to fixing my backup tooling and validating the recovery procedures. I thought it would be a good idea to have rolling nightly backups on a secondary ZFS pool with weekly syncs to an external disk. I ended up using this exact command to purge the rolling backups:
$ find ${BACKUP_FOLDER}/* -mtime +${DAYS_TO_KEEP} -exec rm {} \;
Before we go any further, I want to point out that I was using the variable ${BACKUP_DIR}
, not ${BACKUP_FOLDER}
.
Can you see where this is going? Yeah.. I didn’t pick that up before running my new majestic backup script. Half an hour later I came back to pages of ssh logs resembling /bin/rm: command not found
. Oops. I instantly knew what had happened, I’d completely written off my Proxmox host and I was lucky it removed the /bin/rm
binary before removing any actual data.