https://www.gravatar.com/avatar/7ccf1999fbd849e739bbc3403990689d?s=240&d=mp

Daniel Mason

Site Reliability Engineer, Dad, Pilot

Recovering from a root 'rm' and why you need backups

Last week I was finally getting around to fixing my backup tooling and validating the recovery procedures. I thought it would be a good idea to have rolling nightly backups on a secondary ZFS pool with weekly syncs to an external disk. I ended up using this exact command to purge the rolling backups:

$ find ${BACKUP_FOLDER}/* -mtime +${DAYS_TO_KEEP} -exec rm {} \;

Before we go any further, I want to point out that I was using the variable ${BACKUP_DIR}, not ${BACKUP_FOLDER}.
Can you see where this is going? Yeah.. I didn’t pick that up before running my new majestic backup script. Half an hour later I came back to pages of ssh logs resembling /bin/rm: command not found. Oops. I instantly knew what had happened, I’d completely written off my Proxmox host and I was lucky it removed the /bin/rm binary before removing any actual data.