Mirroring is not backup

January 3, 2009

Once upon a time there was a small web hosting company called Lagomorphics LLC. They ran a blogging service called JournalSpace. It was hosted on Mac OS X servers. By way of backup, they had a second hard drive mirroring their main database.

At some point, they caught their IT guy stealing from the company. They fired him, but not before he managed to sabotage a number of servers.

During the run up to Christmas, someone or something wiped out the entire JournalSpace database. The mirroring software did its job perfectly, mirroring the zapped database across, wiping out the entire backup. So after 6 years, they lost everything. In addition, they had deliberately blocked other sites from archiving their pages, so recovery from the cloud is tricky too.

So, if you were a JournalSpace user, you just lost 6 years of your data. There are a number of lessons to learn here, including:

• Good IT people are valuable; bad IT people are very dangerous.
• Never rely on your web hosting company to keep backups.
• Live mirroring is a tool to ensure that your data is always available (“high availability”). It doesn’t necessarily help when something goes (“disaster recovery”).

I use a tool called rdiff-backup to back up my web sites. Like Apple’s Time Machine, it keeps timestamped snapshots, so you can go back in time to any previous point. I currently have daily snapshots going all the way back to April 2007. If I discovered tomorrow that someone had hacked into and corrupted my web site at the start of December, I could restore the snapshot from the end of November.

Also important is that both Time Machine and my rdiff-backup setup are automatic. I don’t have to remember to take backups; it just happens. Because rdiff-backup uses differential backups, this ludicrous level of backup only takes up 57MB. Also, since the tool works via SSH, I keep the backups on a machine in my house. I can recover from my web hosting provider losing everything; I can recover from my home server crashing. If both my web host and my home server died at once I’d be stuck, but I view that as an acceptably unlikely occurrance.

Filed under: System administration | Comments (0)