Digital clutter and storage costs

Previously, I mentioned about ridding myself of clutter. Being a modern day geek, I’m also a digital packrat rather than physical one. I have a staggering amount of approximately 2 TB worth of data consisting of movies, application installers, pictures, music and a ton of other stuff. I dedicated most of this weekend to clearing up my digital storage, ditching a large number of TV episodes that I would never intend to re-watch, for example. The minimalistic approach to living doesn’t end at the realm of physical belongings.

The cost of storage is not what it seems, despite the plummeting cost of hard drives. Although US$50 will net you a 500 GB drive these days, which may seem to work out really cheaply at $0.10/GB, that cost only covers the basic storage of data, and does not take into calculation the maintenance and upkeep for that bit of data. The cost per gigabyte when the upkeep is taken into account is much higher.

What do I mean by maintenance of data? To insure data against loss either through human fault (e.g. accidental deletion) or mechanical fault (e.g. faulty hard drive) or natural disasters (e.g. fire breaking out at home), that bit of data needs to be backed up on various levels. For me, this is done through a combination of redundant storage technologies for the live data, local backups and offsite backups.

Storage - It takes up more space than it intially appears to.

Assuming I have a file of 50 MB, the amount of space required to maintain that amount of data rapidly swells by at least a magnitude of 3 (one original, one local backup and one at the offsite backup location). Furthermore, at each location, versioning, that is to say, keeping multiple different copies of the file at different points in time, might need to take place if I frequently modify the file. This is to allow me to recover the original if I made changes to the file that I come to regret later. Even if I keep the copies through the use of an extremely efficient differentiation algorithm that only store the changed portion of the file rather than creating an entirely new file, that further adds on to size. My 50 MB file maybe taking up to 160 MB at this point, spreading across various storage platforms, some more expensive than others. Offsite backup through an online provide is notoriously expensive, for instance. At this point, the cost per gigabyte rapidly swells.

Despite the general mentality (largely propagated by storage providers themselves) that on modern day computer, there is no need reason to delete anything, that notion is detached from the truth.

Saved by RAID 5

Raid 5 rebuilding

I was greeted by a stream of loud and high pitched beeping from my server when I returned home on Friday. Panic set in that very moment, but it didn’t last. Within seconds, I had ascertained the source of the problem, and although it was worrisome one, it wasn’t the end of the world, at least not quite yet.

My first thought was “Doh! Server’s overheating!” but cooler minds prevailed. If the server had indeed overheated, it would have shut itself down instead of beeping madly, and it was still responsive. Well, I thought, the only other thing in the server that had an embedded beeper aside from the motherboard was the RAID controller card, so it had to be that. I have a RAID 5 array, consisting of four hard disks set up on the server, and a quick check showed that the status of the array was ‘critical’, with one drive failing.

In a RAID 5 array, a number of similar capacity drives are used, with the equivalent of one of the drive’s capacity being used for storing parity bits. Thus, a RAID 5 array can sustain the loss of one drive before data is irrecoverably lost.

So what did I do? I ran out the next day, bought myself a new 500 GB drive for SGD$87, replaced the failing drive, and a disaster that would have involved a very painful loss of data was averted. Since my RAID controller (a rather dated Highpoint RAID 1740) supported online rebuilding, no downtime was incurred at all.

Lesson to be taken from this: Data is precious and you never know when a drive could fail on you. If your entire life resides digitally on your computer, make sure you have a backup plan.