A Tail of Woe - Playing Hardware God and Losing Data

As many of you may know, I have a lot of computers in my home. I deal with huge amounts of data (mostly video but a lot of other stuff too) and just having it all online means I have more than 10 systems here.

But I have a core of 4 systems that I work with daily and that make up my primary set of working files: My old workstation (pacdat), my new workstation (video), my file server (NFS1) and my backup and domain name master (NETFS)

A few months ago I decided to move much of the data that still resided on my old workstation (P4 2.0GHz- called "pacdat") to a NFS file server (NFS1), including my home directory which is huge.

The old machine had several sets of mirrored drives of various sizes - usually the "sweet spot" size for whenever I purchased them - from 160 Gigs to 300 Gigs. My home directory has grown to outstrip each of these and in fact now has links to several such pairs of RAID 1 arrays. It was my intention to build a RAID 5 array of 320Gig drives that would do me for at least a year or so of growth at present rate - and host them on a single computer that I could mount from several of the systems in my home as needed.

All was going well - until Mother Nature stepped in a couple of weeks ago.

read on for the tail of woe

The synopsis:

• Two of the 6 drives in the RAID array lost tracks at the same spot at the same time it appears - could have survived one but not losing two at once
• the system first slowed down to a crawl - indicating there was a problem and making the copy to another system S L O W
• finally the computer itself died - first thought was the power supply
• nope - new power supply didn't work so replace mother board and RAM (and video cards but that's another story)
• finally gave up trying to recover any data - lost 6 months of "annoying stuff" - no documents or e-mail as they are on another system
• problems with the replacement system continued for two weeks
• now I have a fantastic workstation

The Details (and some thoughts and warnings)

As many of you know, I have a lot of computers around the house. Many are "archive" servers for video, and some are for backup of other systems both in-house and off-site.

I've had a series of personal workstations over the years - typically something near the leading edge but not quite on it. The system is more likely at the leading edge the of price-performance curve - best bang for the buck, at least when I purchase it. I also have the habit of "if it ain't broke, don't fix it" - so once I get my workstation to my liking, I tend to leave it alone for a year or two, or three.

My old workstation "pacdat" is a P4 2.6GHZ with 2 Gigs of RAM and IDE hard disk channels and started with a pair of mirrored (RAID 1) 160Gig drives. It's in a fairly large case with a hefty power supply, and over the course of 3 years I added 2 more pairs of drives, 200s and 300s, each time making another RAID 1 array of them.

My home directory outgrew each drive size in turn, and I ended up with some of my files spread across all three drive sets - which made some things a bit dicy. In fact, at one point I lost some files because I had thought I was deleting symbolic links but ended up deleting the real files. Fortunately I had backups on DVD - one of the last such backups I'll likely ever make as it ran to over 50 DVDs. I've used tapes in the past but today's drives and tapes are both expensive per Gig saved, and fairly slow unless you really spend a lot of money. I'd gotten to the point where I was turning to simply using hard drives as backup media because they truly are the least expensive bang for the buck for large storage.

Having RAID1 (mirror) arrays has proven reliable for me in the past, but I've done a lot with RAID5 (3+ drives with data spread across all such that losing any one drive does not lose data) in the past couple of years and I though I'd give it a try for my own storage this time.

So when I finally decided I'd put a new workstation together, I decided to put a RAID5 array in it. I got a mother board with 8 SATA hard drive ports on it, 4 "native" and 4 more on a secondary controller on the mother board. I purchased 8 320Gig Seagate SATA drives with the intention of setting things up with 7 in the array (giving me the equivallent of 6x320=1200+ Gigs or just over 1 Terabyte of usable storage) with the 8th drive being a "hot spare" just in case something happened to one of the initial 7 drives.

I purchased an Intel mother board - knowing that in general there were drivers for the Intel chipsets for Linux. I then spent the next week trying in vain to get the secondary drive controller working - it was not from Intel (I'll leave the really gory details for a tech article some time)