Well it happened again. Our mysql/sql server lost 2 more hard drives Monday night. It is running a raid 5 array and one hard drive goes down no problem you just replace it and it rebuilds, but you lose to and you lose pretty much everything. The odds of losing 2 drives at the same time is so unlikely that its considered a reliable set-up.
The thing is this is the second time in 4 months that we lost 2 hard drives simultaneously in this server. It runs 4 total and in November 08 we lost two at the same time. We replaced them but left the other two in place. Of course on Monday those 2 failed. Luckily this time we had backups and the site were only down for a few hours. The biggest lesson learned? If you have multiple hard drive fail in a machine simultaneously go ahead and replace any that didn’t fail just in case.