We experienced a major database failure this morning and are in the process of doing a full recovery. We expect the recovery process to take appropximately 3 hours.

-Barry

Everything is back up... We apolgize for the inconvenience.

-Barry

Was this the big ugly full crash we always hope will never happen? If it was, it reinforces the importance of good backups, for everybody, as you never know who is next.

Good job!

Doug Wise
dougw@candleman.com

Setting aside the question of whether there are such things as a 'little nice-looking crash -- this one was nasty enough.

I'll let Barry or an Ops team member explain the gruesomes - but I understand a not uncommon kind of hardware failure [disks in the array designed to failover] was exacerbated by some driver problems as it was rebuilding data. The combination took out a significant chunk of the database which then required a rebuild and data check from scratch.

Things are looking quite good right now but I am sure the Ops teams are continuing to check.

Michael Farris
Infopop Product Evangelist

There was a hardware driver failure, which we now have a fix for, so this type of crash shouldn't happen again. This was one of those times that a full backup was needed, so we are glad that is being done.

-Barry