Previously, I discussed an issue we had with a PACS server losing a hard drive and also having its RAID configuration fail. This time the topic will be an issue we had with PACS data.

To make a long story short, some data was deleted off our servers due to an errant process. Notice I said servers, plural. All of our PACS data is stored in at least two places. Well both were affected, and had the data removed. While certainly annoying, this was not cause for concern. We also put all of our data on tape, so we would just restore from tape, right? Well, it was almost that simple. Everything went smoothly until we came across a bad tape. A few blocks of data refused to be read off this one tape. So far, two levels of defense have failed us… redundant data, and tape backups.

That point is worth dwelling on. It is usually pretty easy to have one level of defense for a given type of situation. In my post about the server hard drive dying, we had our first layer of defense, a redundant hard drive. Unfortunately, an unexpected issue with the RAID card took that hard drive out of commission as well. All too often, a single layer of protection is not enough.

Very fortunately for us, we did not stop at even two levels of redundancy; we have duplicated backup tapes, stored at a remote location, just in case. The second backup tape had the data, and we were back in business. The rest of the data restore was successful.

This time, we had just enough preparedness to end the day with smiles on our faces. What about next time? Several situations could have conspired against us to make it a REALLY bad day. If the data loss was accompanied by a physical disaster such as a fire that destroyed our server room, AND the bad tape of the two happened to be the one off site, we may not have been successful at completely restoring the data. It is always good to look back, even on successful events, to play the what-if game, so that you can be assured success next time. In our case, we had already been planning on getting the data stored in a third system, geographically separate from the two existing data stores. Such a solution does come at a cost, so the question has to be asked, how much planning is enough?

Bookmark to: