Intelerad really won us over years ago with their responsive communication. In sharp contrast to the other vendor under heavy consideration at the time (Agfa), we saw our questions answers and our concerns addressed in a reasonable timeframe time and again. I imagine a significant number of other customers would echo my statement. And with good customer service, an influx of customers is likely to follow.

It might just be that Intelerad went through some growing pains, which ultimately led to my post a few weeks back. Our customer service had declined a bit, and coupled with a new product install and associated “issues” it meant we were not the happiest of customers. Part of this is surely our fault… it is one thing to beta test something like MS Word… but if it doesn’t work quite right, you likely won’t be horribly inconvenienced. The same cannot be said for the world of radiology. It might just be that we have learned our lesson (they don’t call it the bleeding edge for nothing).

Thankfully, there has been much progress on the mammo viewer front. We received an Inteleviewer update that seems to have addressed a number of issues (even some that we just reported, either they are psychic or they have impossibly quick programmers). Other issues have been addressed in a manner that either gives us hope for a timely resolution, or at the very least offers us solace in the knowledge it is being worked on diligently, even if the results might not be forthcoming for a while yet.

I don’t think there is anything remotely resembling a trouble free PACS system. When certain problems arise, all you have to fall back on is vendor support; their ability to solve the issue, and their ability to communicate effectively. Intelerad hasn’t been perfect in this regard, but at the end of the day, as an IT professional, I’d still choose to have Intelerad in my corner over some of the alternatives.

Bookmark to:
         

So let’s take a typical DR setup, for a typical company. This may not reflect your situation exactly, but it is likely going to give you food for thought, and that is all I can hope to do.
SamCo is a company that houses all of its servers in a central location at HQ. They have multiple offices, all connected to HQ. Let’s focus on a specific database server, ServerDB. ServerDB has all the typical precautionary configurations, RAID, redundant power, etc.  Data is being written to tape nightly and taken offsite.

The worry is that a disaster hits and the server is unavailable (for the sake of simplicity, all the other servers are still functioning). So you get another server, maybe even the exact same server hardware, and stick it at your shiny new DR site.

Using some solution, all data from ServerDB is automatically sent to ServerDB2. Everything is golden, you feel ready for anything (so long as ‘anything’ only affects ServerDB!). But just because ServerDB can see ServerDB2, and they are replicated, doesn’t necessarily mean you are done. How do all the clients connect to ServerDB? How are they going to be “programmed” to connect to ServerDB2 when needed? Maybe this is a policy; maybe there is a technological solution. Oh wait, users in HQ can connect to ServerDB2, but what about the other offices? There may be network changes you need to do.

With that taken care of, everything is certainly golden now, right? Maybe we should look at various “disaster” scenarios…

A VIP user goes and deletes a VIP file/data element on the ServerDB. Ah, this one is simple, you don’t even need your DR site for this… get the backup tape and restore the data. Oh wait, the backup tape failed. Good thing you have a very expensive DR site, where all the data is replicated to. Oh, that’s right, replicated servers replicate the data, but they also replicate the deletions. Better hope Mr. VIP didn’t have a hand in signing off on the DR setup, otherwise he is likely to determine that since it didn’t allow him to recover from his disaster, it wasn’t a worthwhile purchase, and the person who requested the DR site may be in for a bit of trouble.

Or maybe the situation is a bit different. Perhaps HQ suffers from an issue which causes ServerDB to go down, and also a loss of connectivity to the DR site. The DR site isn’t very useful if HQ can’t connect to it. No problem, you anticipated this, and asked for redundant networking. However, the decision was made that HQ could make keep busy without ServerDB/ServerDB2 so the redundant networking wasn’t approved. Trouble is, all of the remote sites connect to HQ in a star topology. With HQ down, none of them can access ServerDB2 at the DR site either. Not good.

There are countless scenarios and variations that could occur. I think the hardest part of a DR site/ disaster preparedness strategy isn’t the technical aspects… there is always someone that can provide the answer to a question for a price. The difficult part is knowing what questions need asking, what problems need solving. You really need to do a very thorough analysis of all systems, and try to run through all the possible breakpoints in various combinations. No system is entirely safe. Even Gmail goes down from time to time. The trick is to try to identify the issues you consider most likely, and find the balance between financial outlay, and potential return on investment for each scenario. Of course, when the scenario deemed too expensive to account for given its likelihood of occurrence actually happens, management will likely forget about all the caveats you presented them with when the DR site was approved. I’m still looking for an answer to that one myself!

Bookmark to:
         

There was a backend upgrade that we were very excited to do, as our tests indicated a fantastic improvement in speed for our users. We tested and tested, trying to account for every scenario, as this was a rather large backend change. The biggest concern we had was that once users started using it and inputting data, rolling back became exceedingly difficult. If it was going to break, we wanted to break right away.

The day of reckoning came, and we crossed our fingers. There were no functional differences, so if an issue was going to occur, it was likely to be something quite bizarre. A couple of hours in, we came face to face with our bizarre tormentor. All of a sudden, mid morning, everything ground to a halt. Our super fast upgrade was not so fast after all.

Let me break for an interesting aside. RIS vs PACS: which is more ‘critical’ to business continuity? Looking at this from a workflow perspective (ignoring which system contains data that is more important), our RIS outage highlighted an answer very quickly. We have faced PACS outages, and certainly while disruptive and impactful to the business, staff was able to keep moving… slowly sure, but workflow remained intact. Some credit goes to the distributed nature of Intelepacs. There are single points of failure, but they are few. Modalities can still take images. Rad’s could view images on modalities, dictations could happen with Dictaphones. Certainly none of this is ideal or enjoyable for those impacted, but work did not stop dead. With the RIS down, it did. The RIS moves information around, and automates so many processes, that without it, staff simply didn’t know what to do next. Support staff can manually push items through the system, but doing so is very, very slow for a small team. So while workflow didn’t completely stop, it was practically at a standstill.

The slowness issue we faced was in fact a direct result of the speed gain we enjoyed. As it turns out, we had a query that was very poorly constructed. In our ‘slow’ system prior to upgrade, it never had the resources to get out of hand, so we never noticed it. Under our new system, so many instances of this particular query were able to run, that our server was starved of resources. Restarting it worked for a few moments, but quickly we were down again.

These series of blog posts are about disaster preparedness, not about programming stories, so how does this fit in? The ‘bug’ in our system affected one specific screen. If we could avoid using that screen, everything would be fine. However, the way the system was designed, that screen was needed by everyone to move the study to the next step. It became painfully obvious to us that if only we had put some of this functionality on other screens, work could have still continued. We managed to find and correct this rogue query in a quick, but still unacceptable, amount of time. The next thing we did was add in that functionality to the other areas. I’m sure some people would see this as potential code bloat, but while this gave us a measure of comfort that should something similar happen in the future, we’d be ok, it also made a lot of sense from a usability standpoint, we just ignored it up until that point, while we tackled other issues.

The moral this time is that sometimes you can’t possibly be prepared. While technically, there are ways we could have stress tested the system prior to rollout, they just aren’t feasible in our small shop. Having been through this experience once, it certainly opens our eyes to how we design in the future. We try to be more aware of having multiple methods to achieve a result.  It also highlights the need to revisit systems periodically, and check the performance. Find a baseline, and put checks in place to notify someone if that baseline is exceeded. You may not be able to predict what segment of a system may decide to break on you, but after you do it a few times, you may surprise yourself how often you are indeed able to.

You can never be too prepared…

Bookmark to:
         

To make a long story short, some data was deleted off our servers due to an errant process. Notice I said servers, plural. All of our PACS data is stored in at least two places. Well both were affected, and had the data removed. While certainly annoying, this was not cause for concern. We also put all of our data on tape, so we would just restore from tape, right? Well, it was almost that simple. Everything went smoothly until we came across a bad tape. A few blocks of data refused to be read off this one tape. So far, two levels of defense have failed us… redundant data, and tape backups.

That point is worth dwelling on. It is usually pretty easy to have one level of defense for a given type of situation. In my post about the server hard drive dying, we had our first layer of defense, a redundant hard drive. Unfortunately, an unexpected issue with the RAID card took that hard drive out of commission as well. All too often, a single layer of protection is not enough.

Very fortunately for us, we did not stop at even two levels of redundancy; we have duplicated backup tapes, stored at a remote location, just in case. The second backup tape had the data, and we were back in business. The rest of the data restore was successful.

This time, we had just enough preparedness to end the day with smiles on our faces. What about next time? Several situations could have conspired against us to make it a REALLY bad day. If the data loss was accompanied by a physical disaster such as a fire that destroyed our server room, AND the bad tape of the two happened to be the one off site, we may not have been successful at completely restoring the data. It is always good to look back, even on successful events, to play the what-if game, so that you can be assured success next time. In our case, we had already been planning on getting the data stored in a third system, geographically separate from the two existing data stores. Such a solution does come at a cost, so the question has to be asked, how much planning is enough?

Bookmark to:
         

We were not using AD integration or SSO prior to moving to 7602 (from 7.5.x).

Initially, in the Domain Scan and Active Directory settings, I had entered our domain in the post 2000 format… ie domain.com. Turns out that doesn’t work. I had to switch back to our pre 2000 domain name.

I made the mistake the first time of filling the Computer Account with the name of the computer ServiceDesk was installed on. This replaced the actual computer account for that computer, causing issues. Use a new name that will never be used by any existing or new computer account.

When entering info into the Pass Through Authentication section, it would give warning/link messages, the first about computer account not being made, the second about setting the password. I checked and the computer account was made, so i ignored the errors. I shouldn’t have. I had to use the second link to set the password to match what I entered on the Pass Through Authentication page. Run this on the domain controller.

Restart the servicedesk service.

Ensure IE has the web address of your servicedesk server entered in as a site in the Local Intranet under Security.

If using firefox, type about:config in the URL box. Then search for:
network.automatic-ntlm-auth.trusted-uris
Double click and enter the web address(es) prefixed by http(s)://. If you have multiple URLs, separate them with a comma (,).

Completely exit/restart the web browser.

In our experience (and I did do reinstalls/tests several times), following the above steps worked reliably. Hope this helps.

Bookmark to:
         

After a flurry of activity the past couple years, we are now taking a step back to see where to reassert some energy. Sometimes, when things are growing and changing at a quick pace, it is easy to push aside important, but not seemingly urgent, matters. One such area that has our renewed focus is our disaster preparedness.

I imagine we are in the same boat as other medium sized businesses, no matter the industry. We create and house lots of important data, across various systems, and it is consumed by many staff members in multiple locations. The volume of our data may be higher than average, but there are certainly industries that have much larger numbers. Our data and our reliance on technology is critical to our business, but the same could be said for a multitude of organizations these days.

At the onset of our various initiatives over the past while, we had always made sure to cover the basics. Is hardware redundant? Do we have backups? Is data stored in an offsite location? And by and large, it has been sufficient. We have had emergencies, we have had outages, and we have even ‘lost’ lots of data. However, each time, within a reasonable (from my perspective anyway!) timeframe, we were able to recover and resume normal business workflow. We have never actually lost any data; it has always been recoverable from a redundant location or backup.

So, if we have had such success, why give it any further thought? Isn’t the status quo good enough? Well, no. Any outage is still a disruption, and the more we can do to prevent them, the better off the business is.

The phrase “hindsight is 20/20” rings so true when looking at disaster recovery and prevention. Over the coming posts, I will use real incidents we faced, how we dealt with them at the time, and how we could handle (or prevent them!) in the future.

Most recently, we had a hard drive failure in one of our clinic PACS servers. Truth be told, we have had numerous hard drive failures. However, in all other instances, the business was not impacted at all. We planned for this, and as such have RAID in all of our servers. Every other time, we simply responded to pre-programmed alerts (those are key), and the system recovered to a happy state silently in the background, without affecting staff in the slightest.

This most recent hard drive failure happened to be accompanied by a malfunction of a RAID controller, so we were in fact down. Staff at that location could not process patients very well at all. How did this play out? Staff called us, complaining of a seemingly minor issue. We spent time chasing that issue, which eventually led us to the actual problem of the drive/controller failure. We dispatched someone to the site, and they switched all the equipment manually over to a backup PACS server. Thanks to Intelerads architecture, this redundancy is very straight forward. This process took time, in the meanwhile, staff were using some documented (and some creative off-the-cuff) processes to keep workflow moving, albeit at a slower pace. Simultaneously to the switchover, staff also used a spare drive to replace the dead drive, allowing our PACS vendor to get the server itself back in service.

What can this experience teach us? Well, for one, there is no fool proof plan. We thought we had a dead hard drive covered, but had not anticipated what we faced. Secondly, we were only informed because staff noticed issues… the server itself was still running (not properly mind you), so our alerts did not fire off. Actually, one alert did trigger, but it was incorrectly configured as a low priority alert, that we hadn’t noticed in time. We relied on a manual switch over process. While portions of it could be (and now are) documented and programmed so staff could switch to the backup server with just a button press, there still is equipment that does not have that ability, and does require a technical resource to re-program. That said, technology is available in the way of load balancers which could have (possibly) greatly assisted the fail over.

While we were in a failed over state, waiting for the server to become operational again, workflow was fairly normal, but it was a tad slower than normal. If we did not have a spare part on hand, or something more drastic had failed on the server, we would have had to rely on hardware support to kick in and provide replacement equipment, which could have taken days, and still more time for Intelerad to configure the replacement server. This leads to the question of should we have had a spare, configured, server ready to go? And that appropriately enough takes me back to the question… how much is enough?

Bookmark to:
         

A while back Intelerad put a bug in our Radiologist’s ear about a mammo viewer they were working on. We had been using Threepalm’s offering, which was a fairly nascent offering as it was. The biggest gripe we had with 3palm was that it worked off a separate worklist, which is far from an ideal workflow. Feature wise it was adequate, and after getting some bugs ironed out, it was working. Not to our liking per say, but it was working.

The mammo viewer built into Inteleviewer also works. However to say it is not meeting our expectations would be an understatement. At least we are back to a single worklist workflow.

Let’s touch on some of the good points:

Single worklist. Yes I had already mentioned this, but this is definitely the best “feature”. At the same time that we went to the Inteleviewer mammo viewer, we also got their new worklist. Aside from some quirks, it has been well received. It really is a significant leap forward from Intelerads old Reporting Worklist tool.

The interface. It isn’t so much that the interface is good, rather, since it is essentially Inteleviewer, it is familiar. Threepalm’s layout was quite unique and somewhat non-traditional. It made it difficult for our Rads to learn their way around. MG in IV plays to IV’s strength, it is laid out logically, and features are fairly easy to find.

Administration. Managing the Intelerad world is actually quite easy. Granted, this may be more a case of being liking what you are familiar with, but to me and my team, it lessens the burden. Not to mention having two dicom applications fighting for disk space is never a good thing.

Unfortunately, at the moment that is all I can come up with for positive things to say about Intelerads mammo viewer. I feel it prudent to stop and clarify that the software is not classified as finished yet. Intelerad openly admits they have much to learn about the mammo world. Yes they do.

Now I will touch on some of the not so good aspects.

The first item is not actually a flaw of the mammo viewer at all, but it is directly tied to our mammo workflow. The new worklist function that I previously stated was a significant improvement, has one troubling flaw. It is SLOW. Specifically, whereas the previous version made a case available immediately to the Rad once completed by a Tech, this version has a background process that needs to run, and until it does, the Rad will not be presented with the case, regardless of how frantically (s)he press the refresh button. Initially this was set to several minutes. Intelerad fairly quickly responded to our complaints and lowered the cycle to 60 seconds. However, given our workflow, this is still much too long. Our Techs will mark the case complete, ready for dictation, and then step directly into the reading room to have the Rads confirm everything is in order before sending the patient on their way. Each time they do his, there is the potential for up to a minute of lost productivity while everyone waits for the worklist to present the case on the screen for selection. If you work in radiology you likely understand that over the course of a working day, this results in a very significant loss of productivity, and as such is the bane of the Rads. The part of this that is so odd is that Intelerad seems to find it hard to believe this “little” delay is such a big problem. Perhaps we have an entirely unique workflow… but I don’t expect such to be the case.

Once Rads get the case open, Things don’t improve much. Ok, maybe that is a little harsh. While the mammo viewer does not yet have many modality specific tools (we are assured those are coming in later releases), the fact that it leverages existing Inteleviewer tools means it is certainly adequate. The hanging protocols, while not yet user editable, cover the basics well enough.

The other major gripe we have is around the ability (or inability as the case may be) to handle priors. There are two distinct issues. First of all, frequently the system doesn’t realize there are priors, so they are not displayed automatically for the Rad. We are still going back and forth with support trying to convince them there is a problem. When the software does decide to play nice and show priors, or when the Rad manually pulls them up, it has trouble aligning the priors. Most of the time, the issue has to do with extra issue captured in the scan, which the viewer faithfully tries to display, at the cost of pushing the image all over the view port. Our previous Threepalm software did suffer from this as well, but to a much less significant degree. There are times however when the software just throws reason out the window and essentially hides the breast image outside of the viewable region of the screen. When his happens, you are lucky if part of the marker is still viewable to give some indication that the breast is there somewhere. The image can be panned around so that it comes back into view, but a whole lot of panning is required, and it really does not leave a positive impression with the Radiologist.

Here is hoping that Intelerad can come through and address our major complaints. If so I will gladly post an update.

Bookmark to:
         

Bookmark to:
         

Bookmark to:
         

Bookmark to: