In an effort to decrease our No Show rate, we have enlisted the assistance of Televox, to automatically place appointment confirmation calls. We had done reference checks (which I highly recommend you do!) and were told to watch out for snags during implementation. Otherwise, the feedback was stellar. Sure enough, that is exactly our experience thus far.

Televox is a service that takes information from a company, and uses that information to place telephone calls for you. I am sure it can be used to handle many situations, but we feed it basic information so it can remind our patients of upcoming exams. The company has access to a large number of telephone lines, so it can place many calls simultaneously. It allows the recipient of the call to choose various options, in our case, to confirm or cancel the appointment. It then feeds the results back to us, so we can act accordingly.

Getting all the technical details sorted out was pretty straight forward for us. Others may find this part a bit more challenging if you need to enlist the assistance of a vendor, however our software is developed in house. We found that the documentation they provided did not accurately reflect the resulting data file they sent back to us, so some minor modifications had to be made. They were good about taking some custom values we use, so we could better integrate the results.

The issues we have faced have been more nuisance and annoying items, rather than complete show stoppers. It has been the service surrounding the issues that has left much to be desired. We stated the schedule that the calls need to follow, yet for a couple weeks now, there has been a hiccup preventing calls on a particular day from being placed. We have to manually request they be placed. For this to happen once is possibly acceptable (given it is still the early stages), but the issue wasn’t resolved as promised, and that is the part that is frustrating. We had another issue where we agreed to a nightly schedule to receive information from them, and yet they did not have the data ready at the agreeable time, causing manual work that could have been avoided.

Hopefully these two issues are resolved shortly, and we can stop spending time babysitting what is supposed to be an automatic process. Aside from those issues, early data suggests the service is producing results. It will be interesting to see how our numbers are affected over the long term. Perhaps these early growing pains will be easily forgotten about when the decrease of appointment no shows is considered.

Bookmark to:
         

Intelerad stepped up to the plate (perhaps not as quickly as we would have liked) to deal with the challenges we faced as a result of that upgrade, and has been doing quite a few little and big upgrades to get us back to an efficient workflow. The backend changes are difficult for us to test, but thankfully, they have thus far been trouble free. It is the frontend changes with Inteleviewer I want to touch on. Prior to this whole issue, we generally rolled out an upgrade to all the Rads at once. And luckily, everything went smoothly. We have adjusted that process, and now designate particular reading rooms as test beds. As it turns out, we made this adjustment in the nick of time.

I feel I should step back for a moment and state that Intelerad is trying really hard to get us upgrades that solves various particular issues. For all I know, we are getting these upgrades as soon as they are done being coded (with some brief internal testing). Considering the situation, this is an acceptable short term situation. Anytime something is sped through the process, you gain time, but you may give up quality (although you may not, risks must be weighed).

We received one build of the software that promised many significant fixes, and we were very impressed with those fixes when we did our initial testing prior to upgrading any of the Rad stations. However, during that initial testing, we also found a pretty nasty bug that had made its way into the software. It was obvious enough that we were able to find it without involving a Rad for testing. Upon bringing this to Intelerad’s attention, it was verified, and a fix was provided in short order. We proceeded to retest the fixed version, and found no other noticeable issues, so we upgraded a Rad station. A couple Rads rotated through, and were happy with the improvements. And then a Rad noticed a change in behavior that was not desirable. I hesitate to call it a bug, it may have been, or maybe it was by design, but either way, it was a change that was met with frustration. It was only noticeable during particular workflow, that only some Rads follow, which is why the first couple of Rads had no problems. We have had to hold off on the rollout pending a resolution to this issue, which we hope comes quickly, as we are anxious to get the Rads using this much improved version.

Every company is different, but I strongly suggest there be some sort of testing phase prior to upgrading any critical system. A little time spent up front can save a whole slew of time spent cleaning up a potentially avoidable mess.

Bookmark to:
         

It might just be that Intelerad went through some growing pains, which ultimately led to my post a few weeks back. Our customer service had declined a bit, and coupled with a new product install and associated “issues” it meant we were not the happiest of customers. Part of this is surely our fault… it is one thing to beta test something like MS Word… but if it doesn’t work quite right, you likely won’t be horribly inconvenienced. The same cannot be said for the world of radiology. It might just be that we have learned our lesson (they don’t call it the bleeding edge for nothing).

Thankfully, there has been much progress on the mammo viewer front. We received an Inteleviewer update that seems to have addressed a number of issues (even some that we just reported, either they are psychic or they have impossibly quick programmers). Other issues have been addressed in a manner that either gives us hope for a timely resolution, or at the very least offers us solace in the knowledge it is being worked on diligently, even if the results might not be forthcoming for a while yet.

I don’t think there is anything remotely resembling a trouble free PACS system. When certain problems arise, all you have to fall back on is vendor support; their ability to solve the issue, and their ability to communicate effectively. Intelerad hasn’t been perfect in this regard, but at the end of the day, as an IT professional, I’d still choose to have Intelerad in my corner over some of the alternatives.

Bookmark to:
         

So let’s take a typical DR setup, for a typical company. This may not reflect your situation exactly, but it is likely going to give you food for thought, and that is all I can hope to do.
SamCo is a company that houses all of its servers in a central location at HQ. They have multiple offices, all connected to HQ. Let’s focus on a specific database server, ServerDB. ServerDB has all the typical precautionary configurations, RAID, redundant power, etc.  Data is being written to tape nightly and taken offsite.

The worry is that a disaster hits and the server is unavailable (for the sake of simplicity, all the other servers are still functioning). So you get another server, maybe even the exact same server hardware, and stick it at your shiny new DR site.

Using some solution, all data from ServerDB is automatically sent to ServerDB2. Everything is golden, you feel ready for anything (so long as ‘anything’ only affects ServerDB!). But just because ServerDB can see ServerDB2, and they are replicated, doesn’t necessarily mean you are done. How do all the clients connect to ServerDB? How are they going to be “programmed” to connect to ServerDB2 when needed? Maybe this is a policy; maybe there is a technological solution. Oh wait, users in HQ can connect to ServerDB2, but what about the other offices? There may be network changes you need to do.

With that taken care of, everything is certainly golden now, right? Maybe we should look at various “disaster” scenarios…

A VIP user goes and deletes a VIP file/data element on the ServerDB. Ah, this one is simple, you don’t even need your DR site for this… get the backup tape and restore the data. Oh wait, the backup tape failed. Good thing you have a very expensive DR site, where all the data is replicated to. Oh, that’s right, replicated servers replicate the data, but they also replicate the deletions. Better hope Mr. VIP didn’t have a hand in signing off on the DR setup, otherwise he is likely to determine that since it didn’t allow him to recover from his disaster, it wasn’t a worthwhile purchase, and the person who requested the DR site may be in for a bit of trouble.

Or maybe the situation is a bit different. Perhaps HQ suffers from an issue which causes ServerDB to go down, and also a loss of connectivity to the DR site. The DR site isn’t very useful if HQ can’t connect to it. No problem, you anticipated this, and asked for redundant networking. However, the decision was made that HQ could make keep busy without ServerDB/ServerDB2 so the redundant networking wasn’t approved. Trouble is, all of the remote sites connect to HQ in a star topology. With HQ down, none of them can access ServerDB2 at the DR site either. Not good.

There are countless scenarios and variations that could occur. I think the hardest part of a DR site/ disaster preparedness strategy isn’t the technical aspects… there is always someone that can provide the answer to a question for a price. The difficult part is knowing what questions need asking, what problems need solving. You really need to do a very thorough analysis of all systems, and try to run through all the possible breakpoints in various combinations. No system is entirely safe. Even Gmail goes down from time to time. The trick is to try to identify the issues you consider most likely, and find the balance between financial outlay, and potential return on investment for each scenario. Of course, when the scenario deemed too expensive to account for given its likelihood of occurrence actually happens, management will likely forget about all the caveats you presented them with when the DR site was approved. I’m still looking for an answer to that one myself!

Bookmark to:
         

There was a backend upgrade that we were very excited to do, as our tests indicated a fantastic improvement in speed for our users. We tested and tested, trying to account for every scenario, as this was a rather large backend change. The biggest concern we had was that once users started using it and inputting data, rolling back became exceedingly difficult. If it was going to break, we wanted to break right away.

The day of reckoning came, and we crossed our fingers. There were no functional differences, so if an issue was going to occur, it was likely to be something quite bizarre. A couple of hours in, we came face to face with our bizarre tormentor. All of a sudden, mid morning, everything ground to a halt. Our super fast upgrade was not so fast after all.

Let me break for an interesting aside. RIS vs PACS: which is more ‘critical’ to business continuity? Looking at this from a workflow perspective (ignoring which system contains data that is more important), our RIS outage highlighted an answer very quickly. We have faced PACS outages, and certainly while disruptive and impactful to the business, staff was able to keep moving… slowly sure, but workflow remained intact. Some credit goes to the distributed nature of Intelepacs. There are single points of failure, but they are few. Modalities can still take images. Rad’s could view images on modalities, dictations could happen with Dictaphones. Certainly none of this is ideal or enjoyable for those impacted, but work did not stop dead. With the RIS down, it did. The RIS moves information around, and automates so many processes, that without it, staff simply didn’t know what to do next. Support staff can manually push items through the system, but doing so is very, very slow for a small team. So while workflow didn’t completely stop, it was practically at a standstill.

The slowness issue we faced was in fact a direct result of the speed gain we enjoyed. As it turns out, we had a query that was very poorly constructed. In our ‘slow’ system prior to upgrade, it never had the resources to get out of hand, so we never noticed it. Under our new system, so many instances of this particular query were able to run, that our server was starved of resources. Restarting it worked for a few moments, but quickly we were down again.

These series of blog posts are about disaster preparedness, not about programming stories, so how does this fit in? The ‘bug’ in our system affected one specific screen. If we could avoid using that screen, everything would be fine. However, the way the system was designed, that screen was needed by everyone to move the study to the next step. It became painfully obvious to us that if only we had put some of this functionality on other screens, work could have still continued. We managed to find and correct this rogue query in a quick, but still unacceptable, amount of time. The next thing we did was add in that functionality to the other areas. I’m sure some people would see this as potential code bloat, but while this gave us a measure of comfort that should something similar happen in the future, we’d be ok, it also made a lot of sense from a usability standpoint, we just ignored it up until that point, while we tackled other issues.

The moral this time is that sometimes you can’t possibly be prepared. While technically, there are ways we could have stress tested the system prior to rollout, they just aren’t feasible in our small shop. Having been through this experience once, it certainly opens our eyes to how we design in the future. We try to be more aware of having multiple methods to achieve a result.  It also highlights the need to revisit systems periodically, and check the performance. Find a baseline, and put checks in place to notify someone if that baseline is exceeded. You may not be able to predict what segment of a system may decide to break on you, but after you do it a few times, you may surprise yourself how often you are indeed able to.

You can never be too prepared…

Bookmark to:
         

To make a long story short, some data was deleted off our servers due to an errant process. Notice I said servers, plural. All of our PACS data is stored in at least two places. Well both were affected, and had the data removed. While certainly annoying, this was not cause for concern. We also put all of our data on tape, so we would just restore from tape, right? Well, it was almost that simple. Everything went smoothly until we came across a bad tape. A few blocks of data refused to be read off this one tape. So far, two levels of defense have failed us… redundant data, and tape backups.

That point is worth dwelling on. It is usually pretty easy to have one level of defense for a given type of situation. In my post about the server hard drive dying, we had our first layer of defense, a redundant hard drive. Unfortunately, an unexpected issue with the RAID card took that hard drive out of commission as well. All too often, a single layer of protection is not enough.

Very fortunately for us, we did not stop at even two levels of redundancy; we have duplicated backup tapes, stored at a remote location, just in case. The second backup tape had the data, and we were back in business. The rest of the data restore was successful.

This time, we had just enough preparedness to end the day with smiles on our faces. What about next time? Several situations could have conspired against us to make it a REALLY bad day. If the data loss was accompanied by a physical disaster such as a fire that destroyed our server room, AND the bad tape of the two happened to be the one off site, we may not have been successful at completely restoring the data. It is always good to look back, even on successful events, to play the what-if game, so that you can be assured success next time. In our case, we had already been planning on getting the data stored in a third system, geographically separate from the two existing data stores. Such a solution does come at a cost, so the question has to be asked, how much planning is enough?

Bookmark to:
         

After a flurry of activity the past couple years, we are now taking a step back to see where to reassert some energy. Sometimes, when things are growing and changing at a quick pace, it is easy to push aside important, but not seemingly urgent, matters. One such area that has our renewed focus is our disaster preparedness.

I imagine we are in the same boat as other medium sized businesses, no matter the industry. We create and house lots of important data, across various systems, and it is consumed by many staff members in multiple locations. The volume of our data may be higher than average, but there are certainly industries that have much larger numbers. Our data and our reliance on technology is critical to our business, but the same could be said for a multitude of organizations these days.

At the onset of our various initiatives over the past while, we had always made sure to cover the basics. Is hardware redundant? Do we have backups? Is data stored in an offsite location? And by and large, it has been sufficient. We have had emergencies, we have had outages, and we have even ‘lost’ lots of data. However, each time, within a reasonable (from my perspective anyway!) timeframe, we were able to recover and resume normal business workflow. We have never actually lost any data; it has always been recoverable from a redundant location or backup.

So, if we have had such success, why give it any further thought? Isn’t the status quo good enough? Well, no. Any outage is still a disruption, and the more we can do to prevent them, the better off the business is.

The phrase “hindsight is 20/20” rings so true when looking at disaster recovery and prevention. Over the coming posts, I will use real incidents we faced, how we dealt with them at the time, and how we could handle (or prevent them!) in the future.

Most recently, we had a hard drive failure in one of our clinic PACS servers. Truth be told, we have had numerous hard drive failures. However, in all other instances, the business was not impacted at all. We planned for this, and as such have RAID in all of our servers. Every other time, we simply responded to pre-programmed alerts (those are key), and the system recovered to a happy state silently in the background, without affecting staff in the slightest.

This most recent hard drive failure happened to be accompanied by a malfunction of a RAID controller, so we were in fact down. Staff at that location could not process patients very well at all. How did this play out? Staff called us, complaining of a seemingly minor issue. We spent time chasing that issue, which eventually led us to the actual problem of the drive/controller failure. We dispatched someone to the site, and they switched all the equipment manually over to a backup PACS server. Thanks to Intelerads architecture, this redundancy is very straight forward. This process took time, in the meanwhile, staff were using some documented (and some creative off-the-cuff) processes to keep workflow moving, albeit at a slower pace. Simultaneously to the switchover, staff also used a spare drive to replace the dead drive, allowing our PACS vendor to get the server itself back in service.

What can this experience teach us? Well, for one, there is no fool proof plan. We thought we had a dead hard drive covered, but had not anticipated what we faced. Secondly, we were only informed because staff noticed issues… the server itself was still running (not properly mind you), so our alerts did not fire off. Actually, one alert did trigger, but it was incorrectly configured as a low priority alert, that we hadn’t noticed in time. We relied on a manual switch over process. While portions of it could be (and now are) documented and programmed so staff could switch to the backup server with just a button press, there still is equipment that does not have that ability, and does require a technical resource to re-program. That said, technology is available in the way of load balancers which could have (possibly) greatly assisted the fail over.

While we were in a failed over state, waiting for the server to become operational again, workflow was fairly normal, but it was a tad slower than normal. If we did not have a spare part on hand, or something more drastic had failed on the server, we would have had to rely on hardware support to kick in and provide replacement equipment, which could have taken days, and still more time for Intelerad to configure the replacement server. This leads to the question of should we have had a spare, configured, server ready to go? And that appropriately enough takes me back to the question… how much is enough?

Bookmark to:
         

We use ServiceDeskPlus as our helpdesk and inventory system. When we were researching, it stood out in that it had a good price point, an active community behind it, and most importantly, had both an online demo and a downloadable evaluation.
Since our purchase, the product, and pricing, has changed. We have unlimited nodes for the inventory, but new customers have to pay by the node.

Overall, we like it. Being web based, we can touch it from any PC in the company, or even remotely. However, the coding of the site is such that it does not work fully with iPhones or Windows Mobile browsers (Opera), so some options are not available.

I think our two biggest issues with it are not really its fault. Initial setup and organization (being new to running helpdesk software) kept us from realizing its potential, and maybe still hinders us slightly from time to time. The other issue is just being thorough and always keeping it updated.

The way the inventory side works is not very straight forward… handling PCs and chasis essentially as two different items, and not in a convienent manner. Also, it does a central sweep to detect hardware/software, which does not seem as reliable and acurate as using remote agents.

Another goofy issue is that the idea of using it to track tasks is inheritantly difficult, as the task is not automatically linked to the issue it deals with.

The last big issue that comes to mind, and it would be the item I urge you to evaluate closely before purchase, is its limited searching. To SDP, a task is different from a note, which is different from a request… basically each element is its own entity. And only a few of these are at all searchable, and even then, only individually… you may need to repeat the search multiple times to cover all the bases. It also does not handle boolean searching… so unless you know exactly what you want, you won’t find it. This, in a helpdesk product meant to aid finding information, is a huge issue.

What we quite like about it is the reporting side. We actually use it for time keeping, and it works fairly well. We are always on the lookout for new software, as this product is not perfect, and its ongoing subscription means each year we are financially motivated to seek out alternatives.

If you have specific questions, let me know in the comments and I’ll try to provide more detail.

Bookmark to:
         

Interestingly enough, sometimes when things run too good, it is a bad thing. That reminds me of a situation I had many years ago, as an entry tech. I was tasked with changing a simple network setting in a company’s main server, however with NT4, it required a restart. This was acceptable, a quick 5 minute outage was planned. The trouble is, once the system rebooted, there was not networking at all. Being new, it took me a good (andhighly tense) 10-15 minutes to determine my config change was not at fault…that the network card simply had died. It turns out, this was the first reboot of the server in 3 or 4 years… and the NIC apparently was ready to fail upon the next power cycle. A new NIC was acquired, and a 5 minute “coffee break” turned into an hour outage.

A similar situation just presented itself on our RIS box. It had ran for a good 6 months without a reboot… humming away. ESX, as good as it is, still cannot manage disk space superbly on the fly, so we had to take the server offline to do some storage adjustments. Upon starting up, it acted very strangely… not showing the Disk Manager to the user, amongst other things (which is rather frightening immediately after doing storage maintanence). Other symptoms were that it was slow to boot, and the Server service (and dependancies Computer Browser and Distributed File System) would fail to start.

It turned out to be a very simple issue… we were able to determine that the Print Spooler service was causing issues with those other services, and that we could no longer view printers from the control panel (even with the Spooler service started and restarted). About 6 months ago, a tech had changed the 10 or so printers to keep printed documents. Well this server handles our label printing, so in those 6 months, over 300,000 print jobs were handled. All of these files hanging around inside the SPOOL folder were the root cause of the issue. Cleaning them up returned everything to normal.

Lucky for us, the system was able to run fine, it was simply the reboot that highlighted the growing problem. I know that people like to brag that their servers run for years rock solid without a reboot, and it is true that rebooting stresses the electrical components of a computer, but I am a firm believer in running scheduled reboots periodically (and monitoring the results). I would rather a problem surface during a maintenance window rather than during peak hours, when staff is away on vacation, as a result of a unexpected reboot or similar situation.

Bookmark to:
         

On the surface, this PACS deployment seems like it should be a walk in the park. We develop our own RIS, so the integration between RIS/PACS should be easier than some other organizations. However “should be” is not always “will be”. Part of our problem is man power and prior commitments. We are way behind schedule on a deployment for our new booking application, Purkinje. This means we are still investing energy into it, when we should be focused squarely on PACS.

Also, there just seems to be no shortage of other mini projects or problems needing resolution, and each of these cuts into our precious time. The idea of hiring a new person has been tossed around recently, but it is no magic fix. First of all, as we are well aware now, hiring takes time. Secondly, even if we find a good prospect, getting them up to speed on what PACS/RIS are takes more time. Since the entire company is so built around around those two systems, even simple things like printer problems often are connected to some specialized software.

Then there are all the little things that need attending to. We have to order all the equipment (modalities, PCs, software, scanners, etc), but to do that properly, we have to know where and how the equipment will be used. As others out there can surely attest to, space is often at a premium, so the size and placement of PC towers or printers or scanners is an issue. Buying something that just will not fit where it needs to go is an issue, as it takes time to renovate or acquire a replacement piece. So all of these little things need to be accounted for in advance.

Similarly, since this PACS deployment will affect almost every employee in some way, communicating with the various departments to either get their feedback on a change, or at a minimum, simply giving them advance warning of a change is important. Unfortunately, something that seems like a good idea to us, may seem the complete opposite to the end user. Identifying issues like these in advance, and finding some sort of resolution, is key. If you give the users reasons to dislike the system from the start, they may have little reason to change their outlook later. Conversely, if users see that you have their best interests at heart, they will often be more patient and understanding as the system experiences rough patches during implementation.

Sometimes, timelines necessitate compromise. For instance, we want to go “paperless”. However, there are different degrees of paperless. The most obvious is simply to have no paper work created or used during a patient visit. And while perhaps ideal, this is difficult to achieve. To us, at least initially, paperless will mean that the Radiologist has no paper. However, to simplify deployment to technologists, they will still do worksheets as they do now, and then scan them into PACS.

How these worksheets get scanned in is, again, not a simple issue. Scanning them into PACS versus RIS has various pros and cons. What hardware and software used also plays a role into the bigger picture. We have chosen to go with PACSGear’s PACSScan product, with hardware still a question mark at this point. This product makes it fairly easy for the user, simply select the patient from a worklist and scan. And it allows us to transition to an electronic form down the road. The thing is, it was also the first product we evaluated. Well, second actually, at one point we strongly considered QuickScan Pro to get data into the RIS. However, that involves integration work and time, so the simplicity of PACSScan won out.

The catch is, we already are aware of a severe limitation to PACSScan… data entered on sheets to be scanned in simply can not be harvested and fed to the RIS. Other solutions may have allowed for this, but we had to move fast, and in doing so, may have either limited ourselves, or (more likely) cost ourselves some money. Once everything gets deployed and smoothed out, we very likely will re-evaluate the scanning process, and to get more efficiency in our workflow, may transition away from PACSScan towards another product that does allow for OCR and integration with a RIS. Which could also be a temporary solution, depending on how quick (or slow!) we are to embrace voice recognition for our Rads.

We also have networks to deploy, hardware to build, and a manuals(!!) to write. There is lots to get done, and not much time to do it in. It will certainly be an interesting time. Personally, I am eager to see how it unfolds, and ultimately where shortcuts are taken, and the repercussions of such actions.

My words of advice to those of you planning a PACS deployment of your own is to give yourself double the time you think it will take, and to plan, plan, plan. We are essentially trying to do this in about four months. Maybe, when all is said and done, I will post how four months is plenty of time. For the moment though, four months seems very short (at least for our staff in our situation).

Bookmark to: