How many data centers do you really need?

Reliability and availability are areas of significant concern for any SaaS company, because a lot of people rely on the ability of a single, relatively small group to ensure that their software is working. A significant aspect of availability is disaster recovery (DR): What do you do if, in spite of your best efforts to eliminate single points of failure, something goes terribly, terribly wrong in your data center? How quickly can you get your customers back up and running?

For years, we've operated under the assumption that excellent DR requires two fully redundant data centers, with "flip of a switch" fail-over from one data center to the other. Toward that end, our DCO team has worked with our vendors to design a state-of-the-art replication system that ensures that both data centers are always in sync, and always ready to fill in for each other as needed.

Here's the problem: The cost of that kind of infrastructure is gigantic (essentially double the cost of a single data center). A small or mid-sized SaaS company may spend $1M per year to keep a single data center running. To maintain two data centers with enough capacity that one can take on the load of the other at a moments notice essentially increases the cost to $2M, or maybe a bit less. That can have a devastating on the company's Cost of Revenue (which, for a SaaS company, includes both DCO costs and Support/Help Desk expenses).

We learned recently that NetSuite, one of the leading on-demand ERP vendors (and one that we are intimately familiar with) has operated in a single data center since its inception. In mid-2007, they announced (as a prelude to their IPO in December 2007) that they would be expanding to a second data center in 2008. I can't find any evidence that they have done that and, in fact, the "Cautionary Note" in the press release announcing their record Q2 2008 results warns that, "unexpected disruptions of service at the Company's data center may occur".

Some key questions have to be answered to decide whether a second data center is necessary:

• How redundant is the infrastructure in your data center? Have you eliminated, to the extent possible, any single points of failure? (That's not cheap--but it's a lot cheaper than a second data center.)
• How much do you trust the facility that you are in? Have they demonstrated the ability to absorb power failure without impacting you? Do they have strong bandwidth peering relationships?
• Do you have a comprehensive backup and validation process? Are you certain that your backups are good (i.e., do you restore and test each backup right after it's made)? Do you move your backups off-site frequently?
• In the case of a truly catastrophic event (major earthquake, fire, flood), how long can your customers wait to get back online? Would your customer base revolt if they were offline for 24-48 hours, or could that be absorbed? Do you have a documented and tested DR plan to recover within that time frame?

As we address these questions at AdvancedMD, it will be helpful to talk to other SaaS companies and compare DR strategies. Who isn't doing enough? Who is going overboard? How do industry regulations (like HIPAA) impact DR requirements?

"Free" PMS and EMR software

Every few months, I get an e-mail or see a blog post about a new Open Source, "free" EMR or PMS. Usually, the e-mail is entitled something like, "We'd better keep our eyes on this..."

(By the way, I intentionally capitalize Open Source, because, as far as I'm concerned, it's a brand name. Or, if it's not a brand name, it's a movement. Or a religion. Or a political party. Whatever it is, it's a proper noun, and consequently requires capitalization.

For evidence of this, look at the Wikipedia entry for Open Source. As of this writing, it has the disclaimer, "The neutrality of this article is disputed" at the top. Of course it is! It's difficult to write about your religion and stay neutral.)

But I digress. I don't want to rehash the worn-out debate between Open Source and commercial software. That's about as interesting as Microsoft vs. Apple, Microsoft vs. Oracle, and Microsoft vs. Mozilla. The fact is, if Open Source works for your project, then you should use Open Source. If a commercial package meets your needs, use it.

Having spent most of the past 15 years of my career in the medical practice management software arena, I believe there are two broad categories of medical practices:

Open Source		Commercial Software
Sophisticated internal IT staff		Everyone else
Doctors = Technologists
High threshold of pain
Interest in or need for heavy customization
Equal/greater interest in tech innovation vs. treating patients

I honestly can't see where Open Source projects compete with commercial software like AdvancedMD. First of all, AdvancedMD and other SaaS-based software (is there any other kind?) are essentially free. The only up-front cost is for training and implementation. With commercial software, those services are available directly from the vendor, or from their authorized VAR. With Open Source, you'll have to find someone to provide those services, or you're working with a consultant. Either way, they're not free.

The real cost of software comes in the ongoing maintenance and support. With AdvancedMD, you pay a reasonable, fixed monthly cost. The software is maintained by the same team of IT professionals that maintain our other 3,000 customers. Help Desk support is provided by the same team of Support professionals that serve those same 3,000 customers.

If you choose Open Source, someone has to install and maintain the software, and provide end-user support. The software was free...these services are decidedly not.

That's not to say that there is no place for Open Source. There certainly is, and I'm certain that there are dozens if not hundreds of success stories.

The point is that it is very, very easy to determine whether you are a candidate for Open Source PMS and EMR software or not: If you fall in the left side of the above table, you should consider it. If you're one of "everyone else", well, welcome to AdvancedMD. (Sorry, that really was a shameless plug.)

IE8 compatibility looking good...so far

I downloaded the first beta of Microsoft Internet Explorer a couple of months ago to check out the new features and, while I was at it, find out how well AdvancedMD runs in it. (I blogged earlier about some of my fears about IE8.)

This is an important issue for us, because, historically, new versions of IE and (especially) Windows have caused us a few problems.

Most of the hurdles have come in the form of security enhancements. For example, we sometimes pop up dialogs outside the viewable area of the screen to test for the existence of controls, measure window title bars and borders, etc. Well, a couple of years ago (IE6 SP2), Microsoft decided to stop allowing windows to be opened outside the visible area of the screen (by default). Not a big deal--the only impact was that screens that used to be invisible suddenly started popping up on our users' screens. (Well, they would have if we hadn't identified and addressed the issue before SP2 was released to our customers.) But it was annoying.

Quite often, we see changes in behavior early in the beta process, and the behavior continues through the second beta, or even the release candidate, but the previous behavior returns in the final release. That happened in IE7, where the beta releases blocked pop-ups in the Trusted Sites zone (and we were scrambling to figure out what to do about it), but then the final release restored the previous behavior. (Pop-ups should not be blocked in the Trusted Sites zone by default.)

So, given this history, I was more than a little concerned when, after downloading and installing IE8 Beta 1, I couldn't run AdvancedMD. At all. I couldn't even log in. In fact, the user name, password, and office key text boxes didn't appear, just a scary-looking security alert of some kind.

Well, a few days ago I installed IE8 Beta 2, certain that I'd see the same behavior, and we would have to start exploring the problem and devising solutions.

To my astonishment, though, AdvancedMD runs perfectly under IE8 Beta 2, at least in all of the areas that I tested. Our QA team will continue to validate my findings, but at the moment, I'm very encouraged.

Perhaps the best explanation for this is that Microsoft invested extremely heavily in IE6 SP2 and IE7 to restrict javascript behavior to avoid the wide array of exploits that had become prevalent (and that seriously, perhaps permanently, damaged Microsoft's security credibility). That work is largely done, so they've begun to focus more on the feature set again. And javascript has been so severely restricted at this point that few further changes are required.

Whatever the reason, it looks like the upcoming release of IE8 will be uneventful for AdvancedMD and our users...unless they introduce something in the final release.