Thursday, November 20, 2008

SIAA OnDemand Conference 2008

I just attended the SIAA OnDemand Conference with our CEO and CFO. I actually didn't have high expectations, because the session summaries seemed to imply that the conference was aimed at client/server vendors that were considering migrating to SaaS. As it turned out, there was a broad variety of content, and we all walked away feeling like we had gained a lot.

I learned a new respect for Marc Benioff, Chairman and CEO of Salesforce.com; Josh James, CEO of Omniture; and Zach Nelson, CEO of NetSuite. These guys are true visionaries, and they "get" OnDemand software, even though their views conflict at times.

I was particularly impressed by Zach Nelson. His keynote was actually a "point/counterpoint" session with Anthony Lye, SVP of CRM at Oracle. Lye was obviously a smart guy, but he made a fair number of statements (minimizing the value of multi-tenancy, for example) that really didn't ring true.

The main impression that Lye left me with was how deeply entrenched senior executives at Oracle are in the anti-Microsoft mindset, and how arrogant they are about the superiority of Larry Ellison's vision of the enterprise.

Lye lamented the disastrous effects of Microsoft's "monopoly" on the OS, and said that it stifled innovation, and that it "didn't do any good for anyone." I would imagine that Microsoft stakeholders would dispute that assertion. So would hundreds of thousands of software developers that were suddenly able to administer their own database servers when SQL Server 7.0 was released. Or the millions of home users who could suddenly use a word processor to write letters when Word for Windows was released. And on and on and on.

Anyway, aside from the fact that Lye forgot that he wasn't at an Oracle Microsoft-bashing conference, he clearly didn't "get" SaaS and its value in the small and mid-market. (His primary argument against the value of multi-tenancy was that "none of the customers that we asked wanted to be in a multi-tenant environment." Well, duh! If you had a choice to fly in a private jet or a commercial airliner (for the same price), which would you choose? The point is that multi-tenancy lowers the cost of delivery and, hence, the overall cost of the solution.

But the conference was excellent, and very worthwhile. It was definitely targeted at C-level executives (the vast majority of attendees were CEOs or CFOs), so if you don't fall into that category I'd probably recommend the OpSource SaaS Conference next March (it was recommended to me by a Microsoft rep).

Speaking of Microsoft, they were conspicuously absent from this conference. I'm not sure why, unless they felt it was just too small. Or perhaps they find more value in talking to IT directors and engineers.

Wednesday, October 29, 2008

Microsoft PDC 2008

This summary is not available. Please click here to view the post.

Thursday, September 18, 2008

How many data centers do you really need?

Reliability and availability are areas of significant concern for any SaaS company, because a lot of people rely on the ability of a single, relatively small group to ensure that their software is working. A significant aspect of availability is disaster recovery (DR): What do you do if, in spite of your best efforts to eliminate single points of failure, something goes terribly, terribly wrong in your data center? How quickly can you get your customers back up and running?

For years, we've operated under the assumption that excellent DR requires two fully redundant data centers, with "flip of a switch" fail-over from one data center to the other. Toward that end, our DCO team has worked with our vendors to design a state-of-the-art replication system that ensures that both data centers are always in sync, and always ready to fill in for each other as needed.

Here's the problem: The cost of that kind of infrastructure is gigantic (essentially double the cost of a single data center). A small or mid-sized SaaS company may spend $1M per year to keep a single data center running. To maintain two data centers with enough capacity that one can take on the load of the other at a moments notice essentially increases the cost to $2M, or maybe a bit less. That can have a devastating on the company's Cost of Revenue (which, for a SaaS company, includes both DCO costs and Support/Help Desk expenses).

We learned recently that NetSuite, one of the leading on-demand ERP vendors (and one that we are intimately familiar with) has operated in a single data center since its inception. In mid-2007, they announced (as a prelude to their IPO in December 2007) that they would be expanding to a second data center in 2008. I can't find any evidence that they have done that and, in fact, the "Cautionary Note" in the press release announcing their record Q2 2008 results warns that, "unexpected disruptions of service at the Company's data center may occur".

Some key questions have to be answered to decide whether a second data center is necessary:
  • How redundant is the infrastructure in your data center? Have you eliminated, to the extent possible, any single points of failure? (That's not cheap--but it's a lot cheaper than a second data center.)
  • How much do you trust the facility that you are in? Have they demonstrated the ability to absorb power failure without impacting you? Do they have strong bandwidth peering relationships?
  • Do you have a comprehensive backup and validation process? Are you certain that your backups are good (i.e., do you restore and test each backup right after it's made)? Do you move your backups off-site frequently?
  • In the case of a truly catastrophic event (major earthquake, fire, flood), how long can your customers wait to get back online? Would your customer base revolt if they were offline for 24-48 hours, or could that be absorbed? Do you have a documented and tested DR plan to recover within that time frame?
As we address these questions at AdvancedMD, it will be helpful to talk to other SaaS companies and compare DR strategies. Who isn't doing enough? Who is going overboard? How do industry regulations (like HIPAA) impact DR requirements?

Monday, September 8, 2008

"Free" PMS and EMR software

Every few months, I get an e-mail or see a blog post about a new Open Source, "free" EMR or PMS. Usually, the e-mail is entitled something like, "We'd better keep our eyes on this..."

(By the way, I intentionally capitalize Open Source, because, as far as I'm concerned, it's a brand name. Or, if it's not a brand name, it's a movement. Or a religion. Or a political party. Whatever it is, it's a proper noun, and consequently requires capitalization.

For evidence of this, look at the Wikipedia entry for Open Source. As of this writing, it has the disclaimer, "The neutrality of this article is disputed" at the top. Of course it is! It's difficult to write about your religion and stay neutral.)

But I digress. I don't want to rehash the worn-out debate between Open Source and commercial software. That's about as interesting as Microsoft vs. Apple, Microsoft vs. Oracle, and Microsoft vs. Mozilla. The fact is, if Open Source works for your project, then you should use Open Source. If a commercial package meets your needs, use it.

Having spent most of the past 15 years of my career in the medical practice management software arena, I believe there are two broad categories of medical practices:

Open SourceCommercial Software
Sophisticated internal IT staffEveryone else
Doctors = Technologists
High threshold of pain
Interest in or need for heavy customization
Equal/greater interest in tech innovation vs. treating patients

I honestly can't see where Open Source projects compete with commercial software like AdvancedMD. First of all, AdvancedMD and other SaaS-based software (is there any other kind?) are essentially free. The only up-front cost is for training and implementation. With commercial software, those services are available directly from the vendor, or from their authorized VAR. With Open Source, you'll have to find someone to provide those services, or you're working with a consultant. Either way, they're not free.

The real cost of software comes in the ongoing maintenance and support. With AdvancedMD, you pay a reasonable, fixed monthly cost. The software is maintained by the same team of IT professionals that maintain our other 3,000 customers. Help Desk support is provided by the same team of Support professionals that serve those same 3,000 customers.

If you choose Open Source, someone has to install and maintain the software, and provide end-user support. The software was free...these services are decidedly not.

That's not to say that there is no place for Open Source. There certainly is, and I'm certain that there are dozens if not hundreds of success stories.

The point is that it is very, very easy to determine whether you are a candidate for Open Source PMS and EMR software or not: If you fall in the left side of the above table, you should consider it. If you're one of "everyone else", well, welcome to AdvancedMD. (Sorry, that really was a shameless plug.)

Saturday, September 6, 2008

IE8 compatibility looking good...so far

I downloaded the first beta of Microsoft Internet Explorer a couple of months ago to check out the new features and, while I was at it, find out how well AdvancedMD runs in it. (I blogged earlier about some of my fears about IE8.)

This is an important issue for us, because, historically, new versions of IE and (especially) Windows have caused us a few problems.

Most of the hurdles have come in the form of security enhancements. For example, we sometimes pop up dialogs outside the viewable area of the screen to test for the existence of controls, measure window title bars and borders, etc. Well, a couple of years ago (IE6 SP2), Microsoft decided to stop allowing windows to be opened outside the visible area of the screen (by default). Not a big deal--the only impact was that screens that used to be invisible suddenly started popping up on our users' screens. (Well, they would have if we hadn't identified and addressed the issue before SP2 was released to our customers.) But it was annoying.

Quite often, we see changes in behavior early in the beta process, and the behavior continues through the second beta, or even the release candidate, but the previous behavior returns in the final release. That happened in IE7, where the beta releases blocked pop-ups in the Trusted Sites zone (and we were scrambling to figure out what to do about it), but then the final release restored the previous behavior. (Pop-ups should not be blocked in the Trusted Sites zone by default.)

So, given this history, I was more than a little concerned when, after downloading and installing IE8 Beta 1, I couldn't run AdvancedMD. At all. I couldn't even log in. In fact, the user name, password, and office key text boxes didn't appear, just a scary-looking security alert of some kind.

Well, a few days ago I installed IE8 Beta 2, certain that I'd see the same behavior, and we would have to start exploring the problem and devising solutions.

To my astonishment, though, AdvancedMD runs perfectly under IE8 Beta 2, at least in all of the areas that I tested. Our QA team will continue to validate my findings, but at the moment, I'm very encouraged.

Perhaps the best explanation for this is that Microsoft invested extremely heavily in IE6 SP2 and IE7 to restrict javascript behavior to avoid the wide array of exploits that had become prevalent (and that seriously, perhaps permanently, damaged Microsoft's security credibility). That work is largely done, so they've begun to focus more on the feature set again. And javascript has been so severely restricted at this point that few further changes are required.

Whatever the reason, it looks like the upcoming release of IE8 will be uneventful for AdvancedMD and our users...unless they introduce something in the final release.

Wednesday, August 27, 2008

Microsoft HUG Tech Forum 2008 - Day 2

The Transformative Power of Technology, at the Intersection of Physicians and Patients (Opening Keynote)
Steve Shihadeh, VP, Health Solutions Group, Microsoft Corp.
Clyde Wesp, MD, FAAP, CMO, St. Joseph Health System

Today's Steve, like yesterday's Steve, compared HealthVault to PayPay, in the sense that "no one goes to the PayPal website...they go to eBay or Barnes and Noble..." I'm not sure that's quite accurate--PayPal was, in the early days, just a way to send money to send money to friends and family, and perhaps eBay sellers, instead of mailing a check. Once that business model was successful, online merchants jumped on the bandwagon.

In the case of HealthVault, Microsoft is hoping that vendors, hospitals, and insurance carriers jump on the bandwagon in a leap of faith that other vendors, hospitals and insurance carriers will follow suit.

As I've mentioned before, Google is taking more of a consumer-focused approach, much closer to PayPal's approach than HealthVault's.

Also, PayPal generates its revenue from transation fees. Microsoft claims that theirs is an advertising-based revenue model.

Another difference is that, for the most part, PayPal isn't a central storage facility for...well, anything. They do offer the ability to store cash in a money market account and then use that cash for future transactions, but many PayPal users (myself included) never store any cash in their PayPal accounts. Transactions pull money directly from the user's checking account or credit card to the merchant's account.

But I see what the Steves are saying. There is a need for a central clearinghouse for healthcare information, and it makes perfect sense that the patient should have control over the distribution of his/her information...even if (or especially if) they aren't fully aware of the control that they have (like they would if they were carrying around a PHR on a smart card or flash drive).

Click here for full presentation

One participant suggested that HealthVault would be useful to scientists as a source of participants in clinical trials and in other ways. Steve's response was that Microsoft made a hard choice early on that HealthVault would NOT be used in that way, because they worried that going down that road might inhibit adoption of the platform. Consequently, their position is very firm that they are going to err on the side of security and privacy, potentially at the expense of the life sciences side.

Great Healthcare via Unified Communications: A Developer's Perspective (Developer Track)
Chris D. Mayo, Technology Evangelist for the Unified Communications Platform, Microsoft Corp.


It should be news to no one that Microsoft would like to have all of their products fully entrenched in every company in the world. That's obviously not going to happen, since different IT shops have different platform expertise. But for companies that are already on the Microsoft bandwagon, it makes sense to at least explore Microsoft's offerings and, in the case of communications, consider Microsoft's Unified Communications platform as a replacement for traditional PBXs (or even newer IP-based telephony systems).

From a software engineering perspective, Microsoft has provided a wealth of tools (Web services, drag-and-drop WFC controls, etc.) to make it really easy to build UC features into apps.

Cross Enterprise Document Sharing (XDS.b) Reference Implementation (Developer Track)
Mark Simmons, Health Consultant, SIMPL
Wagner Silveira, Microsoft Technical Architect, Microsoft Corp.

This was an interesting discussion of the SIMPL/Microsoft approach to IHE...not highly relevant to AdvancedMD right now, because we're not pursuing IHE implementation at this point. Another opportunity to catch up on e-mails.

Better User Experience in Clinical Applications (Developer Track)
Anand Gaddum, Director, Healthcare Practice, iLink Systems
William Hughes, Director, Product Innovation, GE Heatlhcare Enterprise IT Solutions Division

Anand and William demo'd a prototype application built on Silverlight. It was very impressive, and only took a team of 5 full- and part-time engineers about 2 1/2 months to build...but it clearly wasn't ready for prime-time.

Still, it reinforces my believe that Silverlight will be a powerful development platform as it matures, and as the runtime becomes more ubiquitous.

In particular, as we try to encourage doctors to adopt new technology, Silverlight provides UI "eye candy" that might help pique their interest.

Building Safer Healthcare Applications in WPF and Silverlight with the Microsoft Health Common User Interface (Developer Track)
Andrew Kirby, Director in Microsoft Services UK, Microsoft Corp.

MSCUI was launched a year ago at the last Tech Forum in Redmond. This session was an overview of MSCUI and a view into how it's been adopted over the past year.

This is a really interesting project, because it isn't a sellable product, or even an initiative that directly results in the sale of Microsoft technology. It's a collection of design guidelines that are totally platform agnostic, as well as a collection of free controls that illustrate the principles of the guidelines using Microsoft technology.

Obviously, one of Microsoft's objectives is to provide enough free controls and other shortcuts using their technology that healthcare organizations will be encouranged to use Microsoft tools to create solutions, so it isn't a purely altruistic endeavor, but it's pretty cool, anyway.

There is a ton of information available at http://www.mscui.net/, so I won't dive in any deeper here.

Surface Technology in Healthcare (Closing Keynote)
Randy Fusco, CTO & Strategist, Microsoft U.S. Healthcare Provider Solutions, Microsoft Corp.

This was a pretty exciting session. Surface started out as an informal collaboration between a couple of departments at Microsoft back in 2001. A few years ago, it was officially funded as a research project by Microsoft, and now it is an actual product.

It's difficult to describe how cool this stuff is without actually seeing it. If you're interested, check out the Microsoft Surface website.

Tuesday, August 26, 2008

Microsoft HUG Tech Forum 2008 - Day 1

I'm at the Microsoft HUG (not "MS-HUG" any more, according to Michael Clifford's opening remarks) Tech Forum in Redmond today and tomorrow. My primary objective is to gain more exposure to HealthVault so that I can bring that knowledge back to my colleagues, but I'll be posting highlights from all of the sessions that I attend.

Opening Keynote
Steve Aylward, General Manager, Health & Life Sciences, Microsoft


I skate to where the puck is going to be, not where it's been.
-
Wayne Gretzky

Steve started his presentation with Microsoft's Health Future Vision video (direct link).
The video has a lot of cool, visionary ideas, but I was especially impressed by the sweet TV in the patient's living room. Being a Tucson native, I liked her low-care landscaping (in contrast to the huge lawns in the Salt Lake City area). I was also happy to see that shopping carts wouldn't change.

In all seriousness, the video demonstrates a vision that Bill Gates has talked about for a long time: The convergence of media, computing, telecom, etc. My favorite part is when the patient uses her cell phone as a TV remote. Having 4 kids, that would change my life. (I'm not sure why the patient's "Digital Wallet" wasn't part of her phone or some other device.)

Microsoft's growth in healthcare market:

2000
63 employees
< $100 million in sales in healthcare market 2008
> 700 dedicated employee
> $1.2 billion in US
> $2 billion in sales world-wide

In 2008, Microsoft invested $8B in R&D. Steve couldn't say how much of that was invested in healthcare, but indicated that it was a large percentage.

Aside from core initiatives like Office, Windows, BizTalk, etc., healthcare is among Microsoft's largest areas of investments.

Other notes:
  • Amalga HIS is for emerging markets, where there isn't already a lot of infrastructure in place.
  • Amalga Unified Intelligence System: I don't know what this is.
  • HealthRamp: "Internet health platform that enables third-parties to create valuable health-related services." That description (pulled from one of Steve's slides) confirms my earlier assessment that Microsoft is taking a more vendor-based approach than Google.
Unified Communications in Health Care (Developer Track)
Dr. Albert R. R. Kooiman
Bill Crounse, MD

Unification (via Office 2007) of:
- E-mail and unified messaging
- Instant messaging
- Enterprise telephony
- Conferencing

Dr. Kooiman opened with a demo of Outlook: He selected an e-mail message, then clicked a button to open an IM window with the sender. With another click, he called the sender on the phone. Presumably, another click or two would allow him to conference in another participant to the conversation.

The "Call to Action" for this session was to encourage developers to use Office 2007 as a platform to integrate unified communications into their applications.

Dr. Crouse introduced this video on unified communications on his HealthBlog, and showed it during this presentation. To be honest, I find UC to be more applicable to hospitals than ambulatory clinics, but perhaps there is a more relevant application for larger clinics.

Solving the Difficult Problems of Healthcare and Life Sciences with the Latest Generation of Microsoft Technologies (Developer Track)
Tim Huckaby, CEO, InterKnowlogy

Tim was invited to present because his company has done a lot of cool projects for healthcare organizations using Microsoft technologies. Also, Microsoft is IngerKnowlogy's largest client.

Tim demo'd a WPF application built on top of SharePoint Web services for Scripps research. Visually, the data is represented as complex, hyperlinked, rotatable 3D images, but they're stored as large text files in SharePoint. The graphical performance of WPF is impressive, and the rendering code was written by one developer in one week (although InterKnowlogy had invested heavily in WPF prior to embarking on this project).

Why hasn't WPF been widely adopted?
- It is huge: Larger than ASP and WinForms combined.
- It's a thick client, and thin clients are the current rage.

WPF is built on top of DirectX, and it makes sophisticated graphics much easier to deal with.

Silverlight only contains about 18-20% of the classes available in the full WPF framework.

InterKnowlogy has some some cool Silverlight demo apps.

Microsoft HealthVault Architecture Overview (Developer Track)
Sean Nolan, Chief Architect, Microsoft Corp.

Our bet: Patients as the hub of of communication.
- Sean Nolan's HealthVault presentation

Reasons to Integrate with HealthVault

  • Private and Secure Storage
  • Authenticate Users, Manage People Relationships
  • Share Securely, Authorize Data Access
  • Application Interoperability
  • Device Connectivity
  • Application and Device Discovery
  • Developer Assistance

How does Microsoft monetize HealthRamp? (There are no fees or charges to use the platform.) They see healthcare being like travel - opportunities for advertising in the future.

HealthVault can be used in "Native" or "Copy" model:

  • Native: Healthvault replaces traditional database and authentication mechanisms
  • Copy: HealthVault is an external repository, you can pull data out of it or push data into it (import/export, merge, sync)

HealthVault supports both Windows Live ID and Open ID.

Interesting: All access to HV goes through the XML API (Basic XML over -HTTP interface, POST a request and receive a response)...NOT SOAP! (Although SOAP and WSDL will probably come in the future.)

This is interesting to me because we took the same approach with our core (XMLRPC) API 9 years ago. (In our case, SOAP wasn't fully baked, but we continue to believe that our lightweight version of SOAP is better suited for our application.)

Security, authentication, and access control are very important in HealthVault.

HL7 Based Data Warehousing: Fast Build to Data Use (IT Pro Track)
Eric V. Washburn, CTO, Athena Advanced Technologies/HVHS

This was a great opportunity to catch up on e-mail. (Eric did a good job, but his audience was pretty small--few organizations need to do what he did.)

Xbox: Gaming machine or healthcare platform?

If you thought that the Microsoft Xbox 360 was a gaming machine, like I did, you're a couple of years behind the curve.

I've attended 2 sessions at the Microsoft HUG Tech Forum so far, and both of them included slides that included the Xbox 360 in the Microsoft technology stack that is targeted at healthcare. The Xbox? I was intrigued, so I googled "Xbox healthcare". Here's what I found.

Almost 2 1/2 years ago, Dr. Crounse at Microsoft wrote about how the Xbox can be used to "help cure healthcare woes," based on research being done by Dr. Harold Goldberg. A month earlier, an AP article covered the work that Dr. Goldberg had been doing with the Nintendo Game Cube to encourage young people with diabetes to communicate with their doctors and manage their conditions.

It seems unlikely that Nintendo is interested in making heavy investments into healthcare, Wii Fit notwithstanding. So Microsoft, already generating about $2B in revenue from healthcare, is in an excellent position to leverage the Xbox in the healthcare market, particularly among young patients.

At CES in Las Vegas this year, Cerner demo'd their Cerner Care Console (announced in this press release), which is essentially just a customized Xbox 360. It allows hospital patients to play games, of course, but also provides education, a feedback mechanism, maps, information about the patient's doctors and other care providers, etc.

In my role as Chief Architect at AdvancedMD, it may become important that I get one of these in my office...for research purposes. That realization alone justifies my trip to the Tech Forum!

Tuesday, August 19, 2008

ICD-10 and ANSI X12 Version 5010: October 2011?

The Department of Health and Human Services (HHS) is working towards switching from the current set of healthcare EDI transaction standards (ANSI X12 Version 4010) to Version 5010 on October 1, 2011. That switch is partially driven by a need (or strong desire) to switch from ICD-9 to ICD-10 code sets, which are already in use in most of the world.

As promised, the recent announcement of this switch gives providers, insurance carriers, clearinghouses, and vendors 2 years to retool our systems to accommodate the new standards. For most of us, that won't be a huge problem (although we've got a lot of work ahead of us). We'll have to wait and see how well the state payers do...

In any case, it will be interesting to see how these changes ultimately affect AdvancedMD and our customers. Certainly the cost of migration will be significant (more for our customers than for us, I would imagine), but maybe we can hope for some benefits.

For years we've complained about the oxymoronic nature of the term "healthcare standards". The ANSI 837 (claim) and 835 (ERA) transactions were supposed to standardize communication between healthcare providers and insurance companies. In some ways they were successful, but, apparently, some carriers (mostly Medicaids) decided that the standards were inadequate. They ended up finding hundreds of creative ways to violate HIPAA and stick weird data all over the place.

Maybe if we're really, really lucky, Version 5010 will accommodate these carriers' requirements without the need for a bunch of "companion guides". Or maybe the switch to Version 5010 will encourage payers with antiquated adjudication systems to upgrade them so that they can obey the rules.

Hey, it doesn't hurt to hope, right?

Whatever the payers do, we'll soon be embarking on projects to accommodate these new standards so that we can be way ahead of the curve if and when they go into effect in 2011.

Monday, August 18, 2008

I "Wordled" my blog...

I love creative visual representations of boring data, so I was intrigued by Wordle. From the Wordle website:
Wordle is a toy for generating “word clouds” from text that you provide. The clouds give greater prominence to words that appear more frequently in the source text.
Cool! Here's the word cloud for this blog:

I've spent a few minutes staring at the word cloud, trying to figure out why some words (e.g. "Alistair") appear more frequently than others (like "data").

Oh, great--I just gave "Alistair" more importance! If I keep talking about "Alistair", eventually he's going to dominate the whole word cloud! What if Alistair himself gets annoyed by my repeated use of his name, and posts a comment, signing it "Alistair"?

Okay, that's enough...I'm off to post this entry, and see what the new word cloud looks like...

Tuesday, August 12, 2008

Living in the hotbed of Agile Software Development

I was reminded today how fortunate I am to be here in Utah, where Agile was born and continues to thrive.

(The Agile Manifesto was conceived at The Lodge at Snowbird, not half an hour from my house. One of the authors, Alistair Cockburn, lives in Salt Lake City, as do other powerful Agile advocates, like Jeff Patton.)

[Humorous likenesses of Alistair and Jeff used without permission, and, most likely, at risk of serious retribution.]

I was asked by Jonathan Rayback, another local Agile mover and shaker, to talk today in our local Agile in Management meeting about our first experience with the Walking Skeleton concept (conceived and named by Alistair over 10 years ago). I didn't have much time to work on it (due to vacations and a pressing work schedule), but I threw together some thoughts and slides and drove to the meeting, feeling a little nervous, but confident that I had some useful experience to share.

What I didn't know was that Alistair himself would be there. Right! I'm supposed to explain the Walking Skeleton concept to the guy who created it!

Within my little world, this was akin to a physicist explaining the Theory of Relativity to Albert Einstein, or a political scientist explaining the Monroe Doctrine to...well, James Monroe!

Of course, Alistair was extremely gracious and helped me feel at ease from the very beginning. More importantly, he critiqued my presentation very gently, yet honestly. I walked away feeling that I could make the same presentation to a different group in full confidence that I was representing his concept accurately and effectively.

What a great opportunity, though, to rub shoulders with some of the "great ones" of the Agile movement. I've never been in a room with Alistair when I didn't learn something.

One of the interesting things about Alistair is his skepticism of Scrum. Not that he's opposed to it. In fact, whenever Mickey, Sheridan or I tell him how great Scrum is working for us, he expresses his enthusiasm (and shock?) over our successes. It just seems that he's seen a lot of failures in half-hearted or misguided attempts to implement Scrum.

In any case, I think you'd be hard-pressed to find a better place to do Agile than Utah. Now if we could just get Ken Schwaber and Jeff Sutherland to move to Salt Lake City...

Wednesday, August 6, 2008

Microsoft releases SQL Server 2008

Our long wait is over:

http://www.microsoft.com/Presspass/press/2008/aug08/08-06SQLServer2008PR.mspx

We've been getting along just fine with SQL2K5, but some of the features in this new release are really going to make our lives easier.

One great example is what they call "Transparent Data Encryption". At first blush, this doesn't seem like a particularly useful feature for us, because our database servers are stashed behind so many layers of physical and network security that it would be extremely difficult for anyone malicious to get at them. And hard drives with PHI never, ever leave the data center.

But in spite of all of that security, we have had a couple of requests from our more security-sensitive clients to encrypt their data at rest. To date, the only reasonable approach to that problem is to encrypt the disk volume that the database is on, inevitably impacting performance in a big way.

This new feature of SQL Server will allow us to physically encrypt their data with minimal fuss, and, presumably, without a noticable performance hit.

I'll post in the future about other key SQL2K8 features that we plan to incorporate into our systems.

Friday, August 1, 2008

Microsoft SQL Server: World's most secure RDBMS

Hey, that's a pretty controversial headline for a mild-mannered blog like this one! But I think it's supported by evidence.

In November of 2006, Enterprise Strategy Group released an "Information Security Brief" that makes the following conclusions, based on Common Vulnerabilities and Exposures (CVE) data from the National Vulnerability Database:
  • Oracle’s results over the past two years show that much work has to be done to bring the vulnerabilities into line with competing database products from IBM, Microsoft, MySQL and Sybase.
  • ESG considers Microsoft to be years ahead of Oracle and MySQL in producing secure and reliable database products.
  • Microsoft’s results are almost too good to believe, and thus serve as a model for other database vendors.
During that same month, David Litchfield did a separate study based on a broader set of data and reported:
  • It is immediately apparent...that Microsoft SQL Server has a stronger security posture than the Oracle RDBMS.
  • The conclusion is clear – if security robustness and a high degree of assurance are concerns when looking to purchase database server software – given these results one should not be looking at Oracle as a serious contender.
Even before those reports were compiled, Cesar Cerrudo of Argeniss put together this presentation in which he provides lists of Oracle security flaws and SQL Server security strengths and asks, in apparent exasperation, "Why do you think [Oracle] is Secure?" And, "Why do you think [Microsoft] is not Secure?"

It's interesting that Microsoft has several pages on its website where you can find articles like these (albeit not these specific ones) touting the security of SQL Server, while I couldn't find anything on Oracles site (and I looked) citing independent analyses that provide evidence that Oracle is more secure than SQL Server...and Oracle has had a couple of years to respond.

OK...so, all of this does NOT mean that SQL Server is better than Oracle. Recent releases of Oracle 11g and related products offer all kinds of features that SQL Server doesn't. I'm certain that there are literally thousands of companies currently using Oracle that would be foolish to consider a switch to SQL Server. There may even be hundreds of companies that should seriously consider switching from SQL Server to Oracle, for any number of valid reasons.

But, c'mon, think about it: Microsoft SQL Server more secure than Oracle??? Are we talking about the same Microsoft and Oracle? Unbreakable Oracle?

And don't forget that ESG found SQL Server to be more secure than MySQL...and MySQL doesn't have a target painted on its back. Hackers exploiting flaws in MySQL would be like animal rights activists vandalizing PETA headquarters. Well, not exactly, but it makes an entertaining simile.

In any case, SQL Server has worked great for us. We're looking forward to using some of the features in SQL Server 2008. I'll try to describe how we end up taking advantage of those features in future posts.

Wednesday, July 30, 2008

2nd Annual AdvancedMD User's Conference - Day 1

We're just about to wrap up the first full day of our annual User's Conference, and it's been great. I'm pretty sure we have the smartest users in the world. I think I get a little bit smarter just from being around them.

From my perspective, one of the most valuable things about these user's conferences is the ability to separate the propaganda that we hear from the media and some of our vendors and partners from the real problems that real people face on a daily basis--the problems that AdvancedMD is designed to solve.

Sheridan heard a great story last night that illustrates one of the main advantages of the SaaS model. (I'll be vague in the details to protect the identities of the parties involved. With my poor memory, I'm sure I'll introduce some inaccuracies, as well.)

Basically, the office manager of a pretty good-sized practice found out that one of her adult children living in a different state had a serious disease--serious enough that she felt that she should move nearby to help out. In spite of that interstate move, she is continuing to function effectively as the office manager of that practice. That's made possible by the "anytime, anywhere" nature of AdvancedMD. All she needs is a Windows computer with Internet Explorer.

This is an example of technology making a real, positive impact on a family's lives, not to mention the success of the medical practice.

Our User's Conferences also give some of our talented employees a chance to shine, and for fellow employees to see them in action.

I attended a session this morning that was put on by a couple of our Support techs, about reading EDI reports. They did an absolutely phenomenal job. Without attending that session, I wouldn't have been aware of the wealth of knowledge that they carry. I plan to tap them as technical resources in the future.

I'm looking forward to learning a lot more from our users and other AdvancedMD employees during the next day and a half of the conference.

Monday, July 28, 2008

Cloud computing: Evolution of the species?

So I'm registering for the Gartner Web Innovation Summit, and I'm seeing two major, broad themes: Web 2.0 (which I understand) and Cloud Computing (which I don't).

From the look of things, everybody and their dog is "computing in the cloud"...except me. How did I get so far behind?

As I've studied up on the technology, I've been reminded of this classic (but obviously modified) drawing:




(I'm probably infringing someone's copyright, but I can't find out who the owner is...let me know if you find out.)

Here's how I see the (possible) evolution of application hosting (assuming that you manage your own servers):
  1. Self-hosting
    Here, we get some Internet connectivity within our office space, grab some IP addresses, build some web servers, and announce ourselves to the world. When it's time to deploy or update the application, we sit on a stool in front of a keyboard and monitor and use an A/B switch to choose which server we want to work on.
  2. Co-location (Phase 1)
    In this stage, we move our servers to a hosted environment, where the data center provides power and Internet connectivity. Aside from that, it all works the same: Every time a server needs to be refreshed, or the application needs to be deployed, someone drives to the data center, sits on a stool in front of a keyboard and monitor, etc.
  3. Co-location (RDP)
    Now we're getting high tech: Instead of running to the data center all the time, we use RDP (if we're running Windows, or some equivalent technology if we're running something else) to hit our servers remotely. But we still own all of the equipment, and we have to get it in there somehow (perhaps using a 3rd-party IT resource).
  4. Cloud computing
    Here's where it gets kind of...um...cloudy for me. Those nuts at Google, Amazon, and Microsoft (among others) are using virtualization on steroids to offer...well, heck, I don't know what! But instead of selling servers or hosting space, they're offering these funky units of computing time like "EC2 Compute Units". Wha...?

Is this where application hosting is going? The idea is pretty amazing: These huge players build megaservers that can host dozens or hundreds of virtual servers, and then they provision them out like web sites or SQL Server instances.

If your business suddenly doubles, you don't hop on a plane to install a bunch of blades in your data center. Instead, you fill out a form online asking for a bunch of EC2 Compute Units, then deploy your app. If traffic falls, you just decommission a few servers, and your invoice reflects the drop in usage.

Anyway, it's cool stuff. To be honest, I don't see AdvancedMD moving into this space any time soon, but it may be a fun spectator sport.

Saturday, July 26, 2008

Who "owns" patient data?

One of the first hurdles that we had to clear as a SaaS company was the objection of providers who were accustomed to keeping their data within their offices. We called it "storing data in the broom closet", since in many offices the actual physical location of their server was no more secure than a utility closet. While the data was certainly NOT secure, it was accessible, or at least perceived as such.



In fact, there are many problems with that arrangement, among them:
  • Dismal disaster recovery (DR) options. I once heard that 60% of magnetic tape backups are unusable, although that number may be high. More conservative estimates vary between 10% and 50%. Within medical offices, where there generally is no dedicated IT staff, I would lean toward the higher estimates.

  • Lack of security. It would be easy for a disgruntled employee to unplug a few cables and carry the whole server out the door, or just bring in a laptop and wirelessly copy data from the server.

  • Risk of physical damage. Hundreds of medical offices were devastated by Hurricane Katrina, for example, and permanently lost huge amounts of irreplaceable patient information.
So, over time, our customers have accepted the fact that they are better off letting AdvancedMD keep their data for them, as long as we provide methods (standard data exports, ODBC access, etc.) for them to get access to it.

Now doctors are being faced with more dispersal of patient information, in the form of electronic prescribing systems, RHIOs, HIEs, PHRs, etc. I'm not a doctor (obviously), but I have to believe that this new sharing of data is a little disconcerting for some.

In the Summer 2008 issue of JHIM, Richard D. Lang, EdD, writes in "Blurring the Lines: Who Owns the Medical Data Home?" (HIMSS membership required) about the very objection that we used to face, but applied in a slightly different way.

Dr. Lang says:
Healthcare IT is evolving from a physician-centric model to
a collection of disparate patient-centric applications where
all constituents contribute to a mélange of databases that
serve people and processes in many different ways. By electronically
diffusing the traditional patient record, this new model blurs
the long-established medical data home.
As a true SaaS company, AdvancedMD assumes ownership of the provider's physical data, even though conceptually the data remains the property of the provider. Similarly, if a practice contracts with a billing service, the lines of ownership become further blurred, as the billing service assumes ownership of whatever data it needs to effectively bill for the practice's services. In that scenario, the billing service contracts with AdvancedMD, not the providers, so we are an additional level removed from the actual healthcare practitioner.

For eight years, we've proven that this data model can work, and, in fact, it works extremely well. It almost seems natural that, over time, patient information will continue to be further dispersed among interested parties that play a role in the patient's care.

As a patient, I kind of like the idea of spreading my information around, as long as it's secure. The next time I need to see a PCP and can't even remember who I saw last, wouldn't it be great if my new doctor could access my medical history without me having to remember it?

I have to believe that AdvancedMD's customers are better prepared for this "brave new world" than those who are still stuck in their broom closets.

Wednesday, July 23, 2008

Events and Travel Plans for 2008 - Redux

Due to scheduling conflicts, I've had to make some changes to my travel plans for the year. Here's what I'm planning now:

HIMSS MS-HUG Tech Forum Redmond, WA, Aug. 26-27
I always learn something from these events in Redmond, beyond Microsoft's sales pitches. Most of the value comes from hearing about the challenges that other HIT people are facing, and how they are dealing with them.

Gartner Web Innovation Summit Los Angeles, CA, Sept. 15-17
I missed the Gartner Enterprise Architecture Summit in June, but this event seems more relevant to our business, anyway.

Microsoft Business Intelligence Conference Seattle, WA, Oct. 6-8
I'm really looking forward to this one. I don't know how they're going to make me more intelligent, but it's worth a shot!

Seriously, we've built our reporting strategy around Reporting Services, so I'd like to take a few days to find out what's going on in that area.

Health 2.0 Conference San Francisco, CA, Oct. 21-23
As the name would suggest, Health 2.0 (like "Web 2.0") is mostly about consumer-focused applications, but I'm interested in exploring how Web 2.0 technologies like mashups can be used within our offerings.

Construx Software Executive Summit Seattle, WA, Oct. 27-29
This conference comes highly-recommended by a business acquaintance who until recently worked for a local EMR company.

Alternate: Microsoft PDC Los Angeles, CA, Oct. 27-30
I really hate to miss this one, because Microsoft has a lot going on (HealthVault, Live Mesh, SQL 2008, etc.), and there's no better place to catch up, IMO. But it conflicts with the Construx summit.

Monday, July 21, 2008

Microsoft's "SaaS Maturity Level"



Microsoft has an interesting way of looking at SaaS-iness. They have a four-level "SaaS maturity model" that looks at an application's scalability, configurability, and "multi-tenant-efficiency" to determine what SaaS level it fits into.

This model has been around for a couple of years, but I was introduced to it in a "CTO P2P Forum" put on by the Utah Technology Council and featuring Nate Bowler, CTO of @Task.

It was obvious from Nate's presentation that they're doing some really great things there, but it was also clear that they haven't reached the 4th level of SaaS maturity, according to Microsoft's model. But that isn't necessarily a bad thing, as Microsoft clearly states.

But what's really interesting to me about Microsoft's SaaS maturity model is that it only seems to address the architectural aspects of SaaS...which, while fundamental to an overall SaaS philosophy, don't in and of themselves make a SaaS offering.

We learned years ago (before the "SaaS" acronym was coined) that there is a lot more to SaaS than software. If your Sales team isn't SaaS-oriented, or your client support teams aren't SaaS-oriented, then you're not going to have much success...because your salespeople won't sell, and your customers won't get adequate support.

For example, how do you price SaaS? If your nearest client-server competitor is selling their system for $50,000 with a 20% annual maintenance contract, do you sell your system for $49,000 with a 15% annual maintenance contract? Of course not. If you're truly SaaS, then you may (and probably will) charge an upfront fee for implementation and training, but your pricing should be a monthly (or annual) subscription model, of perhaps $1,999/month.

So, that brings in steady recurring revenue month after month as long as you can keep the customer. But then how do you pay the salesperson? I don't know the answer, but certainly it will be different from your competitor's compensation plan.

Anyway, Nate's presentation was excellent, but it was clear that their un-SaaSiness extended beyond their architecture into their deployment model and other business decisions. Again, that's not a problem, as it seems to work well for them.

I came to the conclusion that AdvancedMD is just about as SaaSy as you can get. That's pretty cool, considering that our archicture and business model predate the industry's fascination with SaaS by several years.

Saturday, July 19, 2008

A face for radio...and a voice for blogging

You know, you really put yourself out there when you decide to blog. It scared me for a while. Still scares me, in fact. You never know when you're going to write something that sounds brilliant as you're writing it, but actually makes you look like an idiot.

Well, it could be worse. Actually, it is worse. In the absence of our uncommonly articulate VP of Operations, Ken Meyers, I was asked to participate in a Podcast put together by our PR company. The intention was to trumpet our recent AdvancedMD Version 5.6 release. The results speak for themselves.

Don't worry, I'm going to keep my day job.

Friday, July 18, 2008

Development checklists: It's all in the details

Healthcare is extremely complicated, and, consequently, the applications that we've built over the past 8 years are extremely complicated. There are so many buttons and dials and knobs that it's really easy to forget important, but arguably obscure, steps as we enhance and maintain the software.

It doesn't take too long to recognize the need to consolidate those obscure requirements into a single document (or, in our case, three separate documents), so that they can easily be reviewed during the development lifecycle.

We decided to create those documents as "checklists" that a developer can pass through at various points to ensure that, for example, user-level security and auditing requirements have been taken into account. Each checklist is limited to a single page, and applies to one of the following stages of development:

Analysis and Design: Since we're using SCRUM, most of the analysis and design for new features takes place during and shortly after a sprint planning meeting. It is generally at this point that the team is putting together screenshots and/or prototypes and thinking about what the database schema might look like (assuming that they are doing UI-first development).

Coding: These items help ensure that the developer is thinking about indexes and unique constraints on new database tables, commenting code, "leaving a trail of goodness", etc.

Finalization: At this point, the team is integration testing the feature, dotting the i's and crossing the t's, and communicating changes to IT and other teams.

Checklists can help developers to remember the details and avoid obscure bugs...as long as they remember to look at the checklists. We encourage our developers to keep the checklists mounted prominently in their workspaces so that they can be reminded of them, at least during code reviews and pair programming.

Tuesday, July 1, 2008

Why we're a Microsoft shop...and it's okay if you're not

[I've had this post sitting in my Drafts list for a while now, because I'm completely comfortable with the tone...but I'll let the reader be the judge.]

So, I was reading a really informative post on The Health Care Blog about how Kaiser Permanente is working with Microsoft to allow their employees (and, presumably at some later point, all 2 million + Kaiser members) to copy their health records into HealthVault. Seems like a pretty interesting venture, right?

Well, not everyone is happy about it. In particular, the fourth comment to the post lists all of the reasons why no one in their right mind would EVER embark on a project that doesn't feature Java/Open Source exclusively. The most compelling arguments: Microsoft isn't Open Source and Java isn't C#. That's what passes for logic in some circles.

I really didn't feel like such biased, ABM ("Anything But Microsoft") remarks should go unanswered so I answered them in the most respectful but pointed way that I could. Feel free to read my response if you wish.

My point is this: AdvancedMD was the first financially viable browser-based Medical Practice Management System on the market, and we use Microsoft tools and technologies almost exclusively in our development. So I know that it works.

Why do we use Microsoft technology? Well, there are several reasons:

  1. The principal architects of the system (Sheridan Richey and myself) had more experience using Windows, Visual Basic, Microsoft C++, and Internet Explorer.


  2. When we originally architected the system over 8 years ago, Open Source was still an adolescent, rebelling against authority and trying to figure out who it was.


  3. We are Microsoft Gold Certified Partners. What that means is that, for a few hundred bucks and a few hours of effort each year, we get hundreds of thousands of dollars worth of development tools, OS and Office licenses, etc. In a small startup (which we were until just recently), that's a financial windfall.


  4. Over the years, we've learned how to squeeze outstanding performance from MS SQL Server and COM+. And we can still squeeze out a lot more, dedicating the right resources to the effort.


  5. Visual Studio is a one-stop solution for all of our code, from front to back, and its IDE paradigm extends to SQL Server tools.


  6. Microsoft has been active in the healthcare community for a long time, in partnership with HIMSS (via MS-HUG).

So, that's why we use Microsoft technology, and I couldn't be happier. The proof is in the proverbial pudding.

What? You run on Linux? Using MySQL, Python and a collection of open source libraries?

Great...more power to you! Unless, of course the company you're working for is perpetually losing money, in which case...I'm sure you can find a job somewhere else.

Regarding the indisputable statement that "C# is not Java", I would (and did, in my response) make the equally useful comment that "Chocolate isn't butterscotch, and Ford isn't GMC".

Everyone has their preferences, and I'm certain that there are examples of successes that you can hitch your wagon to, no matter which platform and tools you choose.

But here's the important thing: Thanks to emerging (and largely well-established) standards, we can all get along now!

Back in the 90's, Don Box used to say that "COM is love". Well...maybe it was within its own world, but COM certainly wasn't particularly fond of CORBA, or vice versa.

It would be more correct to say that "SOAP is love", or "Web services are love". The comment I mentioned earlier claims that "developers must purchase Microsoft Visual Studio.NET to code HealthVault applications." Of course, that's absurd, because HealthVault is exposed as a Web service, so, by definition (assuming that it was implemented correctly), it can be accessed by any code that can consume a Web service.

(It is true that, so far, the only comprehensive SDK and DDK that Microsoft has provided for HealthVault is written in C#, but they do provide useful sample code for Java and Ruby, and they are working on providing more non-MS platform support.)

By the way, we interact on a daily basis with dozens of partners in many different lines of business and who use all kinds of different platforms and languages, and we get along swimmingly.

I guess that's just a really long way of saying, "I'm okay, you're okay. [Insert smiley here.]"

Monday, June 30, 2008

NAHIT "Defining Key Health Information Technology Terms"

NAHIT recently released a document called (get this):

The National Alliance for Health Information Technology Report to the Office of the National Coordinator for Health Information Technology on Defining Key Health Information Technology Terms

Basically, it has some interesting definitions for some common healthcare terminology. The location of the original document (along with the rest of the NAHIT site) appears to be down at the moment, but John Mertz at NeoTools has conveniently listed the terms for us, so I'll repeat them here:
  • Electronic Medical Record: An electronic record of health-related information on an individual that can be created, gathered, managed, and consulted by authorized clinicians and staff within one healthcare organization.
  • Electronic Health Record: An electronic record of health-related information on an individual that conforms to nationally recognized standards and that can be created, managed, and consulted by authorized clinicians and staff across more than one healthcare organization.
  • Personal Health Record: An electronic record of health-related information on an individual that conforms to nationally recognized interoperability standards and that can be drawn from multiple sources while being managed, shared, and controlled by the individual.
  • Health Information Exchange: The electronic movement of health-related information among organizations according to nationally recognized standards.
  • Health Information Organization: An organization that oversees and governs the exchange of health-related information among organizations according to nationally recognized standards.
  • Regional Health Information Organization: A health information organization that brings together health care stakeholders within a defined geographic area and governs health information exchange among them for the purpose of improving health and care in that community.
I don't know whether there is industry-wide agreement on these definitions, but they're an interesting start for the uninitiated.

Friday, June 27, 2008

Microsoft HealthVault and Google Health

Long-time rivals Microsoft and Google have found something (relatively) new to bicker about: Internet-based personal health records (PHR).

Microsoft HealthVault and Google Health aren't the first PHRs on the block. Existing players include AllOne Mobile, Revolution Health, and dozens of others (see myPHR.com for a lengthy, but still incomplete, list).

In a way, HealthVault and Google Health aren't really PHRs at all, but rather platforms that PHRs can be built upon, or used to aggregate data. Microsoft, in particular, insisted at their recent HealthVault Summit that they don't intend to compete with Google Health, but rather to seek opportunities to integrate with it.

So, where are these guys going with this?

Microsoft HealthVault
Microsoft has had a great deal of success pushing their products through their partners, particularly ISVs. For example, we've benefited in the past from the fact that not only do we use Microsoft servers in our data center, but we also require the use of Microsoft Word for certain functions within AdvancedMD. Microsoft appreciates that, obviously, and helps us out with software licensing and co-marketing opportunities.

In the case of HealthVault, Microsoft hosts an annual HealthVault Solutions Conference where participants "hear directly from healthcare professionals, consumers, and Microsoft product managers to better understand the overall health landscape and product roadmap."

At the most recent conference, 40 vendors demonstrated products that they are building to interact with HealthVault.

Microsoft also announced that they would be awarding $4.5 million in grants to support organizations that are developing applications that are using the HealthVault platform.

Working through partners may help Microsoft overcome a lack of trust that the public has in the company to protect sensitive information, partially due to the highly-publicized security holes in Microsoft Passport.

Of particular interest is Kaiser Permanente's pilot program to provide health information to its nearly 160,000 employees using HealthVault. If the pilot is successful, it is likely that the program will extend to all of Kaiser's 2 million+ members.

Google Health
For its part, Google Health seems to be pursuing the consumer market more aggressively, which makes sense given its huge popularity and trust among everyone from 16 year-old script kiddies to 90 year-old grandmothers.

Even so, it seems reasonable to assume that Google will rely just as heavily on participation from partners (including Microsoft?) to achieve success, as a report from IDC suggests.

Blue Cross and Blue Shield of Massachusetts recently announced that they will be providing their members with a mechanism to import claims data into their Google Health accounts, via their consumer health portal. The integration should be completed before the end of the year.

For a very interesting inside look at Google Health, check out this post by Robert Wachter, a contributor to The Health Care Blog. Fellow contributor Matthew Holt posted this in-depth test drive, also very useful.

So, is this healthcare's 21st century version of the battle between Beta and VHS? (For our younger readers, consider the fight between HD-DVD and Blu-Ray.) Or can the two behemoths coexist?

It's hard to say. As a Microsoft Gold Partner, it is likely that we'll look most closely at HealthVault first. Ultimately, we (and other ISVs and health plans) will need to integrate with both. And they'll need to make nice and integrate with each other.

Thursday, June 26, 2008

Why SaaS? Agility

Ken Meyers, our VP of Operations, recently reported the following statistics:

  1. Industry news reports (http://www.modernhealthcare.com/, among others) identified a 4x or more increase in rejections around national claim flow.
  2. Emdeon is reporting 25% Medicare and Medicaid rejections persisting 1 week after the deadline.
  3. Mysis reported to their customers a 50% increase in call volume after the deadline and asked for patience.
  4. Regional medicare centers are experiencing major spikes in call volumes (http://www.wpsmedicare.com/, among others)
    ...
  5. AdvancedMD experienced a daily call average DECREASE of 4.4% the week after the deadline (including accounting for Memorial Day), and we do not have any information indicating any material increases in rejections from RelayHealth.
So, why do I post this on an architecture blog? It all comes down to the agility that is inherent (or should be, if you're doing it right) in the SaaS model.

Most of our competitors are either Neanderthals (defined here as locally-installed client-server systems) or dinosaurs (text-based systems running on, for example, DOS or AS400s). When CMS announced the dearly beloved NPI requirements (described in boring detail here), most of those guys didn't just have to rush to modify a bunch of code to meet the requirements. They also had to figure out how to get their updates deployed to all of their sites...thousands of sites in some cases. And that assumes that their customers were willing to pay the (sometimes exorbitant) fees for the upgrade.

Once they had the upgrades installed in (most?) of their customer sites, they had to wait for the feedback on why their upgrades aren't working. Then fix the updates, deploy them, and wait for feedback again...and so on.

For AdvancedMD users, the NPI requirement has been, I have to confess, not particularly easy. No major changes in healthcare come without a few skinned knees. But glitches were found and then corrected quickly and, for the most part, transparently, on a weekly basis. By the time May 23rd rolled around, we were ready to go. Our customers even have a handy link to the NPPES NPI Registry so that they can quickly find NPI numbers for their referring providers.

With healthcare regulatory changes being as frequent and confusing as they are, a SaaS technology and business model just makes sense. And it keeps our customers laughing all the way to the bank.*

*Not really...I would imagine that most have ACH and don't physically deposit their Medicare checks. But you get the point.

Friday, May 16, 2008

SQL2008: Solving the file system vs. database BLOB quandary

I found a recent post on The Data Platform Insider blog very interesting:
One of the most exciting new features in SQL Server 2008 is the ability to store files and BLOBs directly in the file system, while maintaining transactional consistency with a SQL Server 2008 database. SQL Server 2008’s new FILESTREAM attribute for VARBINARY data type solves the age old dilemma facing developers and IT Pros: Is it better to store files directly in a database or store them in the file system with path and filenames stored back in tables to maintain the relationship with the database?

We've been fighting with this for years, for all of the reasons cited in the blog posting.

It doesn't solve one big problem, though: Some of our customers have multiple gigabytes of images and documents each. Add that to half a gig or more of transactional data, and then multiply that by a few hundred customer databases, and you've got a real challenge storing and moving database backups around.

To paraphrase (and, apparently, misquote) Senator Everett Dirkson, "A terabyte here, a terabyte there, and pretty soon you're talking a lot of data."

Thursday, May 15, 2008

An oldie but a goodie: How the customer explained it...

I just walked by Steve Burke's desk (he's our Manager of Data Conversions and Imports, or something like that), and noticed a great cartoon that describes the challenges of communicating and faithfully executing customer requirements far better than I have in previous posts. It's been floating around for a few years, but it's pretty clever.

It addresses both the argument for UI-first development and the need for a bridge (in the form of Chief Architect, in our case) between Product Management and Engineering.

Unfortunately, I have searched but have been unable to find out who the author of the cartoon is, so I can't give credit for it. If anyone knows, please submit a comment and I'll add an acknowledgment.

(Click the thumbnail to see the full-sized image.)


Wednesday, May 14, 2008

The NPI debacle in layman's terms

[Disclaimer: You probably don't want to read this. It's dry and boring. I dozed off twice while writing it. It may not even be all that accurate. Plus, you can get the same information from this CMS FAQ.

But they say that the best way to learn is to teach. So, as I struggle to understand how we got into the mess that we're in with NPIs, perhaps the best thing I can do is to try to explain it here.

So, go ahead and read if you like...but don't say I didn't warn you...]

What is the NPI?

The NPI (National Provider Identifier) is a 10-digit number used to identify healthcare providers. (A "healthcare provider"can be an individual person, as in the case of a physician or nurse; or a group of individuals that submit claims to certain insurance carriers as a single business entity.)

The NPI was mandated by the Health Insurance Portability and Accountability Act of 1996 (HIPAA). (Standard unique identifiers are required for both healthcare providers and health plans, but the identifiers for health plans have not yet been implemented.)

What does the NPI replace?

Historically, different insurance carriers have used a variety of different numbers to identify providers. Medicare, for example, used to issue its own proprietary identifiers (PIN, UPIN, OSCAR, NSC). Many Medicaid payers and most commercial payers expected the provider's EIN (Employer Identification Number, also known as Federal Tax ID). Still others required the provider's Social Security Number.

To further complicate the issue, some payers may require multiple identifiers. Others may give providers a choice of enrolling under, say, their EIN or their SSN.

What problems is the NPI supposed to solve?

All of the healthcare providers and insurance carriers in the United States are part of one ecosystem, with many millions of paper and electronic transactions taking place between the various parties every day. It shouldn't be a surprise to anyone that multiple provider identifiers would cause confusion and inefficiency.

One example: Primary claims submitted to Medicare, after being adjudicated by Medicare, are automatically forwarded on to the secondary payer (if there is one). Medicare can use the PIN to identify the provider, but the provider's Medicare PIN means nothing to, say, Medicaid or Aetna. So, in order for the claim to be forwarded to and paid by the secondary payer, the provider must include the EIN...or SSN...or the secondary payer's propietary identifier...or whatever, on the claim.

The NPI only addresses these issues if all providers and carriers switch from whatever identifiers they used in the past to the NPI. Consequently, all HIPAA covered entities (providers, payers, and clearinghouses) will be required to switch.

Who issues NPIs to providers?

The Centers for Medicare and Medicaid Service (CMS) issues NPIs using the National Plan and Provider Enumeration System (NPPES). (NPPES can also be used to look up NPIs.)

Can a single physician or other provider have more than one NPI?

Allowing a single healthcare provider to have more than a single NPI would violate the HIPAA requirement that NPIs uniquely identify a single provider. But this is healthcare we're talking about, so I wouldn't be surprised if it happens.

So, once a provider has an NPI, how do payers find out what it is?

As part of CMS's planning for the NPI transition, they conceived the notion of a "crosswalk" (a commonly-used term in healthcare that has been overloaded for this purpose). Basically, payers are expected to accept both their legacy identifiers and the NPI for a period of time, during which they are supposed to "crosswalk" the identifiers and associate the NPI with the corresponding providers.

On May 23, 2008, this crosswalk period officially ends, and all payers are supposed to accept claims with only the NPI. Of course, again, this is healthcare, so some payers (and we don't know how many) will fail to meet that deadline, or their systems will be so whacked that they will continue to reject claims until they can get their software fixed.

Tuesday, May 13, 2008

No apologies: The reality of technical debt

I attended an Agile roundtable this evening, and one of the sponsors, Jonathan Rayback (an Agile thought leader in the Salt Lake City area) introduced me to the concept of "technical debt". It's been around for a long time (at least since 1992), so I'm late to the party, but the idea really resonates with me.

The term was apparently introduced by Ward Cunningham, and has been expanded upon and clarified by Steve McConnell, among others.

Here's the definition supplied by the venerable (but oft maligned) Wikipedia (hyperlinks removed):

Technical debt is a term coined by Ward Cunningham to describe a situation where the architecture of a large software system is designed and developed too hastily.
No one who has been developing software professionally for more than 5 minutes has been able to avoid technical debt.

Jonathan illustrated the idea with a whiteboard graph that looked something like this:

In this chart, the project was expected to be completed in about 20 days. About 13 days into the project, it became clear that an additional 4-5 days would be required to complete it in a high-quality way. However, a business decision was made to stick to the original schedule by working more hours, cutting corners, or making some other compromise.

The area between the red line (the business-mandated schedule) and the green line (the "ideal" schedule, for the sake of quality) represents the business debt incurred during the course of the product.

Like financial debt, technical debt must be repaid at some point. And, like financial debt, not only the original principal will need to be repaid (in the form of refactoring, bug fixes, etc.), but also accrued interest (customer complaints, support calls, etc.)

To advance the metaphor further, Jonathan pointed out that not all technical debt is bad.

Most of us who own homes owe a sizable financial debt in the form of a mortgage. Did I make a mistake by going into debt to own my home? Certainly not: I estimate that my family's housing expenses over the past 11 years have been far less than they would have been if we had been renting during that time, even if we had lived in a much smaller home. It would have been ridiculous to wait to buy a home until we had saved up enough money to pay for one.

In the early stages of developing our software and service offerings at AdvancedMD, we incurred huge technical debt. We've had good-natured debates about whether that was a mistake or not. On the one hand, our coding and deployment efficiency is lessened by shortcuts we've taken in the past. On the other hand, we were first to market with a web-native PMS (by years), and we remain light years ahead of our nearest competition.

We've also made great strides towards paying off that debt (and minimized the accumulation of new debt as much as possible). We've rearchitected major components of our application over the years, so that, as a whole, I'd put our code up against just about anyone's. Sure, it would have been great if we hadn't had to do that rework, but, again, in most cases the debt was justified.

Not all technical debt is created equal. Here's how Steve McConnell categorizes technical debt:

Non Debt
Feature backlog, deferred features, cut features, etc. Not all incomplete work is debt. These aren't debt, because they don't require interest payments.

Debt
I. Debt incurred unintentionally due to low quality work
II. Debt incurred intentionally
II.A. Short-term debt, usually incurred reactively, for tactical reasons
II.A.1. Individually identifiable shortcuts (like a car loan)
II.A.2. Numerous tiny shortcuts (like credit card debt)
II.B. Long-term debt, usually incurred proactively, for strategic reasons

Only debt in Category I should be a source of embarassment...and, yes, we have our share of that kind of debt, although far less than a few years ago.

I make no apologies for the other types of technical debt that we've accrued, because we've overcome the odds by proving both our technology model (which, in 1999, was utterly original) and our business model.

Monday, May 12, 2008

How is your performance measured and judged?

My transition from head of Engineering to Chief Architect has been marked by one epiphany after another. Here's the latest:

In a previous post, I laid out some of the differences between the Chief Architect and the head of Engineering. An obvious question is: Why can't one person do both? Isn't architecture a key component of Engineering?

The same question can be asked about the separation of Product Management from Engineering (and, by extension, the Chief Architect). Don't they have the same basic goals of building high-quality software?

I think most people can easily distinguish between Product Management and Engineering: Product Management decides what to build, and Engineering builds it. The distinction between Chief Architect and Engineering is less obvious...

...until you think about how the levels of performance of the three departments are judged.

The head of Engineering is judged by the quantity and quality of software that comes from his teams. (The quantity is primarily a product of the developers, while QA has primary responsibility for the quality.)

If software releases are consistently behind schedule, or the support burden following releases is consistently overwhelming, where does the buck stop? With the head of Engineering. (Sorry, Sheridan.)

On the other hand, if software releases consistently fail to resonate with customers, or resources are consistently applied towards projects that yield no revenue or other value, then you have a problem in Product Management. The head of Engineering has neither the authority to decide what gets built, nor accountability.

By the same token, when releases are technically successful (goals are met and quality is high), the Engineering team has every right to celebrate and take credit. And when customers rave over the latest release because the enhancements are both timely and well-designed, Product Management can take the kudos.

So, how will my performance be judged as Chief Architect?

One thing is certain: I am no longer judged by the quantity or quality of code that gets written. I can't be, or I would be so caught up in writing code and helping the Engineering staff keep up with their demands that I would be unable to do my job, which I described in an earlier post.

Nor can it be based on the reception (hot, cold, or lukewarm) of new enhancements, by customers or by AdvancedMD staff.

Instead, my performance will be judged in more nebulous terms:

How well do I communicate our current architecture, its strengths and weaknesses, and our company's technology road map to our CEO, board, and other executives?

How faithfully do the architectural and technology changes that I propose and endorse reflect and support the high-level strategies of the company, as defined by the management team?

How effectively do I bridge the language and culture gap between the Engineering teams and Product Management?

How well do our applications and subsystems scale? How easy is it for our DCO (Data Center Operations) staff to go from supporting about 10,000 providers today to 100,000 providers in just a few years?

How stable and robust are our interfacing and interoperability infrastructures?

How well do I communicate and tout our technical accomplishments to those outside of AdvancedMD?

The prospect of finally, after eight years, having time to devote to these and other issues is exhilarating, and at the same time daunting and just a little bit scary.

The accomplishments of the past have been rewarding. But it won't be long before AdvancedMD will be asking me, "So, what have you done for me lately?" Here's hoping I have a good answer!