Healthcare.gov: How Washington's IT project Leviathan failed us, and here's how we fix a broken system

Government IT projects have a tendency to fall on their rear ends more often than not. After the miserable debacle that was Healthcare.gov last October, I made the case for why the larger than life public face of Obamacare had zero chance of succeeding in original form. Fast forward six months, and after some contractor firings and a public about face consisting of a "tech surge", the website is finally working at nominal levels.

That's not to say no one didn't take the fall for the mess of this bungled IT project gone haywire. The former head honcho of the US HHS, Kathleen Sebelius, had no choice but to step down and take the hushed blame for the mess that unraveled under her command. Publicly, the story goes that she stepped down on her own will. Behind the scenes, I highly doubt this was the entire story.

But I'm not as interested with the political ramifications of this failed project as I am with understanding the underlying issues and more importantly, what we have learned and where we go from here.

To that effect, I received an invite from Colorado State University associate professor John Hoxmeier who asked if I could join his BUS 630 night course via online video feed, and couldn't pass up the opportunity. I was asked to discuss some of my insights about what really caused the Healthcare.gov mess and what lessons we should take away to turn course on future federal IT projects.

That session took place just about two weeks ago, and I thought it would only be appropriate to share some of the topics we discussed that night as a formal followup to my previous article on the Obamacare site.

If the Private Sector Can Do Massive IT Projects Right, Why Can't Government?

Representatives from one of the primary culprits in the Healthcare.gov mess, CGI, claimed last year that the website project was ultimately "challenging, unprecedented and complex." It was a nice, meandering way to admit that they clearly weren't cut out for the work, and rode on their heels of knowing the ropes of government IT contract bidding -- not their technical project expertise -- more of which I will drill into later.

To put things in perspective, our government has paid more than double what CGI was originally budgeted to receive, close to $196 million so far, for their part in piecing together Healthcare.gov. And while Obama went on the offensive to defend the "tech surge" going into fixing the project, the private sector came in to show just how easily Silicon Valley could have built a working Healthcare.gov from the get go for jump change compared to what veteran contracting firms are invoicing the American public for.

HHS has already publicly admitted they have dumped over $677 Million into the Obamacare site (and that figure was through Oct 2013, mind you) only to have signed up a purported 8 million people as of March 31, 2014. Even some simple math tells us that this means the site has come in at a cost of nearly $85 per signup. The likes of Amazon and Microsoft would have been lambasted in the private sector by now if their cloud ecosystems rode on such terrible metrics.

In contrast, take a peek at the simple yet effective Healthcare.gov clone website HealthSherpa.com. The site performs nearly everything that the front-facing part of Healthcare.gov does, including allowing people to easily window shop for plans by zip code, and dig into premium and deductible combinations depending on family size, income, PPO vs HMO, and other factors. It's arguably just as visually appealing as the government's website, just as simple to navigate, and downright easy to even sign up for a plan of your choice.

Meet the brains behind the Obamacare website alternative, HealthSherpa.com, that was built on a budget with far fewer zeros at the end. These coders from San Francisco built the Healthcare.gov-lite clone in just three days. Imagine if the US government tore down the draconian walls keeping Silicon Valley out of most public sector projects. Without deep change out of DC, public sector IT projects have very little incentive to succeed, let alone innovate. (Image Source:  TheBlaze)

Before the internet hounds pounce on me, yes, I will admit that the site does not handle the other half of what the Obamacare site entails: behind the scenes processing between numerous federal agencies and insurance companies. But this still doesn't render the site's usefulness and proof of concept any less worthy.

If you think that HealthSherpa took dozens of programmers, hundreds of man hours, and a large budget to construct, guess again. 3 dudes in their twenties from San Francisco built the site in just about three days on a hamstring budget of a couple hundred bucks. To say they could have single handedly saved the government hundreds of millions of dollars is probably not too far off track.

And HealthSherpa isn't the only example of the private sector leading the innovation charge on a considerably smaller budget. The Sochi 2014 Olympics may be a fading memory at this point, but they laid claim to a massive IT undertaking that went off without any noticeable glitches.

I'm talking about Microsoft's handling of the live video streaming for NBC two months ago, which I covered in a previous story. Windows Azure did all the heavy lifting for nearly all of the American footage that was streamed for the global event, and the scale of the effort puts the Obamacare website to shame in every comparison.

Take this food for thought:

• Healthcare.gov: The troubled website hit a peak concurrent user count of merely 125 thousand people at the height of its toughest workload, which was on the final day of healthcare enrollment period ending March 31. More importantly, at launch, the site was only built and ready to handle a load of 1100 concurrent users. The site itself has been in development for over three years already.
• Azure's handling of 2014 Sochi Olympics streaming: Over 100 million viewers tuned into the 2014 Winter Olympics video feeds, with 2.1 million alone watching the popular USA and Canada hockey game. In total over 10.8 million hours of video was streamed off Azure; 80 percent of that was live. In addition to America, Microsoft Azure was the video backbone for 24 other countries across the world. The solution was so well architected that zero failovers were needed during production and there were (to my knowledge) no public reports of any major glitches faced by end users. And it was reported that Microsoft only began architecting its solution for Azure's Sochi 2014 coverage around summertime 2013 -- give or take roughly a half year from start to finish.

Surprised? CGI wanted to take comfort in the notion of how unprecedented the Obamacare site was in terms of complexity. I'd say broadcasting the largest global sports event for two weeks straight without room for error is pretty darn complex, too. And Microsoft's far from a global media streaming specialist, to boot.

I'm not a developer at heart so I can't attest to the complexities of database design and integration at the kind of scale Healthcare.gov needs. Perhaps comparing the site to Azure's streaming of Sochi 2014 may be in some ways unfair. But drawing back to the amount of money and time that Healthcare.gov has had, not to mention the number of supposed technical minds at the helm of its development, and one must wonder how this could have possibly crumbled so miserably under even the lightest realistic situations.

It's no secret that the private sector can deliver better results in shorter timespans with less money. Yet there is more under the hood causing government IT to fail so miserably time and time again.

US Government IT Contract Bidding: Flawed In Every Way

Our Founding Fathers put forth one of the greatest democracies the world has ever seen. The checks and balances afforded by the institutions of the US government are unparalleled in any other country today. So why can't this same system tackle IT projects in as efficient of a manner as other proven facets of top-down legislation?

Simply put, government contracting in America was molded for a time when projects entailed static infrastructure work like bridges, railroads, roadways, highways -- not complex and fluid IT projects like the Obamacare site. Such last century needs were fairly straightforward in complexity; easy to plan in waterfall project methodology; and most importantly, once projects were finished, they didn't need nearly the kind of ongoing attention and inter-agency cooperation like IT ecosystems require.

Fast forward to today, and you can see how a bureaucracy carved out for a 20th century industrial mindset is abysmal at tackling 21st century IT projects. It also becomes a lot clearer at why pitiful companies like CGI continue to get re-awarded massive projects even when they have terrible track records in the drivers seat.

Government IT sweethearts like CGI and QSSI don't hold much weight in expertise or execution compared to the bevy of firms that make up the best of Silicon Valley today.

But they do have a competency which government loves to bask in: red-tape navigation.

To get a feel for just how misguided the federal government bidding system is, I had the chance to talk about how this process works with a to-be-unnamed client of mine in the construction industry. They provide general construction services for various US government property from military bases to agency branch offices.

Take a fairly simple construction project, like a new building development or expansion of an existing property. The private sector has various guidelines it has to follow to comply with safety and regulations, but it's fairly straightforward and doesn't require vast amounts of endless paperwork.

This notion doesn't exist when handling similar work for the US government. A bundled mess of never ending red tape must not only be complied with up front pre-project, but consistently during the entire project until it's finished (and sometimes, well after the project is done and signed off on). It's a skill that the average construction foreman or engineer cannot even handle; it requires a dedicated position on-staff at the organization I am referring to. Day after day, this person is solely responsible for responding to and submitting leagues of paperwork.

They have a sister company in town that also handles similar work, and they need two full-time people on staff to tackle this crud. It's nothing more than bureaucratic sludge going back and forth, drowning in complex codes and policies and triple checks that no private sector project would ever think of engaging in.

And we don't see skyscrapers, residential homes, or office space buildings falling down or crumbling in any appreciable numbers because of it. But the federal leviathan demands it, and won't work with contractors who aren't skilled in navigating this paperwork minefield with proficiency.

Most of this mountain of back and forth paperwork is nothing more than administrative drivel that confirms the same things many times over in different form and fashion on each go around. But that's the price you have to pay for winning government contracts.

And if you're wondering how CGI and QSSI, just to name a few, fell into the Healthcare.gov picture, it's because they are darn good at navigating all of this red tape. They don't have exceptional IT staffing expertise, project success, or innovation leadership to show for.

They just know how to game the bidding system that is so well rigged favoring patronage instead of results, Silicon Valley has little ability or willingness to provide a meaningful competitive alternative.

Do you think that tech heavyweights like Microsoft, Apple, or Google would have come anywhere close to botching the Obamacare site like the status quo did? Even the worst effort by any of Silicon Valley's best probably would have fared better than what the contractors hired by the government produced.

IT Contractors: Incentivized to Fail

You would think that a company awarded work for Healthcare.gov would have a winning reputation and track record to back up its bid. Guess again. One of the worst offenders that was fired from the project, CGI, has a trail of failure that has riddled its recent history with government IT work.

Here's just a sample of the projects uncovered by the press that CGI has largely fumbled in one way or another:

Canadian Firearms Information System (Project Result: Canned). The Canadian government scrapped this faltering project that was meant to serve as a national gun registry after numerous deadlines were missed by CGI, and the entire project itself was running severely over budget.

Hawaii Health Connector (Project Result: Delayed Launch; Numerous Technical Problems). This was Hawaii's state-run healthcare exchange built by CGI, and it has been fraught with technical issues and cost overruns since the project began. Not to mention, it launched a full two weeks after its deadline to go live.

Hawaii Dept of Taxation (Project Result: Overpayment for Work Never Completed). This massive overhaul led by CGI stunk of deliberate IT project oversight failure, with Hawaii paying CGI millions extra for work that was never even completed. A corporate consultant who was brought on by the Hawaii Dept of Taxation in 2008, James Bagnola, said of CGI's role in the Healthcare.gov disaster: "The morning I heard CGI was behind [Healthcare.gov], I said, my God, no wonder that thing doesn't work."

eHealth Ontario Diabetic Registry (Project Result: Canned). Another flop at the hands of CGI which was canned by the Canadian government in September 2012. The end result, if launched, would have provided a platform that was obsolete from the starting gate.

Vermont Health Connect (Project Result: Late Launch; Technical Problems). CGI concocted this state Exchange just like Hawaii's and numerous others around the nation. And just like the rest, there was a late launch, budget overruns, and numerous other promises that weren't delivered. Massive technical issues existed months after the site went live, and to make matters worse, hackers have broken into CGI's servers hosting the site on numerous occasions already. Very comforting news for Vermont residents.

Numerous other examples are available of how cruddy CGI has been in the federal contracting space, and I didn't even mention Massachusetts' severance of ties with CGI over its own failed state run healthcare exchange. Different customers, similar end results. Piles of technical problems confounded by lack of oversight and amateur attempts at proper security and stress testing where it was necessary.

But herein lies one of the secrets of being a powerhouse government IT contractor having the stature of a CGI: there's very little incentive to be on time with delivery or even to bid on projects honestly.

We saw it with plenty of other bungled federal IT disasters, including but not limited to the dead-end Virtual Case File system the FBI scrapped in exchange for its own home-grown Sentinel project. Contractors like CGI come in with too-good-to-be-true lowball bids on massive endeavors like Healthcare.gov, and then riddle the landscape with change orders that drive up cost.

One would think that the market would have punished CGI for its part in the abysmal Healthcare.gov launch last October. This was anything but the case. CGI actually rode a small bullish stock upswing just post-launch, but more glaring was its October 10 spike that continued rising steadily. Even as of May 2, their stock price remains nearly unchanged from its Oct 1 level during the Healthcare.gov crisis. With how federal IT contracting works today, it's clear that there's little incentive to do a great job -- failure is just as lucrative in this bizarre sector of IT. (Image Source: Washington Post)

There's no shocker, then, in finding out that CGI has already been paid well over $196 Million for its pitiful role in the Healthcare.gov launch. In contrast to the smidge over $93 Million contract CGI boasted about on its own website, you have to question how their part was bloated by more than double original estimates.

To make matters worse, agency leaders like the fired resigned Kathleen Sebelius of HHS, especially ones with little insight or background in large IT rollouts, have zero leverage in reigning in scope change requests or understanding start to finish oversight. As I argued in my class appearance for Colorado State University, leaders with not even a whiff of IT project management experience (like Sebelius) had no place being the end of the line on the largest technical undertaking in US government history.

And the worst slap in the face to the American public footing the bill for this chronic mess was the admission by the White House that Obama had no knowledge of the site's woes prior to launch. Ignorance, incompetence, or just plain left in the dark?

I'll tell you this: if my Presidential legacy was hinging solely around one accomplishment like healthcare reform, I wouldn't have taken a rain check from every chance at getting briefed on the status of the technical implementation supporting this push. It's unconscionable to believe that President Obama only learned of the mess waiting for the American public the same time the country did. But you can make your own conclusions on that end.

A Zoo Full of IT Contractors With No One In Charge

For lack of a better description of the Healthcare.gov development effort, the best description one can pin on the supposed A-Team enlisted for the project is something close to that of a zoo. This is a befitting description seeing as not even Sebelius and the HHS could provide us with an accurate top-down of what the dev team looked like and who was owning up for what.

A reporter from NPR, rather creatively, decided to take it upon themselves to help draw up what the team effort looked like on the Healthcare.gov project. This was all drawn up merely from contractor testimony in Congress. It comes in the form of nothing less than a simple hand made mockup:

(Image Source: NPR)

The drawing pits some of the main contractors involved and denotes what roles they played in this effort. CGI was in charge of the entire shopping experience on the site after an account was created. It was also responsible for front end actions and confirmation emails along with related data processing.

QSSI was the one at the helm for taking care of the data pipeline that interacted with numerous agencies such as HHS, IRS, Social Security Administration, etc. They also concocted the registration management tools used on the website.

Serco, another contractor with a smaller but rather laughable role, was in charge of processing applications for healthcare received in paper form. Mind you, their workers were expected to punch data into the same broken system that end users were struggling with for months.

So in essence, all you were doing if you couldn't use the website was offloading your frustrations onto a data entry clerk on the government contracting payrolls, who merely had the ability to skip the front door (the registration system that forced end users to sign up prior to "window shopping" which was initially targeted by some lawmakers as a way for HHS and the White House to hide the true cost of these new plans).

In the end, who was playing quarterback for all of this? Well, no one really, and that job fell fruitlessly into the lap of a sub agency of the HHS (Centers for Medicare and Medicaid Services to be exact). It was an invisible title in name only, as the agency had next to no experience in managing such a co-mingled rollout involving so many hands.

It's safe to assume that modern software project management concepts like Agile Dev and User Stories were not in the vocab of the leadership at CMS.

Even the White House's own CTO, Todd Park, claimed in congressional testimony late last year that he had no knowledge of the scale or type of testing that was going on in the Healthcare.gov project prior to launch. Even if he was truthful in stating he was not involved pre-launch, that doesn't necessarily excuse him from having no involvement.

Again, the President's entire second term hinges on healthcare reform and this website, and your own dedicated in-house CTO is out and about handling business as usual months before the troubled launch? It's just hard to believe in some respects.

Another big problem with the way the pieces of Healthcare.gov were built? Modern style software development processes, known as Agile, were leveraged for the "front end" user experience -- everything that users see when visiting and working with the website. Yet this wasn't the case for other aspects of the project.

As Pat Morrell eloquently pointed out, the back end of the website wasn't so fortunate and was developed using that 20th century industrial approach I talked about earlier; known in project management circles as Waterfall methodology. Healthcare.gov had no place using a last generation development approach to a modern software project as massive in scope and scale as this website is.

Where do you pin the blame for such a monstrous failure? Kathleen Sebelius may have taken the public fall, but behind the scenes, it's very clear that this was nothing more than a game of government hot potato. No one set of hands ever needed to take possession of responsibility. For it was always "someone else's problem."

Tearing Down the Federal IT Leviathan: Piece by Broken Piece

There's a lot of blame to go around for the sour taste that Healthcare.gov left in everyone's mouth. From contractors that weren't performing basic stress testing, to a lack of clear central leadership at the reigns, down to a messy landscape of co-mingled agile and waterfall project development structures that were followed.

If the only place we can go from here is up, where the heck do we start?

Let's begin by simplifying the federal IT project contracting regulations system, which is by all accounts convoluted, overly complex, and favors large ineffective organizations like CGI by barring fair competition from outsiders who aren't well suited in red-tape navigation. Is the goal of government to keep its belly fat full of bureaucrats who justify their own existence by burying the American people in regulation hell?

I'd argue that it's time we ditch this failing contracting system in favor of one that is open to change, competitive on the free market, and one that tears down the immense barrier to entry that well qualified outside organizations can't currently overcome.

How about new standards in establishing proper accountability for undertakings like Healthcare.gov? Does it make any reasonable sense that roughly 47 contractors were involved in some fashion with the flawed site and not one of them could correctly establish themselves as the overarching technical project lead? Even though that duty ended up informally in the lap of CMS, a non-technical federal agency should have never been tasked with this unreasonable expectation.

Stress testing was another big focus of congressional hearings after the bungled launch. Numerous contractors testified that such testing was seemingly non-existent prior to launch, as the YouTube video on my previous article about this website mess alluded to. Even minimal, basic tasks attempted on the site were falling flat on their face mere days before launch on October 1. That's inexcusable, and goes back to a flawed implementation plan and timeline.

It's already well known that roughly 68 percent of all IT projects are doomed for failure, and a large majority of this is due to improper requirements planning and scope analysis up front before any contractors are even hired. It's safe to say that the US government, and namely HHS, botched this aspect of the Healthcare.gov planning phase.

 

Think the Obamacare site is the only example of government IT projects gone awry? Guess again. Compared to some other miserable examples of public sector IT endeavors, Healthcare.gov pales in many ways. The US Air Force ditched a monstrous ERP system before it ever went live, and the UK had its own Obamacare-esque IT failure with a scrapped NHS medical records platform that has been billed as the "largest IT project failure ever seen". (Image Source: Curiousmatic.com)

I've also got to ask one more time: what was the rush to get this flawed website launched on Oct 1? Seeing that the White House already has a failing grade on meeting Obamacare deadlines (a full 44 of 83 have been missed so far), why couldn't the braintrust at the White House decide to delay the launch, or at the least, stagger the launch in multiple phases? If Healthcare.gov couldn't handle a successful "Big Bang" approach to launching, it could have very well been protracted into a stepped approach.

For example, the site could have went live on October 1 to start live testing of performance under load by the public, and allow them to merely window shop for a month or so. This could have given infrastructure engineers the time they needed to fix backend bugs, and allow for more time of the inter-agency data integrations that plagued the site for months on end. Would the White House really have garnered any larger of a black eye in light of all the other deadlines it has missed?

One must also wonder, as I mentioned, how Kathleen Sebelius was considered even qualified for such a critical role as the public face of this project. Microsoft has technical masterminds like Satya Nadella, and formerly Bill Gates, who have led the firm over the years. Amazon's got Jeff Bezos, another individual with strong a computer science background.

Even Google has Larry Page and Sergey Brin. Sebelius didn't have to be as technically astute as some of these other industry leaders, but was it appropriate for her to lead one of the biggest White House-led technical initiatives in recent memory with next to no IT project management skills or insight? She was a terrible pick, one almost inexcusably destined to fail.

If the White House was serious about spending taxpayer money wisely, it would have recruited the best and brightest to help lead the effort from you guessed it: the private sector. You know, that private sector which can stream the Olympics nearly issue-free to a global population in real time? The one which allows the likes of websites such as eBay and Amazon to thrive and innovate? And the one which is officially taking over for NASA to be in charge of bringing equipment, future spaceships and people up into space?

Those that scream we can't get things of massive nature done without government red tape and oversight should wake up and realize that the private sector is innovating all around us, on minimal budgets, and without sacrificing end result quality. The above examples are just the tip of where this is evident today.

The public sector Leviathan ensuring the status quo on federal IT projects doesn't have to continue indefinitely. An informed public, empowered through an overdue congressional vote shakeup, could slowly but surely topple this headless monster constricting IT innovation at the federal levels.

As soon as we stop expecting par for the game in Washington to be IT project failure, perhaps (only) then can we get serious about handling public sector IT projects sensibly.

Photo Credit: Gajus/Shutterstock

Derrick Wlodarz is an IT Specialist who owns Park Ridge, IL (USA) based technology consulting & service company FireLogic, with over eight+ years of IT experience in the private and public sectors. He holds numerous technical credentials from Microsoft, Google, and CompTIA and specializes in consulting customers on growing hot technologies such as Office 365, Google Apps, cloud-hosted VoIP, among others. Derrick is an active member of CompTIA's Subject Matter Expert Technical Advisory Council that shapes the future of CompTIA exams across the world. You can reach him at derrick at wlodarz dot net