PowerPoint Presentation

IntroductionRohan, CEO of NavaThank you for having me here...Today Ill be sharing with you some of Navas experiences with rebuilding HealthCare.gov, and what happens after a crisis

This is us. Well, some of us.Nava is a public benefit corporation of 20 people headquartered in Washington DC, with offices in San Francisco and New YorkOur mission is to Radically improve how our government serves people

Navigating a crisis.

Navigating a crisisNava was born out of the effort to move beyond a HealthCare.gov that was merely stabilized; our goal was to build our way out of trouble, towards a service that was robust, flexible, and a model for user-centered designCrisis can often be pivotal moments, and navigating those periods is not easy. There are no easy answers, no quick fixes. With the intense, acute focus on the crisis, larger questions can fall away.

What happens after?

What happens afterAnd so what I want to talk about what happens after the crisisAfter the bright lights have dimmed a bit and the adrenaline is goneHow do we build a better future? How do we start looking further out?

How do we build lasting change?

How do we build lasting change? What kind of strategies and tactics get us there?At Nava, we spend a lot of time on this questionAnd well talk about some of our experiences and strategies weve used, that you may find helpfulWhats more important is not the strategies and tactics I talk about today, but that you ask these sorts of questions about the organizations and systems that youre working onAnd I want to acknowledge that this question is incredibly broad, even within our world of government digital services: the answer spans policy, procurement practices, culture, product practices, politics, and more

Where we are now.

Where we are nowHealthcare.gov is on solid groundHard work across Centers for Medicare and Medicaid Services, etc.Call center representatives

In the process, Nava has begun to help other agenciesSimilar stories everywhereAt SSA...At Census...federal level...state level...

Lets talk about complexity.

Where do you even start?How do we break the cycle?

Weve begun to see a pattern in the complexity after a crisis folks feel caught, trappedIm here to say that in that complexity, there is a huge amount of hope for creating firm, future-facing foundations

First things first where do you start?

Folks that we talk to who are responsible for large, complex, government programs feel caught:

Caught between a rock and commercial off the shelf software"we tried custom it didn't work, it's complex, it's costly, it's insecure, we got locked into one vendor""we are trying COTS, we can't get the product to change for what we need it to do"Going incremental with legacy systems? Takes forever, costs a lotStart over? Low success rate, throw away lots of investments that are workingAnd what Nava has found is thatWe can modernize complex architecture and systems without throwing away all your existing investments

These struggles are shared:Policy people working on legislation for decades watch the implementation fall flat on its face without any way to steerFolks on the IT side within gov struggling with requirementsAuditing legacy systems only to find a fractal-like complexityCIOs guide modern RFPs but contracting officers are unable to distinguish true skill from skillful RFP responsesMeanwhile, the systems causing issues become more and more urgentThe feeling of having no good optionsThe only way to start over is to throw everything away or do something so marginalized it has effectively zero impactOr just not knowing where the inefficiencies in the entire lifecycle are, only knowing costs in terms of production (lines of code or etc)

A playbook for building the future.

A playbook for building the futureIm going to share our strategies and some tactical examples at HealthCare.gov about how to move from crisis management to firm foundations

Close the distance

Find the seams

Set records

At a high level, its not rocket scienceThe first thing we do, is close the distanceGovernment work is made more complicated by the complexity of the organization: multiple contractorsCombat that by getting really close in your collaborationProduct, development, design, requirements: tight loops of agile, research, prototyping

In preparing for battle, I have always found that plans are useless but planning is indispensable.Dwight D. Eisenhower

What youre searching for is to adapt quickly, as a substitute for knowing the truth upfront.

Agile isnt just a label, its a way of responding to changing requirementsFor a successful technology, reality must take precedence over public relations, for nature cannot be fooled. Richard Feynman (on the Challenger explosion)Cut through to the core of user needs, requirements

[shas note from rohan speaking: closing the distance is a way of managing complexity, closing the distance across multiple dimensions is very important]

App 2.0.

What youre searching for is to adapt quickly, as a substitute for knowing the truth upfront.

Agile isnt just a label, its a way of responding to changing requirementsFor a successful technology, reality must take precedence over public relations, for nature cannot be fooled. Richard Feynman (on the Challenger explosion)Cut through to the core of user needs, requirements

App 2.0 the new health insurance application

In the last two years, Nava has worked together with CMS to redesign the health insurance application from top to bottom from the infrastructure all the way up to the user experience

Working directly with requirements people let us work find simple paths through the legislation

User ExperienceNumber of ScreensApp 2.0(handling 70% of consumers)Old insurance application

76 screens 16 screensCall center complaints down

Fully responsive designBig deal nowadays - 20% of traffic on mobileAnd a big portion of the population doesnt have access to a computer but they still have a smartphone for their work

Conversion RateApp 2.0(handling 70% of consumers)Old insurance application

Conversion rate 65% 95%We lose a 1/7 of the number of people we used to lose, 7x improvement in terms of people that decide to pull out their hair

Time to CompletionApp 2.0(handling 70% of consumers)Old insurance application

Time to completion 21 min 9 minFaster than Geico

Mike Davidson, former VP Design at TwitterA prototype is worth a thousand meetings.

Build proof of concepts to answer doubtsGive people something to react toWorking software >> excel or wordSomething is wrong when: We can start once we document the requirementsWell have that in 6 monthsThat will take 10,000 additional hours

Close the distance

Find the seams

Set records

This one is close to my heartAnd where complexity becomes opportunity in a concrete senseFind the threads to pull on that let you move through the problem one meaningful step at a timeBusiness process seam, technical seam, use case seam

Soft launching[not sure if this quote is good] http://www.shirky.com/weblog/2013/11/healthcare-gov-and-the-gulf-between-planning-and-reality/ It is hard for policy people to imagine that Healthcare.gov could have had a phased rollout, even while it is having one. - Clay Shirky

Find the seamCreate account page -> SLS -> account flows

You dont have to throw everything away to start over again.

You can use what you have, you dont have to throw everything away to start over

Soft launchhttp://www.shirky.com/weblog/2013/11/healthcare-gov-and-the-gulf-between-planning-and-reality/ It is hard for policy people to imagine that Healthcare.gov could have had a phased rollout, even while it is having one. - Clay Shirky (not sure if this quote is good :P)

Find the seamCreate account page -> SLS -> account flows

App 2.0 65% of the most common case of usersSlow rollout: 0%, 1%, 10%, 50%, 100%A/B testingStrangler pattern

SLSHot swap at the interfaceData migration

Legacy system4-6 weeks for new infrastructure4+* hour maintenance windowsSticky sessionsNot mobile friendlyApp 2.0: under the hoodApp 2.030 minutes for a new virtual data centerZero downtime deploysStateless serversResponsiveThrottled roll-out

65% of consumers: found the seamThrottled the roll out so that we could gradually go from 0%, to 1%, to 10%, etc to 100% eventually, with minimal risk

CMS staff, OIG report on HealthCare.govApp 2.0 went into production 30 days early so we could see it in the wild during special enrollment. It was very successful.

Immutable infrastructureBake binaries + code into imageAuto-scalingRollbacksGeneralized infrastructureReuse code across apps / microservicesCoreVPC: PaaS for government

Found the other seam at the DevOps layer, built our own PaaSMade sure that our work here was simultaneously government compliant, and fully modernReliable, scalable, secure, flexible

Ultimately, these investment result in UX improvements, even if infrastructure doesnt seem to directly affect users[Find the investments that start paying off immediately]

Close the distance

Find the seams

Set records

Set recordsSLS supporting 20 million accounts, greatSet the right metrics,Planned downtime is still downtimeHow far can we push this?Thinking proactively to see how systems will breakBillion user load test

Scalable Login System.

Drop-in replacement1/100th the cost, 1000x performance

Scalable Login SystemCurrently supports over 21 million users< 0.001% error rate1/100th the cost, 1000x performance

Backend system, not sexy, critical to user experience

The billion user load test.

How far can we push this?

7754 tps was attained for 1 hour with the 90th percentile at 128ms and zero errors.

Brendan Neutra, Scaling HealthCare.gov to a Billion Users7,754 transactions per second was attained for one hour with 128 millisecond response times and zero errors.

7754 tps was attained for 1 hour with the 90th percentile at 128ms and zero errors.

Build ambitionGoing up the hierarchy of needs (Program Integrity to Futureproofing to Prototyping)Magic: emergent effectsBuild new foundationsthis is a dramatic time in IT modernization effortsthis is a moment for gov to lead in terms of scalability, security, privacy, etc

We can do this.

Promote good behavior dont police bad behaviorCelebrate successes, learn from failures

Thanks!Rohan Bhoberohan@navahq.com