Presentation: Building Confidence in Healthcare Systems Through Chaos Engineering

I'm an engineer at Cerner Corporation, and I work within our Service Platform Engineering group. We focus on the common service needs that support our electronic medical record called Millennium. This includes how we build, deploy, and operate the services supporting these clinical workflows within the Millennium system.

I try to bring a basic practitioner's perspective of how we started changing an existing system which includes the technology choices to help in that change. We had a lot of existing services that were labeled as these enterprise Java-based services running within technologies like IBM WebSphere Application Server which introduced challenges, how it is configured, how is tuned and optimized. Those environments became very crystallized around how we build and operate something. As we started introducing newer technologies that was changing how these service workloads operated, we had to take a lot of different approaches to build confidence in the newer system. This required us bringing different groups of our organization together as a single team, to ensure that all the parts of our infrastructure supporting our system was part of these tests. As we identified scenarios to test, we wanted to make sure these experiments were based on our production systems, not on an isolated lab that could have different characteristics. As part of this talk, I will share the tactics of how we applied traffic management patterns so we could shadow or replay traffic in these environments to safely introduce these experiments in the production environment, while not impacting existing user workflows.

Right, we actually were inspired by Netflix on these approaches, but knew we had to show how we could safely introduce this approach in our environments. This required us to think about our existing change control policies, how we manage live traffic into these environments, and how we had to be closely aligned as a team when doing these tests. We found that we were building confidence and safety in the system by introducing these types of experiments within our infrastructure, since we were proactively identifying and learning about different behaviors of the system. As others have shared on chaos engineering, we weren’t trying to just break things in production and then see what would happen. We were planning experiments that were expecting to successfully handle the introduction of the system failure, but we often would learn about newer compounding failures or something that was more sensitive to the failure than expected. Hopefully, as I share some of our stories and how we approach introducing these experiments, it can illustrate how we worked through some of the barriers in a regulated environment to get this included and how the system was improved as a result.

I would like to give them some tangible things of how we introduced this approach and share the benefits we found as a result. In addition, I plan to share specifics on how we used certain technologies to do the types of tests. These types of experiments were rich with learning and always introduced surprises which forces you to focus upfront on observability in your system. My hope is to share how these learnings helped us prioritize what we viewed were important capabilities in our delivery system, so that we could incrementally build the system and get insight to how it handled these planned failures in earlier iterations. As a result, my goal is that people can walk away having a toolkit of examples and guidance on how we approached it, and hopefully they can leverage these ideas within their own organizations.