QCon San Francisco 2021 November 1-5, 2021 | | Avoiding Alerts Overload From Microservices

What You’ll Learn

Focus on the stuff that matters when it comes to developing Microservices
Understand how to build a system with support in mind
Understand practical approaches to optimizing alerts and the infrastructure to support them

Abstract

Microservices can be a great way to work: the services are simple, you can use the right technology for the job, and deployments become smaller and less risky. Unfortunately, other things become more complex. You probably took some time to design a deployment pipeline and set up self-service provisioning, for example. But did the rest of your thinking about what “done” means catch up? Are you still setting up alerts, run books, and monitoring for each microservice as though it was a monolith?

Two years ago, a team at the FT started out building a microservices-based system from scratch. Their initial naive approach to monitoring meant that an underlying network issue could mean 20 people each receiving 10,000 alert emails overnight. With that volume, you can’t pick out the important stuff. In fact, your inbox is unusable unless you have everything filtered away where you’ll never see it. Furthermore, you have information radiators all over the place, but there’s always something flashing or the wrong color. You can spend the whole day moving from one attention-grabbing screen to another.

That team now has over 150 microservices in production. So how they get themselves out of that mess and regain control of their inboxes and their time? First, you have to work out what’s important, and then you have to ruthlessly narrow down on that. You need to be able to see only the things you need to take action on in a way that tells you exactly what you need to do. Sarah shares how her team regained control and offers some tips and tricks.

Question:

What is the focus of your work today?

Answer:

I’m the tech lead for the Content platform at the Financial Times. The platform handles publication of content from multiple content management systems, annotating that content with metadata via concept extraction and editorial curation, and making all of that information available via a set of APIs, so any product within the FT or outside that delivers our content has a stable base to build on.

As part of this work, we are completely revamping our metadata from a taxonomy based metadata - essentially, lists of terms in a variety of categories such as authors, people, companies - with an ontology based one, i.e. based on real things, with an ability to navigate the relationships between those things. As an example, in our new metadata, an author is a person who is a writer. This means we don’t have the same name appearing as a term within people, authors and brands (this is highly confusing to deal with) and gives us a lot more flexibility to show content based on that metadata.

We have five development teams working on our system, which is made up of nearly 300 microservices. We were early adopters of docker, building a lot of the cluster management tools ourselves. My focus this year is making sure that all the work we do on our platform works towards our functional and architectural goals - no local optimizations - and that our production stack is as stable and easy to use as possible so we spend time on the things that matter to our business: for example, we are migrating to Kubernetes and replacing some of our hand written tools.

Question:

What’s the motivation for your talk?

Answer:

This was the first microservices architecture I’ve worked on, and one of the early things I learnt is that you just can’t operate microservices the way you did a monolith.

Microservices make the code easier to reason about and deploy frequently, but you have to do DevOps to make it work, and you have to keep a keen focus on building things for operability.

In the early days, my inbox was full of alerts every day, and it was very hard to work out what was something I needed to take action on. We’ve invested a lot of time in finding ways to solve that problem and I wanted to share that experience with others.

Question:

How you you describe the persona of the target audience of this talk?

Answer:

Tech Lead/Architect/Developer/Senior Management: anyone operating a microservices architecture or planning to.

Question:

How would you rate the level of this talk?

Answer:

It’s a technical talk. It assumes you know about microservices, that you have been responsible for supporting a system.

Question:

QCon targets advanced architects and sr development leads, what do you feel will be the actionable that type of persona will walk away from your talk with?

Answer:

There are lots of concrete suggestions of things to install or develop that will help with operating a microservices-based system, but I hope the main actionable will be that you have to care about this stuff and work on it constantly.

Question:

What do you feel is the most disruptive tech in IT right now?

Answer:

Serverless feels like it could completely change things. At the moment I’m seeing us use Lambda mostly for more ‘housekeeping’ tasks rather than for production-critical systems, but that’s going to change and I can see the attraction of not having a server to maintain.

I wonder how easy it will be to support a system made up hundreds of functions running on hardware you can’t see. It’ll be a whole new set of observability challenges!

Speaker: Sarah Wells

Principal Engineer @FT (Financial Times)

Sarah Wells is currently leading work at the FT on building a semantic publishing platform, making it easy to discover and access all the FT’s published content via APIs in a common and flexible format. Sarah has been a developer for 15 years, working across consultancy, financial services, and media. She is more dev than ops, but definitely shifting. Her recent focus has been on Go, microservices, containerisation, devops, and how to influence teams to do the right things.