Presentation: Evolution of Edge @Netflix

Yes, I can definitely talk about that. We could talk about such a journey in general terms - how you start as a new company, usually several engineers, three-tier architecture and most likely a monolithic application. Over time the company, if successful, grows, builds more functionality to the app as well as adding engineering capacity. At some point you have a little to much in a single app and might see engineers stepping on each other toes - time to introduce an isolation between domain components and potentially break the monolith aparat. While monolith is being separated into services - there are at least two questions to answer: how do you route requests to the right backends and also how you replicate common functionality (such as logging, authz, rate-limiting, DDoS protection) in all the services.

Leaving routing alone for now, one could say - let’s implement common logic as a base server, a library that is used as a foundation of all the services in the company. It’s absolutely possible to go this way, but it comes with challenges around keeping the base library up-to-date across deployments and how to support polyglot development (multiple programming languages or stacks). Alternatively, an API Gateway can be introduced to implement common functionality such common functionality, while keeping logic external to the services and ensuring uniform implementation (version) across the board. As the organization grows - being able to perform infrastructure updates without involving service owners is essential. API Gateway team can iterate easily, add features or push security updates without being blocked on other teams updating the library or sidecar or even restarting their application. Plus it solves the routing issue as well.

Absolutely. One of the things that I want to touch upon is what happens when the company goes multi-region (multiple datacenters). One needs to answer an important question - how to route clients, how do they get to your ecosystem? There are several ways of doing Global Traffic management, but chances are you’ll end up using DNS-based approach. Keep in mind that with multiple data centers occasionally you’ll need to evacuate some of them - shift all of the traffic away from it. Usually, it is done with a DNS flip: adjust DNS records, wait for DNS changes to be propagated and traffic to start flowing to new locations. Unfortunately, many clients and resolvers do not honor TTL of DNS - therefore traffic might be hitting the “unhealthy” DC after evacuation. In such cases, API Gateway can help to route requests from one region to another by doing cross-region traffic proxying - anything to ensure customer satisfaction. Since I'm touching client routing to the data center I will briefly talk about the service Netflix built recently that is used as a last-mile API gateway. This service is deployed in ISP and IX locations and is the first point of contact for many of our clients. The reasoning behind introducing such layer is to reduce the time needed for the initial TLS handshake and also reduce the time to recover from TCP errors. While doing that it also provides a more granular and responsive traffic routing capabilities and improves the failover experience - traffic can be controlled without relying on DNS propagation.

I hope Engineers and Architects who attend this talk will learn about a set of potential problems that might be ahead as business grows. It's useful to have an example of how companies approached similar issues and understand the trade-offs, even if you don’t take the same steps. It’s not a pure engineering problem, it’s a problem of scaling the organizations. By growing the number of teams, a lot of potential communication channels between them are introduced as - it’s better to designing your ecosystem to reduce the number of required connections. API Gateway helps with that.

Last year the team build a new service - Raju. The goal was to detect anomalous behavior of backend services and alert service owners on such events. That was done without changing a single line of code on the backends. The idea is that API Gateway has a centralized understanding of the health of the ecosystem -  observing all the requests going through and the responses sent to clients. Applying some statistical methods to such data we were able to identify issues at scale. Having a holistic view of your ecosystem is very powerful and it can be leveraged. Another one, maybe less positive, if you make you API Gateway config self-servicable and allow Engineers to adjust routing rules globally - provide tools to assess the impact, otherwise unexpected things may happen.