Presentation: Taming Large State: Lessons From Building Stream Processing

We are part of the Netflix Data Science and Engineering, and specifically our team supports the creation and maintenance of data products that are used for the purpose of personalization. We process data at huge scale, making it available for the training of machine learning models. We've been experimenting with creating a lot of low-latency data sets to increase the frequency at which the models can be trained and to provide a faster feedback to the model.

The core problem that we're trying to address is that for the past three, four years stream processing has exploded, it is typically been associated with low-latency event processing. It's not as commonly used for state for aggregation of data that batch is typically used for. This is the core data problem that we're trying to address, that it is possible to do stateful processing and produce low-latency data, which is usually done through batch. But to do it in streaming is what we're trying to solve.

I would say largely it's a data architecture talk, because a lot of constructs and the challenges we faced were a result of taking a really large batch dataset that required holding a lot of state and do streaming, and all the problems that come with that. That's fault tolerance, data recovery, data reprocessing. It is about how do you think of processing this in streaming. However we have solved it in Flink, so some of the things we've used are unique. The features that we've leveraged to solve these problems are features of Flink. But I'm sure other engines also have similar features.

This talk would appeal to an audience of a large variety. As an architect, this talk would give you a good insight into how to think about creating a system that can handle this level of scale. As a data engineer, this will give you enough points to consider what kind of jobs are a good candidate to work with streaming. Because not everything needs that kind of investment. I think this talk could appeal to a variety of audiences. One thing is that we will cover a lot of technical depth. So having that prior knowledge of what it takes to build these large scale distributed systems would be good to have.

On a high level, what it entails to convert a large batch job into a low-latency stream processing job. The different constructs of creating a streaming application. And also we will talk about the challenges that were encountered on the way and how we went about solving them.

Some basic knowledge of how stream processing engines interact with queues like Kafka and what some I think basic terminology, what does it mean to do data processing or what does it mean to handle duplicate checklists? Knowing this terminology and constructs is enough. I don't think anyone needs to know the specific features or constructs of Flink or Spark, but just a broader knowledge of what stream data processing is.