Streaming Data
Past Presentations
ETL is dead; long-live streams
What happens if you take everything that is happening in your company—every click, every database change, every application log—and make it all available as a real-time stream of well-structured data? I will discuss the experience at LinkedIn and elsewhere moving from batch-oriented ETL to...
The Power of Distributed Snapshots in Apache Flink
Come learn how Apache Flink is handles stateful stream processing and how to manage distributed stream processing and data driven applications efficiently with Flink's checkpoints and savepoints. Over the last years, data stream processing has redefined how many of us build data pipelines....
Data Decisions With Realtime Stream Processing
At Facebook, we can move fast and iterate because of our ability to make data-driven decisions. Data from our stream processing systems provide real-time data analytics and insights; the system is also implemented into various Facebook products, which have to aggregate data from many sources. In...
Patterns of Streaming Applications
Stream processing engines are becoming pivotal in analyzing data. They have evolved beyond a data transport and simple processing machinery, to one that's capable of complex processing. The necessary features and building blocks of these engines are well known. And most capable engines have a...
The Whys and Hows of Database Streaming
Batch-style ETL pipelines have been the de facto method for getting data from OLTP to OLAP database systems for a long time. At WePay, when we first built our data pipeline from MySQL to BigQuery, we adopted this tried-and-true approach. However, as our company scaled and our business needs grew,...
Interviews
Data Decisions With Realtime Stream Processing
QCon: What's the focus of your work and of the team that you're on at Facebook?
Rajesh: My team is working on stream processing, and we are part of the real-time data organization which focuses on faster, simpler, and smarter delivery of data. We want to reduce the time to results for people and our data driven products and people wait on that rely on data driven. Our organization encompasses the stream...
Read Full Interview