Presentation: Predictive Datacenter Analytics With Strymon

Track: Stream Processing In The Modern Age

Location: Bayview AB

Day of week:

Slides: Download Slides

Level: Intermediate

Persona: Architect, CTO/CIO/Leadership, DevOps Engineer, General Software, Technical Engineering Manager

Abstract

A modern enterprise datacenter is a complex, multi-layered system whose components often interact in unpredictable ways. Yet, to keep operational costs low and maximize efficiency, we would like to foresee the impact of changing workloads, updating configurations, modifying policies, or deploying new services.

In this talk, I will share our research group’s ongoing work on Strymon: a system for predicting datacenter behavior in hypothetical scenarios using queryable online simulation. Strymon leverages existing logging and monitoring pipelines of modern production datacenters to ingest cross-layer events in a streaming fashion and predict possible effects of such events in what-if scenarios. Predictions are made online by simulating the hypothetical datacenter state alongside the real one. Driven by a real-use case from our industrial partners, I will highlight the challenges we are facing in building Strymon to support a diverse set of data representations, input sources, query languages, and execution models.

Finally, I will share our initial design decisions and give an overview of Timely Dataflow; a high-performance distributed streaming engine and our platform of choice for Strymon’s core implementation.

Speaker: Vasia Kalavri

PMC Member of Apache Flink (Core Developer Graph Processing API) & Postdoctoral researcher at the ETH Zurich Systems group

Vasia Kalavri is a Postdoctoral researcher at the ETH Zurich Systems group, where she is working on distributed data processing, data center performance, and graph streaming. She is a PMC member of Apache Flink and a core developer of its graph processing API, Gelly. Vasia has a PhD in Distributed Computing from KTH, Stockholm and UCLouvain, Belgium, and she has previously interned at Telefonica Research and data Artisans.

Find Vasia Kalavri at

Similar Talks

Coding without Complexity

Qcon

CEO/Cofounder @darklang

Ellen Chisa

Scaling Patterns for Netflix's Edge

Qcon

Playback Edge Engineering @Netflix

Justin Ryan

AWS Cloud Development Kit (CDK)

Qcon

Developer Tooling Advocate @AWSCloud & CDK Core Contributor

Richard Boyd

Helm 3: A Mariner's Delight

Qcon

Principal Program Manager @Microsoft & K8s Release Lead for 1.16

Lachlan Evenson