Presentation: Anti-Entropy Using CRDTs on HA Datastores @Netflix

I work on distributed datastores at Netflix, primarily as a developer on Dynomite, a highly available globally distributed datastore backed by Redis. Dynomite is typically used by globally distributed services within Netflix that have require low-latency with varying levels of consistency.

A fundamental challenge with a highly available datastore is data consistency across replicas. Traditionally, systems have had a number of ways of dealing with it, the most important of them being having a repair process that takes care of repairing replicas out of sync. This talk aims to show how we can use CRDTs to implement a self healing system for data-structures natively supported by Redis and the practical challenges involved.

I hope to leave the audience with a better understanding of CRDTs after the talk, how they may identify opportunities where the application of CRDTs are a "good fit", and hopefully with this understanding, how they may apply it to their own use cases.

Having worked as a developer on databases and also supporting managed databases for a while, I believe the next big disruption in this space is having self-managed databases that gets rid of a lot of the mundane administration tasks typically done by humans today. However, this will definitely take a while to achieve with possible incremental progress; just as self-driving cars may slowly be able to take over mundane driving tasks but not be adequate for complicated scenarios, so will databases slowly shed the need for human intervention as systems become smarter and learn to better manage themselves. Professor Andy Pavlo is the leading researcher in this space and it's quite exciting to track his group's progress in this regard.