Presentation: Handling Real-time Distributed Data Ingest
Abstract
Software solutions, such as those for personalization, metering, IoT require processing of extremely large volumes of data in real time. High-speed data ingest and processing poses several challenges such as
- Managing large volume of data sometimes arriving in bursts
- Receiving data from multiple sources
- Filtering, analyzing or forwarding data with different formats
Redis, the high-speed, open source in-memory database platform offers data structures and messaging services, that enable combining high-speed data ingest and real-time analytics. This session will cover the architectural and design challenges in accomplishing fast data ingest and real-time analytics, and in combining the both. The second half of the session will focus on performing a live demonstration of combining fast data ingest and real-time analytics on Redis using
- Redis Pub/Sub – publish and subscribe messaging framework
- Redis List data structure for message queueing and filtering
- Time series data modeling using Redis Sorted Set and Pub/Sub
The demonstration will show how to perform real-time analytics with examples in filtering, classification, aggregation, and deduplication.
