Description:
This session builds on your existing knowledge of batch data processing! We'll dive into the world of data streaming and give you the skills to seamlessly integrate a real-time pipeline into your data lake. Discover how you can use Apache Kafka and Apache Spark to ingest and process information as it's generated, unlocking the power of continuous data flow. Learn how this real-time data can be seamlessly integrated into your existing data lake and eventually fed into your data warehouse to perform even deeper analytics. Gain valuable insights from a combined batch and real-time approach that enables you to make faster, more informed decisions.

Agenda:

1. What is data streaming?

– Understand the concept of continuous data flow.

– Real-time vs. batch processing.

– Benefits and use cases of data streaming.

2. Data streaming channels

– APIs (Application Programming Interfaces)

– Events (signals generated by the system)

– Webhooks (HTTP callbacks triggered by events)

3. Data streaming components

– Message broker (Apache Kafka)

– Producers and consumers

– Topics on data categorization

– Stream processing engine (Apache Spark Structured Streaming)

4. Solution design and architecture

– Real-time data source integration

– Use of Kafka for reliable message delivery

– Spark Structured Streaming for real-time processing

– Write processed data to the data lake

6. Question and answer session

– Have your questions answered by the moderators.

Why participate:
– Stay up to date: Gain a comprehensive understanding of data streaming, a critical aspect of modern data engineering.

– Gain real-time insights: Learn how to use streaming data for instant processing and analysis to make faster decisions.

– Master Kafka and Spark: Discover the power of Apache Kafka as a message broker and Apache Spark Structured Streaming for real-time data processing.

– Build a robust data lake: Discover how to integrate real-time data into your data lake to achieve a unified data repository.

– Ask the experts: Get your questions answered by data technology experts during the question and answer session.

Please RSVP to secure your spot for this session. We believe in fostering a welcoming and inclusive environment where everyone's unique perspectives are valued and contribute to our shared success.

Please take the opportunity to connect with your friends and family and share this video with them if you find it useful.