Given Datadog's scale, with over 18,000 customers sending trillions of data points every day, analyzing the volume of incoming data can be challenging. One of the largest internal log sources at Datadog is network logs, and the ability to analyze and interpret these is critical to keeping Datadog secure. To make this task easier, we've built a flow analysis pipeline that alerts on network-level indicators of compromise (IOCs) such as IP address, port combinations, and data exchanged.

In this session, Andrew Krug, Technical Evangelist, will speak with Théo Guidoux, Software Engineer on the Threat Detection Platform team, and Anna Pauxberger, Software Engineer on the Security Platform team. They will discuss how they built the pipeline using Datadog and Apache Spark and how they optimized the process using new AWS features and the Datadog Cloud Security Platform.

By the end of the session, you will better understand why network traffic flows are an important tool for detecting security threats and how you can start collecting, optimizing and analyzing this data in your organization.

Please take the opportunity to connect with your friends and family and share this video with them if you find it useful.