Video: Leveraging Databricks AutoLoader: Better Visibility of CloudTrail Logs (Hebrew)

S3 logs generated by AWS CloudTrail provide organizations with essential visibility into user activity and resource utilization within their AWS infrastructure. However, working with raw CloudTrail logs can be challenging due to their size, complexity, and the need for optimal storage and query performance. Our SecOps team had 180TB of these logs in an S3 bucket, which took forever to query, so they came to us looking for a better solution. In this talk, we discussed our journey to find the best solution for this problem, and why we ended up using Databricks AutoLoader, an automatic and scalable data ingestion mechanism, to do it. We talked about the various approaches we attempted to use; AutoLoader with its advantages and features and the lessons we learned along the way. <a href="https://medium.com/riskified-technology/video-leveraging-databricks-autoloader-better-visibility-of-cloudtrail-logs-hebrew-b22c43059d1c">Click Here</a>