Reducing Data Platform Cost by $2M

<p>Welcome to the Razorpay technology blog! In this blog, we are primarily focusing on cost savings by the Data platform team.</p> <p>We have some exciting news to share with you! Through our efforts to reduce costs, we successfully managed to cut down our platform expenses by approximately $2M per year.</p> <p>In this blog, we will reveal the secrets behind our success and provide valuable insights into how we improved our operations, increased efficiency, and paved the way for a more sustainable and cost-effective future.</p> <h1>High-level Data Platform architecture</h1> <p><img alt="" src="https://miro.medium.com/v2/resize:fit:700/1*Qz67VHU58tXkBdl8yD9yng.png" style="height:893px; width:700px" /></p> <h2>Data Platform</h2> <ul> <li>We collect data from transactional aka OLTP systems like MySQL etc and push them to S3 via CDC pipelines. These CDC pipelines push the data to Kafka from Mysql via the Maxwell replicator.</li> <li>Spark pipelines run every few minutes and read the latest data from Kafka and push it to S3 in parquet/delta-lake format. Also updates the event metadata.</li> </ul> <p><a href="https://engineering.razorpay.com/reducing-data-platform-cost-by-2m-d8f82285c4ae"><strong>Click Here</strong></a></p>