Tag: Kafka

A Gentle Introduction to Kafka Consumer Group

You may have heard about Kafka if you are studying for a system design interview or dealing with data-intensive applications. Even if you don’t work with distributed systems before, you probably have heard of “Kafka” during a water cooler chat with your colleagues. In the system...

MLOps-Building a Real-time Data Pipeline with Kafka: Two Projects-A Step-by-Step Guide

Apache Kafka is a distributed streaming platform that allows for the processing, storage, and real-time analysis of vast amounts of data. Developed at LinkedIn and later contributed to the open-source community, Kafka is designed to handle data streams from various sources and deliver them to multip...

How We Scaled to 100 Million Active Users Using Kafka and Golang — Eventual Consistency

Nowadays, we have reached an era where the most popular startups reach millions of users within less than a year. During my experience as a software developer, where I had the privilege to work on a couple of them, I’ve seen that the most common bottleneck within a backend service is caused by...

MLOps-Building a Real-time Data Pipeline with Kafka: Two Projects-A Step-by-Step Guide

Introduction: What is Kafka? Apache Kafka is a distributed streaming platform that allows for the processing, storage, and real-time analysis of vast amounts of data. Developed at LinkedIn and later contributed to the open-source community, Kafka is designed to handle data streams from various...

kafka terraform: Mongey/kafka provider setup

Obtaining bootstrap servers and ca.cert are straight forward. Getting cert.pem and key.pem requires some work. Below describe how to obtain these fields. Bootstrap Servers bootstrap_server can be obtained either from console UI: Amazon MSK / Clusters / <c...

Real-Time Logistics, Shipping, and Transportation with Apache Kafka

Logistics is the detailed organization and implementation of a complex operation. It manages the flow of things between the point of origin and the point of consumption to meet the requirements of customers or corporations. The resources managed in logistics may include tangible goods...

Scalable and Reliable Kafka Ingestion from Rails

A key aspect of fleet management is real-time location tracking of all the entities in your fleet (these entities can be trucks, drivers, or trailers). To this end, KeepTruckin introduced Fleet View last year, a new experience for fleet managers, which displays a single, unified, real-time...