How three lines of configuration solved our gRPC scaling issues in Kubernetes

It all started with a question I asked our senior software engineer: “Forget the speed of communication. Is it really better for you to develop communication in gRPC instead of REST?” The answer I didn’t want to get came immediately: “Absolutely yes.” Before I asked this question, I was monitoring a strange behavior of our service during a rolling update and mostly when scaling pods up. Most of our microservices have historically communicated via REST calls without any issues. We have migrated some of these integrations to gRPC mostly because of the overhead of REST we wanted to get rid of. Lately, we have observed several issues that pointed in the same direction — our gRPC communication. Of course, we followed suggested practices for running gRPC without service mesh in Kubernetes like those described in <a href="https://medium.com/swlh/balancing-grpc-traffic-in-k8s-without-a-service-mesh-7005be902ef3" rel="noopener">this blog post</a> and used a headless service object on the server, and deployed client “round-robin” load balancing with DNS discovery in gRPC, etc. <a href="https://medium.com/jamf-engineering/how-three-lines-of-configuration-solved-our-grpc-scaling-issues-in-kubernetes-ca1ff13f7f06">Learn More</a>