7 Frameworks for Serving LLMs

While browsing through LinkedIn, I came across a comment that made me realize the need to write a simple yet insightful article to shed light on this matter: <blockquote> “Despite the hype, I couldn’t find a straightforward MLOps engineer who could explain how we can deploy these open-source models and the associated costs.” — Usman Afridi </blockquote> This article aims to compare different open-source libraries for LLM inference and serving. We will explore their killer features and shortcomings with real-world deployment examples. We will look at frameworks such as vLLM, Text generation inference, OpenLLM, Ray Serve, and others. <blockquote> Disclaimer: The information in this article is current as of August 2023, but please be aware that developments and changes may occur thereafter. </blockquote> <h1>“Short” Summary</h1> <img alt="" src="https://miro.medium.com/v2/resize:fit:1000/1*Yym4eiSJn7fOQSXnAt-KYg.png" style="height:392px; width:1000px" /> Comparison of frameworks for LLMs inference Despite the abundance of frameworks for LLMs inference, each serves its specific purpose. Here are some key points to consider: <a href="https://betterprogramming.pub/frameworks-for-serving-llms-60b7f7b23407">Read More</a>