A Deep Dive Into Guidance’s Source Code
<p><em>This is a text about a library that is used to build applications with LLMs (Large Language Models). This article is rather dense, so don’t feel frustrated if you don’t understand everything at first. Beware: I might also have misunderstood some (or many) things in the process.</em></p>
<p>So if you have no idea what guidance is about, I’d recommend that you start reading the article I wrote about using it to build a docstring generator.</p>
<p>Also, you can try the following:</p>
<ol>
<li>The official library documentation.</li>
<li>A small library I wrote to serve an LLM with guidance behind an HTTP server</li>
<li>Take a peek at the Brain Chulo repository. There is some really interesting guidance prompts there written by another developer, which deserves their own article.</li>
</ol>
<p>On another day, I might go back to the basics, but today I want to dive deep into how the guidance library works.</p>
<p>Why a deep dive?</p>
<p>First, we’d like to understand how it works fully to better understand its limitations.</p>
<p>Second, we’d like to be able to extend the library to other libraries besides Hugging Face. For instance:</p>
<ol>
<li>The infamous Recurrent-Neural-Network Language Model — RWKV</li>
<li>ExLlama — a memory-efficient implementation of the LLaMA model</li>
</ol>
<p>So buckle up. Today we’re diving deep into guidance source code. Before we start, let’s set up some reproducibility. I’m looking at the main branch of guidance on the day 20 of June of 2023, at this commit: <code>5f7fa7f6eef6455e6940fe743c5bfdb557330d0b</code>.</p>
<p>To keep things simple, I often copy snippets of code from the repository and delete the commented lines of code. Believe me, there’s a lot of commended code in this library.</p>
<h1>Guidance — A Template Language</h1>
<p>The top-level construct in Guidance is called a Program. Through a program, one defines the flow of the guided program that the LLM must follow.</p>
<p>But how is it implemented? From the outside, a guidance program looks very similar to a Jinja2 template. Which means there’s probably some parsing happening, right? Exactly, the implementation uses a Grammar Language.</p>
<p><strong><a href="https://betterprogramming.pub/a-deep-dive-into-guidances-source-code-16681a76fb20">Read More</a></strong></p>