A Deep Dive Into Guidance’s Source Code

<p><em>This is a text about a library that is used to build applications with LLMs (Large Language Models). This article is rather dense, so don&rsquo;t feel frustrated if you don&rsquo;t understand everything at first. Beware: I might also have misunderstood some (or many) things in the process.</em></p> <p>So if you have no idea what guidance is about, I&rsquo;d recommend that you start&nbsp;reading the article I wrote about using it to build a docstring generator.</p> <p>Also, you can try the following:</p> <ol> <li>The&nbsp;official library documentation.</li> <li>A small library I wrote to serve an LLM with guidance behind an&nbsp;HTTP server</li> <li>Take a peek at the&nbsp;Brain Chulo repository. There is some really interesting guidance prompts there written by another developer, which deserves their own article.</li> </ol> <p>On another day, I might go back to the basics, but today I want to dive deep into how the guidance library works.</p> <p>Why a deep dive?</p> <p>First, we&rsquo;d like to understand how it works fully to better understand its limitations.</p> <p>Second, we&rsquo;d like to be able to extend the library to other libraries besides Hugging Face. For instance:</p> <ol> <li>The infamous Recurrent-Neural-Network Language Model &mdash;&nbsp;RWKV</li> <li>ExLlama &mdash; a memory-efficient implementation of the&nbsp;LLaMA model</li> </ol> <p>So buckle up. Today we&rsquo;re diving deep into guidance source code. Before we start, let&rsquo;s set up some reproducibility. I&rsquo;m looking at the main branch of guidance on the day 20 of June of 2023, at this commit:&nbsp;<code>5f7fa7f6eef6455e6940fe743c5bfdb557330d0b</code>.</p> <p>To keep things simple, I often copy snippets of code from the repository and delete the commented lines of code. Believe me, there&rsquo;s a lot of commended code in this library.</p> <h1>Guidance &mdash; A Template Language</h1> <p>The top-level construct in Guidance is called a Program. Through a program, one defines the flow of the guided program that the LLM must follow.</p> <p>But how is it implemented? From the outside, a guidance program looks very similar to a Jinja2 template. Which means there&rsquo;s probably some parsing happening, right? Exactly, the implementation uses a Grammar Language.</p> <p><strong><a href="https://betterprogramming.pub/a-deep-dive-into-guidances-source-code-16681a76fb20">Read More</a></strong></p>