There are no emergent abilities in LLMs

The 2022 paper titled “Emergent Abilities of Large Language Models” [1] has been the source of much speculation — and much hysteria — about the future of AI [2], [3]. The paper’s central claim is that large language models (LLMs) display “emergent abilities” as they scale to larger and larger parameter counts (hundreds of billions of parameters are common model sizes now). In the paper, the authors define emergence as follows: <blockquote> “We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” </blockquote> In this article, I’m going to demonstrate why these claims are flawed, and by the end of the article, I’m certain you’ll be convinced that unexpected abilities do not suddenly “emerge” in large language models. The code to reproduce my plots can be found in this Colab notebook. <h1>The Source of the Claim</h1> In the paper, the authors use the plots below as the basis for the claims of “unpredictable” and “emergent abilities.” Figure 2 from the paper is the basis of the claims. In this article, I’ll scrutinize Figure 1 (shown below) from the paper, which is analogous (and virtually identical to) Figure 2. <a href="https://betterprogramming.pub/there-are-no-emergent-abilities-in-llms-2bb42e17ce7e">Read More</a>