-
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Authors:
Stella Biderman,
Hailey Schoelkopf,
Quentin Anthony,
Herbie Bradley,
Kyle O'Brien,
Eric Hallahan,
Mohammad Aflah Khan,
Shivanshu Purohit,
USVSN Sai Prashanth,
Edward Raff,
Aviya Skowron,
Lintang Sutawika,
Oskar van der Wal
Abstract:
How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools…
▽ More
How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \textit{Pythia} to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at \url{https://github.com/EleutherAI/pythia}.
△ Less
Submitted 31 May, 2023; v1 submitted 3 April, 2023;
originally announced April 2023.
-
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
Authors:
Katherine Crowson,
Stella Biderman,
Daniel Kornis,
Dashiell Stander,
Eric Hallahan,
Louis Castricato,
Edward Raff
Abstract:
Generating and editing images from open domain text prompts is a challenging task that heretofore has required expensive and specially trained models. We demonstrate a novel methodology for both tasks which is capable of producing images of high visual quality from text prompts of significant semantic complexity without any training by using a multimodal encoder to guide image generations. We demo…
▽ More
Generating and editing images from open domain text prompts is a challenging task that heretofore has required expensive and specially trained models. We demonstrate a novel methodology for both tasks which is capable of producing images of high visual quality from text prompts of significant semantic complexity without any training by using a multimodal encoder to guide image generations. We demonstrate on a variety of tasks how using CLIP [37] to guide VQGAN [11] produces higher visual quality outputs than prior, less flexible approaches like DALL-E [38], GLIDE [33] and Open-Edit [24], despite not being trained for the tasks presented. Our code is available in a public repository.
△ Less
Submitted 4 September, 2022; v1 submitted 18 April, 2022;
originally announced April 2022.
-
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Authors:
Sid Black,
Stella Biderman,
Eric Hallahan,
Quentin Anthony,
Leo Gao,
Laurence Golding,
Horace He,
Connor Leahy,
Kyle McDonell,
Jason Phang,
Michael Pieler,
USVSN Sai Prashanth,
Shivanshu Purohit,
Laria Reynolds,
Jonathan Tow,
Ben Wang,
Samuel Weinbach
Abstract:
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe \model{}'s architecture and trainin…
▽ More
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe \model{}'s architecture and training and evaluate its performance on a range of language-understanding, mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.
△ Less
Submitted 14 April, 2022;
originally announced April 2022.
