Stable LM 2 1.6B Technical Report
Authors:
Marco Bellagente,
Jonathan Tow,
Dakota Mahan,
Duy Phung,
Maksym Zhuravinskyi,
Reshinth Adithyan,
James Baicoianu,
Ben Brooks,
Nathan Cooper,
Ashish Datta,
Meng Lee,
Emad Mostaque,
Michael Pieler,
Nikhil Pinnaparju,
Paulo Rocha,
Harry Saini,
Hannah Teufel,
Niccolo Zanichelli,
Carlos Riquelme
Abstract:
We introduce StableLM 2 1.6B, the first in a new generation of our language model series. In this technical report, we present in detail the data and training procedure leading to the base and instruction-tuned versions of StableLM 2 1.6B. The weights for both models are available via Hugging Face for anyone to download and use. The report contains thorough evaluations of these models, including z…
▽ More
We introduce StableLM 2 1.6B, the first in a new generation of our language model series. In this technical report, we present in detail the data and training procedure leading to the base and instruction-tuned versions of StableLM 2 1.6B. The weights for both models are available via Hugging Face for anyone to download and use. The report contains thorough evaluations of these models, including zero- and few-shot benchmarks, multilingual benchmarks, and the MT benchmark focusing on multi-turn dialogues. At the time of publishing this report, StableLM 2 1.6B was the state-of-the-art open model under 2B parameters by a significant margin. Given its appealing small size, we also provide throughput measurements on a number of edge devices. In addition, we open source several quantized checkpoints and provide their performance metrics compared to the original model.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
RITA: a Study on Scaling Up Generative Protein Sequence Models
Authors:
Daniel Hesslow,
Niccoló Zanichelli,
Pascal Notin,
Iacopo Poli,
Debora Marks
Abstract:
In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences belonging to the UniRef-100 database. Such generative models hold the promise of greatly accelerating protein design. We conduct the first systematic study of how capabilities evolve with model size for autoregressive tr…
▽ More
In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences belonging to the UniRef-100 database. Such generative models hold the promise of greatly accelerating protein design. We conduct the first systematic study of how capabilities evolve with model size for autoregressive transformers in the protein domain: we evaluate RITA models in next amino acid prediction, zero-shot fitness, and enzyme function prediction, showing benefits from increased scale. We release the RITA models openly, to the benefit of the research community.
△ Less
Submitted 14 July, 2022; v1 submitted 11 May, 2022;
originally announced May 2022.
