Training LLMs over Neurally Compressed Text
Abstract
In this paper, we explore the idea of training large language models (LLMs) over highly compressed text. While standard subword tokenizers compress text by a small factor, neural text compressors can achieve much higher rates of compression. If it were possible to train LLMs directly over neurally compressed text, this would confer advantages in training and serving efficiency, as well as easier handling of long text spans. The main obstacle to this goal is that strong compression tends to produce opaque outputs that are not well-suited for learning. In particular, we find that text naïvely compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome this, we propose Equal-Info Windows, a novel compression technique whereby text is segmented into blocks that each compress to the same bit length. Using this method, we demonstrate effective learning over neurally compressed text that improves with scale, and outperforms byte-level baselines by a wide margin on perplexity and inference speed benchmarks. While our method delivers worse perplexity than subword tokenizers for models trained with the same parameter count, it has the benefit of shorter sequence lengths. Shorter sequence lengths require fewer autoregressive generation steps, and reduce latency. Finally, we provide extensive analysis of the properties that contribute to learnability, and offer concrete suggestions for how to further improve the performance of high-compression tokenizers.
1 Introduction
Today’s large language models (LLMs) are almost exclusively trained over subword tokens. The tokenizers used to produce these tokens—often BPE [23, 56] or Unigram [37], as implemented by the SentencePiece library [38]—are compressors that typically achieve ~4 compression over natural language text [74].111We refer here to “token-level” compression rate, i.e., the length reduction between a raw UTF-8 byte sequence and the corresponding sequence of subword tokens. If instead we measure the number of bits required to encode the two sequences, subword compression typically delivers ~2 or less compression, depending on vocabulary size, which typically ranges from 32k to 256k. See Section 3.4 for discussion. While these tokenizers “hide” the character-level makeup of each token from the LLM [74, 44], this downside is widely seen as outweighed by the significant benefits of compression. Compared to raw byte-level models, an LLM trained over subword tokens sees ~4 more text per token, allowing it to model longer-distance dependencies, ingest more pretraining data, and predict more text at inference time, all without increasing compute.222The increased cost of the input embedding and final softmax layers due to increased vocabulary size is negligible for all but the smallest models.
Given these advantages, it raises the question, could we compress text further to achieve even greater gains? It is well known that autoregressive language models can be turned into lossless text compressors, and recent work has shown that LLMs can easily achieve 12 compression over English text [16].333Specifically, the authors show that Chincilla 70B [30] can compress 2048-byte subspans of enwik9 at a 12 bit-level compression rate. Can we simply train an LLM over this neurally compressed text?
In this paper we explore various options for doing so, focusing primarily on the idea of using Arithmetic Coding (AC) [73], which is known to reach the near-optimal compression rate for a particular model that assigns probabilities to text continuations. Figure 1 presents our high-level approach. First, a small language model “M1” is trained over raw byte sequences. Next, this frozen model is used to compress pretraining corpus text by applying a standard compression algorithm like AC. The resulting compressed bitstream is then chunked into tokens, which are used to train “M2”, a language model that directly reads and writes neural-compressed text.
Given a perfect probabilistic model of the raw byte sequence, the compression step would output a fully-compressed bitstream that would be indistinguishable from random noise, and hence unlearnable by M2. In reality, M1 can never be perfect [78], so the M1-compressed output will still contain learnable patterns. We explore whether using compression powered by a relatively small M1 is able to “remove” the simple structure that M1 understands from the input—e.g., patterns of spelling, word frequency, and basic grammar—while retaining any higher-level structure that M1 fails to model—e.g., patterns requiring “deeper” reasoning and long range coherence. A larger M2 would then learn to model this higher-level structure, without needing to relearn the low-level structure removed by M1.444Intuitively, training M2 could be seen as analogous to fitting the residuals of M1 [21]. In theory, this process could be repeated by training an even-larger M3 model on text compressed by M2, and so on.
In practice, we find that text compressed via Arithmetic Coding is not readily learnable by a standard transformer-based LLM, with resulting models predicting tokens at chance. Interestingly, this result holds even when M1 is reduced to a context-free unigram model, suggesting that the challenge of modeling AC-compressed text stems from the difficulty of learning the AC compression and decompression process itself. We verify this hypothesis by showing that even the sub-tasks of AC-compressing and AC-decompressing text are not learned well beyond a few initial tokens.
To aid learnability, we propose compression via Equal-Info Windows, a simple technique that breaks text into contiguous windows and compresses them via Arithmetic Coding independently. Rather than splitting text into windows of equal text length, we track the number of bits output by the compressor, and close each window just before it exceeds a set information threshold (e.g., 32 bits of information). This has the advantage that when chunking the subsequent bitstream into M2 tokens, there is a stable mapping from N tokens to one window (e.g., four 8-bit tokens one 32-bit window). At each window boundary, we reset both AC algorithm and the M1 model context. This ensures that each window may be mapped back onto raw text without any additional information.
Through ablations on window size and M2 vocabulary size, we find that Equal-Info Windows make learning of AC-compressed text possible across a range of settings. However, we also observe that learning progresses gradually, starting with tokens at the left edge of each window, and for longer windows, the model learns little about the tokens near the right edge. Our best-performing setting uses short 16-bit windows that each correspond to a single 16-bit M2 token. Despite resetting the compression algorithm every 16 bits, we still achieve ~5.3 token-level compression overall, which exceeds standard subword tokenizers. Remarkably, our best M2 models outperform byte-level baselines on perplexity benchmarks (bits/byte) for fixed computation budget (FLOPs/byte). This shows that learning over neural-compressed text can be effective.
At the same time, our best M2 models underperform subword baselines. We suspect this is due at least in part to the relatively unstable mappings our neural tokenizers induce between words and tokens. By contrast, standard subword tokenizers induce essentially stable word-to-token mappings, which likely makes the token sequences they output well-suited for LLM training.555See Appendix L for some counterexamples to subword tokenizers producing stable word-to-token mappings. We illustrate this contrast through qualitative examples. Whether a neural tokenizer can reach a high level of compression while maintaining high learnability for LLM training is an interesting question for future research.
Our main contributions are as follows: (1) Outline advantages and challenges of training over neural compressed text. (2) Compare LLMs trained over different tokenizers along two axes: bits/byte and FLOPs/byte. (3) Show that standard LLMs can’t learn to model vanilla AC-compressed text. (4) Show that GZip-compressed text is learnable by standard LLMs, but not competitive. (5) Propose compression via Equal-Info Windows, and show that it enables learning over neural compressed text.
2 Motivation and Background
2.1 Advantages of Training over Neural-Compressed Text
Training LLMs over compressed text is appealing for many reasons. We discuss three advantages in detail below.
Efficiency
The most straightforward advantage is efficiency. By compressing the same text into a shorter token sequence, the model can process more text for the same computational cost. In particular, a model trained over compressed text will see more text during training compared to a model trained over raw text, given an equal compute budget. Increasing the amount of data seen in pretraining is often an effective means of improving performance [35, 30]. Processing text more efficiently also confers benefits at inference time, reducing the serving cost for handling a request of a given prompt and continuation length. In addition to reducing the raw compute needed for inference, compression can also improve inference latency, since generating better-compressed output requires fewer sequential autoregressive steps.
Longer Context
A second advantage is that working with compressed text allows modeling longer contextual dependencies. In vanilla transformer-based models, computation for the self-attention layer scales quadratically with the sequence length, . This has limited the sequence lengths used by such models in practical settings to ~k tokens.666Exploring sub-quadratic attention mechanisms is an area of active research [1, 70, 36, 75, 5, 9, et alia]. However, regardless of the cost of attention, compressing the input increases the effective context “for free”. If, via compression, each token represents (on average) bytes of raw text, then the resulting LLM can model dependencies across longer distances compared to a raw text model operating over the same token sequence length. While the benefits of modeling longer context (beyond ~ bytes) are modest when viewed merely as perplexity gains [51], the ability to condition on long context is critical for many applications, such as retrieving content from a document, or answering a coding question provided documentation.
Distribution of Compute
A third potential advantage of training over compressed text is that information will be spread more uniformly across the sequence. By the nature of compression, a text span that is relatively predictable (e.g., a boilerplate notice) will be more compressible than a span with high perplexity (e.g., a unique product serial number). When an LLM is trained over well-compressed text, each token will represent roughly an equal amount of information. Since the LLM allocates equal compute to each token, this amounts to allocating more compute for “harder” text spans. This adaptivity is similar in spirit to “Adaptive Computation Time” (ACT) [27], which learns to allocate additional compute at some sequence positions in an end-to-end manner, but with the advantage that in our case the computation remains “dense”—identical operations are applied at each position.777It should be noted that ACT learns to allocate more compute where it is useful, as opposed to merely where the predictions are hard. For example, ACT learns to not waste compute on inherently unpredictable text spans. We expect that as a heuristic, allocating more compute to higher-perplexity text spans is valuable, but leave this to future work to verify.
2.2 Challenges of Training over Compressed Text
Learnability
It is not at all obvious what types of compression are “transparent” enough to be learnable through a standard LLM training process. Strong compression can be seen as removing as much redundant or predictable information from a sequence as possible. Consequently, the bitstream output by a good compressor is inherently hard to distinguish from random noise. In this work, we explore the setting where M2—the model trained over compressed text—has a larger capacity than M1, the model used for compression. In principle, this setup should allow M2 to extract additional information from the signal even after M1 has compressed it. However, for strong enough M1 compression, the resulting bitstream may be too noisy to detect any signal.
As a prerequisite for M2 to effectively predict continuations of compressed text, we anticipate that it is necessary for M2 to have the ability to decompress bits text and compress text bits. These sub-tasks are challenging in their own right. First, M2 needs to accurately “simulate” M1 in order to know the probabilities it assigns to the text, which determine the output of compression.888For Arithmetic Coding, not only would M2 need to know the probabilities M1 assigns to the observed text, but it would also need to know the probabilities assigned to many unobserved symbols. This is because Arithmetic Coding operates over cumulative probabilities, i.e., the probability that the next symbol is e or any alphabetically preceding symbol. Training models to mimic other models can be difficult [41], and even in settings where models do learn to copy the behavior of another network [29], this is often only when looking at which symbol was assigned the highest probability—the actual probabilities assigned often differ [60]. Second, M2 needs to learn the compression procedure itself. In our case, this means tracking the Arithmetic Coding algorithm, which requires maintaining high-precision numerical state across long contexts. We investigate these sub-tasks in detail in Section 5.2.
A further learnability challenge is the high level of context sensitivity needed to interpret a bitstream of compressed text. When chunked into tokens, a particular bit subsequence (e.g., 10111001) can map onto the same token despite having no stable “meaning” across occurrences. We show examples in Section 6.1, where a token maps to many different underlying text forms, necessitating strong contextual understanding. While LLMs are robust to some level of polysemy, as highlighted by the success of Hash Embeddings [62] where multiple unrelated words share a single token representation, we suspect this has its limits.
Numerical Stability
An additional technical challenge is that compression methods can be sensitive to the precise model probabilities used. To achieve lossless compression in our setup, it is critical that the M1 probabilities match during compression and decompression. This can be hard to guarantee in practice, as there are many sources of numerical noise in LLM inference, especially when running on parallel hardware. An expanded discussion of numerical stability issues can be found in Section 3.7.
Multi-Model Inference
Finally, a specific challenge of training over neural compressed text is that multiple models need to be stored and run side-by-side in order to perform inference. We assume that if M1 is relatively small, this additional overhead is not a significant drawback compared to a standard tokenizer, which is also a separate model that is needed to tokenize text input and detokenize LLM outputs. In evaluating our approach, we include M1 compute in our calculations of total inference cost (FLOPs/byte).
2.3 Compression
In this work, we focus on lossless compression, which aims to encode a sequence of input symbols, , into a bitstream while minimizing the expected length of the bitstream. Compression methods are often factored into a “modeling” component and a “coding” component [45]. The input sequence can be viewed as a sample from a true distribution , , with a standard autoregressive decomposition, . The “modeling” component aims to approximate with . While some compression algorithms assume static probabilities for each symbol, stronger algorithms are “adaptive”, meaning that symbol probabilities may change based on context. In this work, we use context-aware transformer-based language models to represent .
The “coding” component of a compression algorithm converts the input sequence to a bitstream of length . To maximize compression, we want a coding algorithm that minimizes the expected number of bits in the bitstream, . This is done by assigning shorter bit sequences to common symbols and longer sequences to less common ones.999This process can result in extremely uncommon sequences becoming longer under compression, as no algorithm can compress all possible input strings [45]. In practice, natural language inputs are highly compressible and these edge cases are inputs that one would not recognize as natural language. The expected length is lower bounded by where [57]. This means that, given a near-optimal coding algorithm, the achievable level of compression derives from how well the model approximates .
2.4 Arithmetic Coding
Arithmetic Coding [53, 49] uses a model to compresses a sequence to a bitstream, which is the binary expansion of a float . The float is found by assigning successively smaller sub-intervals to each symbol , with the final interval enclosing . An interval is made of an upper and lower bound, and its size is given by . Starting with , at each step of encoding, the interval for the symbol is created by partitioning the interval based on the cumulative distribution of given the previous context, . The size of this interval is given by . Thus:
where is the symbol before in a strict ordering of , i.e., is the previous token in the vocabulary. Finally, the bitstream of minimal length that represents the binary expansion of a number inside the final interval is used as the compressed representation.
Equivalently, the binary expansion can be seen as maintaining a bitstream prefix and creating successive intervals by partitioning the current interval in half. If the first interval is chosen, a bit is appended to the bitstream prefix , while choosing the second interval appends a .
Once the final interval is computed, smaller and smaller bit intervals are created until reaching a bit interval that is fully enclosed by . At this point, the corresponding bitstream is the final compressed representation.
The coding component of Arithmetic Coding is nearly optimal: the output bitstream will have a length of bits when using infinite precision. In the finite precision setting using bits, an extra bits are added [31]. See [73] for an example implementation. In our experiments, we use precision . The practical effect of using a finite precision implementation of Arithmetic Coding is that the model’s cumulative distribution gets quantized to integers using bits. This results in a minimum probability of being assigned to all tokens.
2.5 Related Work
Recent work has looked at using large language models for compression, but has not to our knowledge attempted to train subsequent models over the resulting compressed output. Works like [16] use a transformer language model as the modeling component of Arithmetic Coding, but they do not train over compressed output nor do they make modifications to the compression algorithm to facilitate learnability by downstream models. Additionally, they focus on the setting of compressing fixed-size sequences of bytes. By contrast, our models operate over input sequences of fixed token length. This allows for models with higher compression rates to leverage longer contexts, as more bytes are included in the input.
[63] proposes changes to Arithmetic Coding to make it more amenable to use with LLMs—namely, they rank sort the logits from the model before creating text intervals, . This could help alleviate issues stemming from errors in M2’s simulation of M1. However, they do not train models on top of their compressed output.
Some approaches to “token-free” (i.e., purely character- or byte-level) language modeling down-sample the input sequence via convolutions [13, 61], which could be seen as a form of end-to-end neural tokenization. However one important distinction is that the resulting tokenization is “soft”—outputting high-dimensional vectors and not implying a discrete segmentation—in contrast to our tokenization that outputs discrete tokens.
Methods for learning discrete tokenization end-to-end have also been proposed [11, 25]. In the case of MANTa [25], the learned segmentation appears to be fairly semantic (i.e., respecting word and morpheme boundaries), which could be an advantage over our approach. However, they lack our bias towards encoding an equal amount of information per token.
In modeling audio, it is common practice to use learned tokenizers that compress the raw input signal to discrete tokens from a fixed-size codebook [64, 3, 12, 6]. However, this compression is lossy, whereas we focus on lossless compression.
Other recent work focuses on using the “modeling” component from well-known compressors to do other tasks. [34] uses the model from GZip to perform text classification. [68] uses the Arithmetic Decoding algorithm with an LLM as the model to do diverse parallel sampling from that LLM. One could imagine that the “model” of our compressors (M1) is a teacher for M2, but unlike these other applications, the M1 values are not used outside of compression.
[40] also explores learning over compressed text, but with several key differences. First, they use n-gram language models [57] while we use LLMs. Second, their model is conditioned on compressed bitstreams but produces a distribution over the raw, uncompressed, bytes while our M2 models predict directly in the compressed space. Additionally, they only consider static Huffman coding [32] as the algorithm to compress model inputs. While this avoids the context sensitivity issues we outline in Section 2.2, it results in a far worse compression rate compared to the adaptive compression methods we use. One important distinction is that their equal-information windows are overlapping, and used as a sliding window to provide context to their n-gram language model. By contrast our equal-information windows are non-overlapping, and used to segment text into a series of equal-length bitstrings that can be interpreted independently by M2, and whose boundaries are easily identifiable, as they map to a fixed number of M2 tokens.
Concurrently, [26] explores how the compression performance of a tokenizer correlates with downstream model performance. They find that tokenizers that compress better perform better, which generally aligns with our findings, particularly in the large vocabulary setting, see Fig. 6. However, we find that using the strongest compressors is detrimental to learnability, as seen in the AC line in Fig. 3. These conflicting results likely stem from differences in tokenization strategy. Their work is restricted to BPE-based compressors while we explore stronger compressors built on LLMs and Arithmetic Coding. The qualitative differences between these classes of tokenizers are explored more in Section 6.1.
3 Methods
For each experiment, we compress long contiguous sequences of training data using different methods. For several, we use M1—a byte-level language model—as in the compression algorithm. We then chunk the compressed output into tokens and train M2 models over those tokens.
3.1 Training Data
All training data used is English web text from C4 (en 3.1.0) [52]. After tokenization, each document in C4 has an <EOS> token appended to it. We concatenate documents together to generate a long sequence of text. Using UTF-8 byte-level tokenization, the average document length is bytes, thus these long sequences have an average length of bytes. Despite the document breaks, we consider these long sequences “continguous” for the training of language models. These sequences are then split into individual examples, which are shuffled using the deterministic dataset functionality from SeqIO [54].
3.2 Training M1
The model used for compression is a decoder-only Transformer model [67]. It uses the m size seen in Table 4 and a context length of . We use a batch size of , an rsqrt decay learning rate schedule () starting at with warmup steps, and a z-loss of . The model is trained for steps using the Adafactor [59] optimizer. The feed-forward layers use ReLU activations [47, 22], and we use distinct learnable relative attention embeddings [58] at each layer. We use a deterministic SeqIO dataset and train using Jax [7], Flax [28], and T5X [54]. The final validation performance of the M1 model is bits/byte, a standard measure of perplexity, see Section 3.8. M1 and M2 are both trained on the C4 training data, but the final validation data used to evaluate M2 is unseen during M1 training, therefore there is no information leakage. This is similar to how LLM tokenizers are often trained on same dataset that the LLM is subsequently trained on.
3.3 Compression Methods
When compressing C4 training data, we use an example length of bytes and apply one of the following compression techniques (see Appendix H for more methods we considered). This results in compressed examples that are, on average, much longer than our target sequence length of M2 tokens. Thus, each example fills or nearly fills the model’s context window with a compressed sequence made from contiguous raw bytes. We compress examples using each method, allowing us to train each M2 model for steps without repeating data.
Arithmetic Coding: In this setting, we use a decoder-only transformer language model to model , that is, when creating the interval , the partitions for each possible character, , are calculated using the probabilities for the next token output by the transformer.
The compressor model is run over contiguous text sequences of bytes. The generated logits are used as the model distribution for Arithmetic Coding. We use the Range Encoding (a finite-precision implementation of Arithmetic Coding) implementation from TensorFlow Compression [4] with a precision of . The range encoding implementation uses integers with precision bits. This is enough to encode -bit float logits, so should not cause numerical issues as our models are trained using bfloat16. While the compressor model is only trained on sequences of length , it uses relative position embeddings in its attention layers. Thus, it can be applied to longer sequences. Some works observe decreased performance as inputs are scaled to lengths beyond those seen in training [66, 51], but we find that compression performance is similar in the two settings. Compressing sequences of length yields a compression ratio of while compressing sequences of length yields a ratio of . This suggests the performance drop from long sequences has minimal effect on compression, or that the increased contextual information makes up this difference.
We will see that text compressed in this straightforward manner is not readily learnable by M2. Thus, we explore alternative compression methods that modify the “modeling” and “coding” components for better learnability. Table 2 shows how our different approaches affect the compression ratio.
Static Logits Arithmetic Coding: One potential difficulty of learning over compressed text is that the “modeling” component of the compression algorithm is hard to learn—that is, the second language model (M2) has trouble learning to simulate the probabilities the compressor model (M1) assigns to bytes.
To weaken the compressor model, we replace the context-sensitive LM model with a static byte unigram model—that is, the model’s distribution is the same for all byte tokens in the input, i.e., . This distribution is estimated using the byte unigram statistics from the C4 training data.
Equal Information Windows: The difficulty in modeling compressed text could also be because the “coding” component of the compression algorithm is hard to learn. That is, the language model is not able to track the state variables used in Arithmetic Coding.
Our proposed method of weakening the coding component of Arithmetic Coding compression is to reset the AC encoder once it has output a set number of bits, creating windows of fixed size where each window is an independently AC-compressed sequence. This process is illustrated in Fig. 2. Windows will represent a variable amount of text, but as each window is created via compression, we expect roughly the same amount of information per window.
In addition to resetting the AC encoder, we also reset the M1 model’s context. This means that each bits of output can be decoded independently, at the cost of a weaker M1 model due to the lack of context. As each window is fully self-contained, the model no longer has to learn to track Arithmetic Coding state variables over long distances.
In cases where “spare bits” are available at the end of a window (but not enough to add an additional symbol of text), we pad with zeros. This complicates the decoding algorithm, but the compression scheme remains lossless. See Appendix I for further discussion and an alternative padding approach that gives similar results.
When compressing an additional character would result in a bitstream that is greater than bits long, i.e., more than binary expansions are needed to create an interval that is enclosed by , the bitstream (padded to bits as necessary) representing the input up to and including character is emitted. Then both the AC encoder and M1 model are reset. That is, is calculated as if ; the bit interval is also reset to . Similarly, M1 is only conditioned on inputs that are part of the current window, the inputs after . That is, .
We use to denote the bits per window, and for the vocabulary size of M2. For example, EqualInfoAC represents AC encoding with 16-bit Equal Info Windows and 8-bit M2 tokens (vocabulary ).
GZip: As a baseline, we also explore training over text compressed using GZip [17] as implemented in the Python [65] zlib library using the default compression level. GZip uses the DEFLATE algorithm—a combination of Huffman Trees [32] and LZ77 [77]. First LZ77 is used to replace repeated substrings in the text with pointers back to the original substring. Then a Huffman Tree is built for the current—LZ77 compressed—example and used to compress it. Note that this setting is dynamic, as the Huffman tree, and hence the binary codes for each character, are unique to the example. These experiments explore a setting where both the modeling and coding components of compression are different from Arithmetic Coding.
3.4 Tokenization of Compressed Text
Method | Token Compression Ratio | Bit Compression Ratio |
---|---|---|
SentencePiece | ||
AC | ||
AC |
Most compression methods output a bitstream, but training M2 directly over bits would not be ideal. As M1 was trained over UTF-8 bytes, the bit-level output of compression would result in M2 being applied to much longer sequences. Additionally, models are generally trained with vocabulary sizes much larger than two. Thus, we need a method to segment the bitstream into tokens, creating a more standard sequence for training language models.
We convert the bitstream into a token sequence by grouping every bits into a token—resulting in a vocabulary size of . We explore settings of {, }, resulting in vocabulary sizes of and . As the tokens are created from the compressed bitstream, we expect the distribution of tokens to be more uniform than the usual Zipfian [76] distribution of word or subword tokens, allowing us to use larger vocabularies without encountering issues of rare or unattested tokens.
Throughout this work, we focus on the “token compression ratio” —the ratio between the input and output token sequence lengths. It is important to note that the meaning of “token” can differ between the input and output sequences. Generally, the input sequence is one byte per token, while output tokens represent multiple bytes. This is in contrast to the more standard “bit compression ratio” —the ratio of input bits to output bits. As we aim to reduce the computational overhead of running LLMs by training them on compressed input, we are more concerned with reducing the number of tokens that M2 consumes. This difference is elucidated in Table 1. While SentencePiece results in a sequence length reduction of , the larger vocabulary means that bits are required to represent each token. As such, the bit compression ratio is only , which is much lower than our AC-based compressors. Similarly, creating -bit tokens from the output of Arithmetic Coding does not change the bit compression ratio—the total number of bits is unchanged—but it does reduce the number of tokens in the sequence, and thus the number of tokens the LLM must process. We compute compression ratios over the C4 dev set, which is unseen during M1 training.
To highlight the differences between the tokenization methods above, we measure the performance (as bits/byte on a sample of the C4 validation set) of two trivial models for each tokenizer in Table 3. The “uniform” model naïvely assigns equal probability to each token, regardless of context. The “unigram” model also ignores context, but assigns probabilities based on the global token frequencies observed in the training data. With byte-level tokenization, each UTF-8 byte encodes to a single -bit token, so the uniform model achieves bits/byte. For more powerful tokenizers, the uniform model is stronger, indicating that the tokenizer itself has some language modeling ability. We observe that our compression-based tokenizers (AC, EqualInfoAC and GZip) output a near-uniform distribution of tokens across their vocabulary. This is reflected in the near-zero gain over “uniform” achieved by modeling unigram statistics.
Method | Compression Ratio |
---|---|
AC | |
StaticAC | |
EqualInfoAC | |
EqualInfoAC | |
EqualInfoAC | |
EqualInfoAC |
Method | Uniform bits/byte | Unigram bits/byte | |
---|---|---|---|
Bytes | |||
SentencePiece | |||
AC | |||
StaticAC | |||
EqualInfoAC | |||
EqualInfoAC | |||
EqualInfoAC | |||
EqualInfoAC | |||
GZip |
3.5 Training M2 on Compressed Data
Each M2 model is trained for steps with a batch size of and a sequence length of . Thus each model trains on billion tokens. Of these, the vast majority (over %) are non-padding tokens; see Appendix C for details and Table 13 for the exact size of each dataset. As methods with higher compression ratios cover more raw text per token, we also include the total number of bytes in each dataset. Shuffling of training sets is seeded, and dataset state is checkpointed during training, so each training run results in the model seeing each example exactly once.
Models are trained at four sizes, as shown in Table 4, with m, m, m, and b parameters, excluding embedding parameters. When the compressed bitstream is chunked into 8-bit tokens, the M2 model has a vocabulary size of . With 16-bit tokens the vocabulary increases to . All M2 models have a sequence length of tokens. Thus, when training on 16-bit tokens, twice as many bytes are seen per example and in training overall, as compared to 8-bit tokens. All other hyperparameters match those used in M1.
Parameter Count | Embedding Dim | #Heads | #Layers | Head Dim | MLP Dim |
---|---|---|---|---|---|
m | |||||
m | |||||
m | |||||
m | |||||
b |
3.6 Baselines
We compare our M2 models against baseline models trained with two standard tokenization methods, described below. All hyperparameters, including sequence length (), match those used for our M2 training above.
Bytes
SentencePiece
3.7 Numerical Stability
Arithmetic Coding depends on the creation of “intervals” that cover each symbol in the vocabulary based on the quantized cumulative distribution of a model’s logits when predicting the next token. As such, a small change in the logits due to numerical noise can result in vastly different output bitstreams. This can make the practical use of neural language models in compression difficult. Common sources of noise include changes in batch size, parallel computation, changes to compute infrastructure (CPU vs. GPU vs. TPU, different TPU topology, etc.), changes to inference (computing the logits for the whole sequence at once vs. computing logits for a single token at a time using KV caches), and changes to the longest sequence length in the batch.
Methods like the rank-sorted algorithm used in LLMZip [63] may help alleviate these issues as only the order of tokens needs to match between settings. The development of alternate methods of LLM-based compression should keep numerical stability issues in mind and ideally alleviate these issues in the design of the algorithm. Increasing the level of quantization could also help reduce numerical noise issues, as differences would mostly be lost in quantization, but this would have a negative impact on the compression ratio.
3.8 Evaluation
As the tokenization scheme varies across the approaches we consider, models cannot be directly compared on “per-token” metrics such as negative log likelihood loss . Rather, following previous work [14, 2, 10, 24, et alia], we report perplexity in terms of “bits-per-byte”, , which scales the model’s loss by the token-level compression rate.
We also compare models on how much computation (FLOPs) is required to perform inference over a given length of raw text (bytes). More specifically, we calculate M2’s expected FLOPs/byte by scaling FLOPs/token—approximated by (excluding embedding parameters) following [35]—by the token-level compression rate (as tokens/byte). For methods using an M1 model during compression, the FLOPs/byte cost of M1 is added.101010While there is a computational cost to running GZip over the input text, we ignore it as it is insubstantial compared to the cost of running M2 model inference. For more details on the evaluation metrics see Appendix G.
We evaluate models on a sample of the C4 validation set. During evaluation, the model is run over batches or ~ million tokens. These tokens represent different amounts of text based on the compression method, making it impractical to run evaluation on the same sequence of bytes for all methods. To confirm that our validation samples are large enough to be representative, for each method, we train five m parameter models with different seeds. We find the final performance to be extremely stable, with the largest standard deviation in bits/byte being . Thus, the variance introduced from sampling the validation set is negligible. See Appendix B for more information about variance.
4 Results
Simple methods of training over neural-compressed text fail
As seen in Fig. 3, the most obvious approach—compression using Arithmetic Coding with M1 assigning next-token probabilities—fails to learn anything. Regardless of scale, the model only learns to output a uniform distribution over tokens, the performance of which is denoted by the dashed line. As the Arithmetic Coding procedure is near optimal [45], the compression ratio is essentially determined by the loss of M1. Thus, even though the M2 model learns nothing useful, when scaled by the compression rate, this setting ends up with the same performance as the M1 model. Similarly, models trained over data compressed with StaticAC—where M1 is replaced with a static unigram model—fail to learn. This result suggests that the difficultly in learning stems from the complexity or brittleness of the Arithmetic Coding process itself, rather than from M2’s inability to model M1. Note that the weak “modeling” component of this compression scheme results in a much lower compression rate and thus worse bits/byte performance, despite the model also learning a uniform distribution.
SentencePiece is a strong baseline
Our SentencePiece baseline outperforms all other methods, including our Bytes baseline, across all model sizes. On the surface, this result seems to run counter to the recent findings of [16], where their byte-level models outperformed subword (BPE) models at medium and large scales. The discrepancy is due to prioritizing different metrics. They report the model’s bit compression rate on fixed-length ( byte) sequences. While this is one type of “fair” comparison, it disadvantages subword models, as they are trained to model dependencies longer than bytes (but never evaluated on this ability), and are allotted fewer inference FLOPs to process the same text, as compared to the byte-level models. Additionally, bit compression ratio penalizes subword models for having larger vocabulary sizes. By contrast, our evaluation tests what perplexity models achieve on sequences of the same length they were trained on, and compares models at matching FLOPs/byte cost. This aligns with our end goal, which is to train an LLM that achieves the best perplexity at whatever sequence length it can handle, given a fixed budget for training and inference.
Equal-Info Windows make AC learnable
Fig. 3 shows that EqualInfoAC outperforms the byte-level baseline at most model sizes, with the gains increasing with scale. In addition to better bits/byte performance, training over compressed data has the advantage of using fewer FLOPs/byte for a given model size—seen in the leftward shift of the EqualInfoAC curve compared to the Bytes curve—due to shorter sequence lengths.
Using -bit tokens (k vocabulary) increases performance further. EqualInfoAC outperforms the Bytes baseline at all model sizes. It underperforms the SentencePiece baseline, but the gap diminishes with scale.
However, EqualInfoAC outperforms the SentencePiece baseline in terms of tokens/byte. Models using EqualInfoAC take fewer autoregressive steps to generate the same text than models using SentencePiece encoding. This has the potential to reduce generation latency, at the cost of reduced compute efficiency. This is a tradeoff that is often worth making in production. For instance, speculative decoding [42] is a popular approach that performs redundant computation in order to potentially accelerate auto-regressive steps.
It is noteworthy that the EqualInfoAC M2 models learn well despite being trained on data that has nearly uniform unigram statistics, as we saw in Table 3. In the best case, our billion parameter M2 model achieves bits/byte. This is a large gain over the naïve uniform ( bits/byte) and empirical unigram ( bits/byte) models from Table 3, and approaches the performance of a parameter-matched SentencePiece model ( bits/byte), despite using % fewer FLOPs/byte.
It is apparent from Fig. 3 that if FLOPs/byte were held constant, SentencePiece would achieve slightly better bits/byte than EqualInfoAC. However there is another axis along which EqualInfoAC may still be preferred. Setting aside inference FLOPs, all our SentencePiece models require % longer sequences to encode the same text when compared to our best EqualInfoAC setting (, k). This means that regardless of FLOPs used, the SentencePiece models will take more decoder steps at inference time. It is up to the practitioner whether it is “worth it” to trade off some bits/byte performance in order to achieve shorter sequences. In many serving scenarios, decoder steps are a practical bottleneck for determining system latency, and there are cases where one may be willing to incur extra inference cost to reduce latency (e.g., speculative decoding [43]). To this end, it may be advantageous to scale up an EqualInfoAC model to recover bits/byte performance while retaining the reduced latency. This can be seen visually in Fig. 4.
GZip is not competitive
Training over GZip-compressed text is relatively ineffective. M2’s performance when trained over GZip highlights a counter-intuitive trend. While the GZip M2 models actually learn, it would still be preferable to train over AC-compressed text—even though those models do not learn. This is due to the weak compression offered by GZip. The poor compression rate, coupled with weak learning, means that the GZip M2 models’ bits/byte performance lags behind even the m parameter M1 model.
Short windows are the best
We see a similar effect in Fig. 5, which ablates the EqualInfoAC window size. In terms of bits/byte, the shortest -bit windows perform the best. However, the next-best setting is the longest -bit windows, despite the fact that these M2 models fail to learn almost anything beyond the uniform distribution. This unintuitive trend stems from the fact that longer windows translate to better compression rates (see Table 2). If we remove the effect of compression rate by looking at bits-per-token (Fig. 5(b)), we see a clearer monotonic trend—increasing window length makes it harder to learn, as we move closer to simply running Arithmetic Coding over the whole sequence. For and -bit windows, performance improvements with scale are small, but present; see Table 10 for exact numbers.
Larger M2 vocabulary is helpful
Tokenizing compressed text using a larger -bit vocabulary (k) results in a higher token compression rate, seen in the leftward shift of each curve in Fig. 6.111111The same trend holds for larger and -bit windows, but the performance increase with scale is so slight that we omit them from the graph. See Table 10 for the exact values. For Arithmetic Coding methods, larger vocabulary also improves bits/byte, seen as a downward shift in the curves. However, for GZip, we see the opposite trend. Arithmetic Coding and GZip differ the most in their coding component, which suggests that the reason for this difference could lie there. Note that the header and footer present in GZip-compressed data do not explain this difference, see Appendix E. For EqualInfoAC, moving from to k results in each window corresponding to a single token, which increases the “stability” of the token text mapping. This could be one reason for the performance gain; see Section 6.1 for more discussion of “stability”.
Emergence with scale is unlikely
Given the recent findings of [55], we anticipate that continuing to scale models beyond billion parameters is unlikely to deliver an “emergent” ability to learn over AC-compressed text, since the bits/byte metric we use is smooth.
Results persist under “scaling laws” paradigm
When scaling models, [30] recommend that training tokens should be scaled linearly with model size. However, in our experiments above, all models see the same number of tokens, regardless of model size. Consequently, our largest models may be somewhat “undertrained”.121212The undertraining of our b models is also visible in their validation loss curves, which still have a significant decreasing slope at steps, showing the models have not yet converged. To test whether following the “scaling laws” recommendation influences our results, we reevaluate our models at earlier checkpoints selected to maintain a constant ratio of training data to model size. We find that all core trends are unchanged in this setting. See Appendix D for details.
5 Additional Experiments
At this point, we have established that while the simplest approaches to training over compressed text fail, there are alternate compression schemes that are learnable. In this section, we conduct additional experiments to shed light on which aspects of different compression methods are difficult to learn and what contributes to their learnability.
5.1 Bitstream tokenization is not the main source of difficulty
The compression algorithms we consider output a bitstream, which we later chunk into tokens of a fixed bit depth (e.g., -bit tokens). As such, it is common for the bits representing a single character or UTF-8 byte to be split across multiple tokens. Compounding this issue is that the value of these tokens are contextually determined and may differ depending on the surrounding bytes.
The fact that both -bit and -bit token chunking strategies work suggests that this is not too much of an issue for the model. To further investigate this, we train two models—one m and one m—on the raw bitstream output by Arithmetic Compression, i.e., each token is either a or a and the vocabulary has a size of . We use the same hyperparameters as in Section 3. Working at the bit level means that the output sequence is now longer than the input sequence, which was UTF-8 bytes. As such, this setting is not practical in the real world.
When trained to convergence, the two models have cross entropy losses of for the m parameter model and for the m model—not meaningfully better than the naïve uniform distribution, which yields a loss of . This failure mode is the same as in Fig. 3, which suggests that AC encoding itself is the main source of difficulty, as opposed to any issue around tokenization or vocabulary size.
5.2 Transformers struggle to learn Arithmetic Coding
Arithmetic Coding is a sequential algorithm that involves tracking multiple state variables as the input (byte) sequence is consumed. Each token in the output sequence represents multiple transformations of these variables, e.g., transformations when using -bit token chunking. Theoretically, only transformer layers are needed to have a computational path through the model layers that can process a sequence of tokens as a chain, where each token conditions on the previous one. While most of our transformers have the capacity to model these sequences—only our m model has fewer layers—we see in practice that the Arithmetic Coding algorithm is still difficult to learn.
Task | Accuracy | Cross Entropy |
---|---|---|
Decompression | ||
Byte Level LM | ||
Compression |
To directly diagnose the ability to track Arithmetic Coding, we format AC compression and decompression as sequence-to-sequence tasks. The input provides the model with the true text, so we expect a model that is able to learn Arithmetic Coding should achieve an accuracy of . We compress sequences of bytes using M1 and Arithmetic Coding.131313We use shorter raw text sequences to keep the final sequence length of inputs + targets manageable. We concatenate the bytes and AC output tokens to create the compression task. For the decompression task, we simply flip the order—AC output tokens first and then bytes. The target tokens (bytes or tokens) are shifted by the input vocabulary size, ensuring that they have distinct values. We use a decoder-only transformer as our model with a causal attention mask, i.e., even during the input sequence, future tokens are hidden from the model. We train models with m parameters. Loss, gradients, and evaluation metrics are only computed on the target tokens.
In the decompression task, the target tokens are bytes. By ignoring the inputs and just modeling the outputs, the decompression model can achieve decent performance without actually leveraging the input data. To control for this, we also train a byte-level language model baseline on the same sequence-to-sequence data, excluding the input tokens. If the decompression model is actually learning to decompress Arithmetic Coding, we would expect stronger performance than the byte-level baseline. As we see in Table 5, the baseline model, which does not see the input tokens, has the same performance as the decompression model.141414The slight gain is statistically insignificant, . Clearly, the models trained for decompression are not actually learning to do decompression.
The model trained for compression actually shows some signs of learning. Training a language model directly on the compressed output results in the model learning a uniform distribution over tokens, see Fig. 3. When the model is able to attend to the input text, we see that the performance in Table 5 is better than the uniform distribution (which would have a cross entropy loss of ). While this method shows some hope for the learnability of Arithmetic Coding, the need to include the input sequence negates the main advantage of compression, i.e., applying the model to a shorter sequence. Additionally, the compressor’s performance is far from the it should be able to achieve.
We also find training on these sequence-to-sequence datasets to be less stable than training on the language modeling datasets. In our experiments, large performance swings and divergence were relatively common.
5.3 Larger vocabulary helps beyond increasing the compression ratio
Our best results training over compressed text use EqualInfoAC with -bit windows and vocabulary size at either k (best) or (second-best). One clear advantage of the model is that it has a better token compression rate, so sees twice as much raw text during training. To assess whether its performance gain is due entirely to this advantage, we train a m parameter M2 model over the same dataset, but reduce its sequence length from . This model trains on half as many tokens, but sees the same amount of underlying text as the model.151515To compensate for the smaller number of tokens in a sample of batches from validation set when each example is tokens, we compute our evaluation metrics over batches.
Table 6 shows that even in this setting, the model with larger vocabulary is stronger.161616It may be possible to achieve further gains by increasing the token bit depth further. However, most deep learning frameworks do not support using unsigned data types for inputs, and the resulting large vocabulary size can cause a computational bottleneck in the final softmax layer. In fact, most of the bits/byte gain (% absolute) is due to the structural change in tokenization, as opposed to the additional text seen. One possible explanation for its strong performance is that the model uses exactly one token to represent each equal-info window. We’ll see in the next section that in EqualInfoAC settings with multiple tokens per window, any non-initial tokens are highly context-dependent, and learning proceeds on a curriculum from the “easy” window-initial tokens to the “harder” window-final tokens.
Tokenization | Comparison | Bits/Byte |
---|---|---|
EqualInfoAC | ||
EqualInfoAC | byte matched | |
EqualInfoAC | token matched |
6 Analysis
In this section we examine how neural compression based tokenizers differ from standard tokenizers, and conduct additional analysis on training dynamics and learnability of compressed data. This analysis leads us to several recommendations for future work developing new compression schemes that aim to be learnable by transformer models while delivering stronger compression than subword tokenizers.
6.1 EqualInfoAC is less stable and less semantic than SentencePiece
Input Text | The three currently living species are: African savanna elephants, African forest elephants, and the Asian elephants. |
---|---|
SentencePiece Tokens | [The] [ three] [ currently] [ living] [ species] [ are] [:] [ African] [ ] [s] [a] [v] [anna] [ elephant] [s] [,] [ African] [forest] [ elephant] [s] [,] [ and] [ the] [ Asian] [ elephant] [s] [.] |
EqualInfoAC [b=16, v=65k] Tokens | [The th] [ree c] [urrently l] [iving ] [species] [ are] [: A] [frica] [n sav] [anna] [ ele] [pha] [nts, ] [Afr] [ican ] [forest ] [eleph] [ants, ] [and the ] [Asi] [an e] [lep] [hant] [s.] |
While the performance of our EqualInfoAC model approaches that of our SentencePiece baseline, qualitative analysis shows that the two tokenization schemes differ in many regards. Table 7 illustrates some of these differences.
First, we observe that SentencePiece produces a relatively stable text token mapping.171717See Appendix L for some corner cases where this is not the case. For example, “elephants” appears three times in the sentence, and maps stably to the same two-token sequence in all cases: [ elephant] [s]. Similarly, both occurrences of “African” map to the same token: [ African]. By comparison, the EqualInfoAC tokenization is relatively unstable, with each occurrence of these words being segmented in a different way and yielding a different token sequence.
Second, we find that the SentencePiece tokenization is more “semantic”, by which we mean that the segmentation it induces aligns better with meaningful linguistic units—words and morphemes. While there are some exceptions, e.g. “savanna” being parsed as [s] [a] [v] [anna], the more common case is whole words being parsed as single tokens (e.g., currently), or into meaningful morphemes (e.g., elephant-s). By comparison, EqualInfoAC tokenization appears to almost entirely disregard word and morpheme boundaries. As one example, we see “Asian elephants.” parsed as [Asi] [an e] [lep] [hant] [s.].
Despite these differences, there is an important similarity between SentencePiece and EqualInfoAC: they are both stable in the token text direction. That is, a given token ID, e.g., token #, will always map to the same output text. This “transparent decoding” property likely makes it easier for a downstream model to learn over these tokens.181818Padding to reach a specific window size can require extra computation to discern between padding and characters that compress to all zeros, however we find in Appendix I that it is not an issue for M2 models.
Token | Window Position | Window Text |
---|---|---|
[lew ] / [lea] / [led] / [len] / [less] / [led] / [les] / [lew ] | ||
[thoug] / [ust] / [ this] / [etti] / [npo] / [thoug] / [ un] / [imag] | ||
[ord a] / [or k] / [ord] / [or f] / [or al] / [or a ] / [ore i] / [ora] | ||
[ery] / [s may] / [cian] / [onte] / [h de] / [cri] / [opp] / [ides] |
When we move to versions of EqualInfoAC that contain multiple tokens per window, such as EqualInfoAC, this transparency is destroyed for all non-initial tokens within a window. This is illustrated in Table 8. When the same token appears window-initially in different contexts, we see the window text has a stable prefix—e.g., token # always maps to the prefix “le-”. However, when occurring as the second token within a two-token window, there are no apparent correspondences between window text.191919A repeated text substring that happens to be aligned with a window multiple times is one of the few cases where the second token will represent the same text. As EqualInfoAC window length increases, the proportion of tokens that are stable decreases. This may explain the observed difficulty of learning over longer windows. The window text for all instances of these tokens can be seen in Appendix M.
Note that Table 8 examines window text, as opposed to token text correspondences. This is because for multi-token windows, the mapping from tokens to text is not well defined. More specifically, each character maps to a particular subsequence of the compressed bitstream, but these may not align with token boundaries.202020This can be a source of instability, even in window-initial tokens, see Appendix L. Fig. 7 illustrates the mapping between characters, bits, and tokens. We find that many windows contain a character (shown in purple) whose bits are split across two -bit tokens.
Fig. 7 also highlights that window-initial characters are not being well compressed, with the window-initial token often only covering one or two characters. This is due to our EqualInfoAC procedure fully resetting M1’s context at every window boundary. With no context, M1 cannot make confident predictions, leading to more bits being needed to represent the initial character. On the positive side, this setup guarantees that a window can be decoded in isolation, which should aid learning. However it is worth exploring in future work whether maintaining some M1 context across windows could improve the compression ratio without hurting learnability.
6.2 AC decoding is learned step-by-step
As Arithmetic Coding is a sequential (left-to-right) and contextual algorithm, the text represented by a given token will differ based on the previous token. As such, a model should perform better on a token if it has a strong understanding of the token before it. When using EqualInfoAC compression, each window represents an independent Arithmetic Coding document. As we move deeper into the window, more and more AC decompression must be done to understand the token.
To understand how a token’s position within a window affects learning, we track across training the average accuracy at each position within the -token windows of a m parameter EqualInfoAC model.212121The absolute accuracy of the EqualInfoAC model is relatively poor, but its relatively long window provides the clearest illustration of the positional trends. We observe similar trends for shorter windows where the model has stronger performance. Fig. 8 shows both raw accuracy (left) as well as the increase over “trivial” accuracy (right), which we define as the maximum accuracy achieved in the first steps of training. Looking at accuracy increase highlights the “sequential learning” trend by discounting any part of accuracy that is text independent. In particular, we note that window-final tokens have a non-uniform distribution due to the use of window-final padding bits (see our EqualInfoAC formulation in Section 3.3), which can be learned without any understanding of the text.
We observe two interesting trends. First, there is a clear ordering as to when the model starts to make meaningful (non-trivial) progress on a given position. The initial token (#1) is learned first, followed fairly quickly by #2 and then #3. Later tokens are only “unlocked” after training steps, suggesting that the ability to model these tokens builds on a foundation of understanding the preceding tokens within the window.
The second trend concerns the accuracy reached at each position. Here, we observe an increase in accuracy from #1 < #2 < #3, followed by a decrease from #3 < #4 < #5 and so on.222222The final token #8 also fits this trend when looking at the increase over non-trivial accuracy. The raw accuracy in this position is higher than previous tokens #4–7, due to the skewed distribution introduced by window-final padding. We interpret the increase across the first three positions as due to the benefit of extra leftward context. This is akin to the initial byte in a word being harder to predict than the following bytes. The decreasing performance at tokens #4 and beyond suggests the model is unable to track AC decompression indefinitely. While the model clearly learns to decompress longer sequences as training progresses, reliably decoding past bits of AC output appears to be a challenge.
6.3 Learnable distributions are less uniform
A well-known result in the compression literature is that there can be no recursive compression [45]. The compression algorithm removes information captured by its model, resulting in a uniform output that appears random to the original model. However, our setting is not recursive compression. Instead, a separate and larger model is trained on the compressed output, which should be able to capture new patterns in the bitstream.
Despite this, the output of compression using M1 appears very uniform, as evidenced by the minimal gains from modeling the unigram token distribution in Table 3. Therefore, it seems reasonable that this uniformity could make it hard for M2 to learn (as all patterns must be contextual). We investigate this by plotting the KL divergence [39] between the observed empirical distribution and a uniform distribution for different segmentations of the bitstream. If the underlying distribution of bits was truly random and independent, then the distribution of unigrams for some bitstream segmentation should remain uniform as and therefore the KL divergence should remain close to zero. On the other hand, if the distribution diverges from uniform, there is contextual information to be learned when training an LLM to model .
We segment the bitstream either into bit n-grams, where successive n-grams are allowed to overlap, or into n-bit tokens, following our M2 tokenization procedure—see Section 3.4. We only plot tokenization into -bits that are factors of , otherwise tokens would cross window boundaries in the EqualInfoAC setting.
As a baseline, we used the cryptographic secrets package in Python to generate bitstreams that should be truly random and independent. As such, the KL divergence should remain at when segmented in the same way as the compressed data. The reason this does not hold in Fig. 9 is that the maximum likelihood estimate of entropy, , is negatively biased [48]. In Fig. 13 we see that when using a Miller-Madow estimator [46] to correct for this bias, the expected KL of is well within sampling noise bounds. To account for noise in the entropy estimation, we plot th percentile intervals of the KL divergence between the observed entropy from disjoint samples of the data and the uniform distribution.232323As the number of bits in a segmentation grow, the vocabulary size increases exponentially, requiring many more samples. Thus we expect noise in the entropy estimate to grow with . This holds, but it is obfuscated by the log scaling in Fig. 9. In fact, the magnitude of the noise for settings such as GZip and EqualInfoAC is larger than for AC or RNG. This noise behavior is seen in Fig. 12. See Appendix J for more information on entropy estimation and bias correction.
The AC and RNG lines in Fig. 9 are very similar and their sampling noise intervals have large overlaps. This suggests that the data generated by AC compression with M1 is difficult to distinguish from random data.242424For , the AC entropy is statistically significantly less than the RNG entropy, however, differences in the mean entropy only start to appear after ~ decimal places. This is a possible explanation for why M2 models trained on AC data only learn to output a uniform distribution, as seen in Fig. 3.
In Fig. 9, we see that GZip is the least uniform, which is expected as it has the worst compression rate among these settings. However, the segmentation into tokens does not result in much extra information. This is again suggestive that the differences between the “coding” components of GZip and Arithmetic Coding are important for learnability. It is also a possible explanation of why GZip is the one setting where using -bit tokens does not improve performance.
Similarly, Fig. 9 shows that EqualInfoAC has the most information among the Arithmetic Coding approaches. Given that this is the most learnable setting, it suggests that non-uniformity of the bitstream may be important for learning. We also see a large increase when moving to -bit tokens, providing a further possible explanation for why larger vocabulary is helpful (see Section 5.3). Finally, we note that StaticAC has less information than EqualInfoAC, suggesting that weakening the “coding” component of Arithmetic Coding is a more effective way to retain information and increase learnability for M2.
7 Conclusion
We have shown there is promise in the idea of training LLMs over neural-compressed text. In the best case, this will allow training over text that is better compressed than standard subword token sequences, while maintaining learnability. This an appealing prospect, as models that read and write more text per token are more efficient to train and serve, and can model longer dependencies.
While the “very simplest” approach does not work (training directly over a tokenized AC-encoded bitstream), we showed that a relatively simple modification—compression via Equal Info Windows—already brings us within striking distance of popular tokenizers. When measured in terms of perplexity achievable at fixed inference cost (FLOPs/byte), we find that our method outperforms raw byte-level models, and comes increasingly close to the performance of SentencePiece tokenization as scale increases to billion parameters.
While bespoke compression methods have developed around different modalities (e.g., text, audio, images, video) and different applications (e.g., delta-of-delta for regular repeating timestamps [50]), to our knowledge, no efficient compression methods have been designed specifically for use as LLM tokenizers. We are optimistic that future work will create such methods. Compared to today’s subword tokenizers, we expect these methods (i) will deliver higher compression rates, (ii) will come closer to equal information per token, thus allocating compute more effectively, and (iii) will give models a more direct view of the underlying raw text, thus helping on spelling and pronunciation tasks. As a tradeoff, we expect these neural tokenizers will be somewhat less stable in their text token mapping, but perhaps not so unstable as our approach here. In particular, we think it is worth exploring methods under which a given word typically maps to a relatively small number (tens not thousands) of relatable token sequences.
One direction we left unexplored is the idea of passing information between the compressing model (M1) and the LLM trained over compressed text (M2). Some additional signal of M1’s internal state or output may be helpful for M2 to accurately simulate M1, which is a prerequisite to flawlessly encoding and decoding M1-compressed text.
For hill-climbing in this space, we found it useful to iterate on the sequence-to-sequence sub-tasks of compression and decompression, which should, in theory, be learnable with high accuracy. Specifically, if future work can devise a strong (~) compressor that a transformer can be trained to accurately encode and decode, we expect that this will be an ideal candidate for tokenizing text for LLMs.
Acknowledgements
We would like to thank Ben Adlam, Grégoire Delétang, Rosanne Liu, and Colin Raffel for detailed comments on an earlier draft. We’re also grateful to Peter Liu, both for helpful discussion, as well as for building some of the infrastructure that made our experiments easier to run. Finally, we thank Doug Eck, Noah Fiedel, and the PAGI team for ongoing guidance and feedback.
References
- [1] J. Ainslie, S. Ontanon, C. Alberti, V. Cvicek, Z. Fisher, P. Pham, A. Ravula, S. Sanghai, Q. Wang, and L. Yang. ETC: Encoding Long and Structured Inputs in Transformers. In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268–284, Online, Nov. 2020. Association for Computational Linguistics.
- [2] R. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones. Character-Level Language Modeling with Deeper Self-Attention. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):3159–3166, Jul. 2019.
- [3] A. Baevski, H. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NeurIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
- [4] J. Ballé, S. J. Hwang, and E. Agustsson. TensorFlow Compression: Learned Data Compression, 2024.
- [5] I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The Long-Document Transformer, Dec. 2020.
- [6] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour. AudioLM: A Language Modeling Approach to Audio Generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523–2533, 2023.
- [7] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
- [8] P. F. Brown, V. J. D. Pietra, S. A. D. Pietra, and R. L. Mercer. The Mathematics of Statistical Machine Translation: Parameter Estimation. Comput. Linguist., 19(2):263–311, jun 1993.
- [9] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating Long Sequences with Sparse Transformers, Apr. 2019.
- [10] D. Choe, R. Al-Rfou, M. Guo, H. Lee, and N. Constant. Bridging the Gap for Tokenizer-Free Language Models. CoRR, abs/1908.10322, 2019.
- [11] J. Chung, S. Ahn, and Y. Bengio. Hierarchical Multiscale Recurrent Neural Networks. In International Conference on Learning Representations, 2017.
- [12] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu. w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250, 2021.
- [13] J. H. Clark, D. Garrette, I. Turc, and J. Wieting. Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Transactions of the Association for Computational Linguistics, 10:73–91, 2022.
- [14] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- [15] S. DeDeo, R. X. D. Hawkins, S. Klingenstein, and T. Hitchcock. Bootstrap Methods for the Empirical Study of Decision-Making and Information Flows in Social Systems. Entropy. An International and Interdisciplinary Journal of Entropy and Information Studies, 15(6):2246–2276, 2013.
- [16] G. Delétang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness. Language Modeling Is Compression. In ICLR, 2024.
- [17] P. Deutsch. GZIP file format specification. RFC 1952, May 1996.
- [18] P. Deutsch and J.-L. Gailly. ZLIB Compressed Data Format Specification. RFC 1950, May 1996.
- [19] J. Duda. Asymmetric Numeral Systems: Entropy Coding Combining Speed of Huffman Coding with Compression Rate of Arithmetic Coding, Jan. 2014.
- [20] B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1):1–26, 1979.
- [21] J. H. Friedman. Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics, 29(5):1189–1232, 2001.
- [22] K. Fukushima. Cognitron: A Self-Organizing Multilayered Neural Network. Biological Cybernetics, 20:121–136, 1975.
- [23] P. Gage. A New Algorithm for Data Compression. C Users Journal, 12(2):23–38, 1994.
- [24] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling, Dec. 2020.
- [25] N. Godey, R. Castagné, É. de la Clergerie, and B. Sagot. MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2859–2870, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
- [26] O. Goldman, A. Caciularu, M. Eyal, K. Cao, I. Szpektor, and R. Tsarfaty. Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance, Mar. 2024.
- [27] A. Graves. Adaptive Computation Time for Recurrent Neural Networks, Feb. 2017.
- [28] J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee. Flax: A neural network library and ecosystem for JAX, 2020.
- [29] G. Hinton, O. Vinyals, and J. Dean. Distilling the Knowledge in a Neural Network, Mar. 2015.
- [30] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre. Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems, 2022.
- [31] P. G. Howard and J. S. Vitter. Analysis of Arithmetic Coding for Data Compression. Information Processing & Management, 28(6):749–763, 1992.
- [32] D. A. Huffman. A Method for the Construction of Minimum-Redundancy Codes. Proceedings of the IRE, 40(9):1098–1101, 1952.
- [33] J. D. Hunter. Matplotlib: A 2D Graphics Environment. Computing in Science & Engineering, 9(3):90–95, 2007.
- [34] Z. Jiang, M. Y. R. Yang, M. Tsirlin, R. Tang, and J. Lin. Less is More: Parameter-Free Text Classification with Gzip, Dec. 2022.
- [35] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling Laws for Neural Language Models, Jan. 2020.
- [36] N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The Efficient Transformer. In International Conference on Learning Representations, 2020.
- [37] T. Kudo. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia, July 2018. Association for Computational Linguistics.
- [38] T. Kudo and J. Richardson. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium, Nov. 2018. Association for Computational Linguistics.
- [39] S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79–86, 1951.
- [40] M. O. Külekci. Compressed Context Modeling for Text Compression. In 2011 Data Compression Conference, pages 373–382, 2011.
- [41] B. Lester, J. Yurtsever, S. Shakeri, and N. Constant. Reducing Retraining by Recycling Parameter-Efficient Prompts. arXiv preprint arXiv:2208.05577, aug 2022.
- [42] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding, 2023.
- [43] Y. Leviathan, M. Kalman, and Y. Matias. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 23–29 Jul 2023.
- [44] R. Liu, D. Garrette, C. Saharia, W. Chan, A. Roberts, S. Narang, I. Blok, R. Mical, M. Norouzi, and N. Constant. Character-Aware Models Improve Visual Text Rendering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16270–16297, Toronto, Canada, July 2023. Association for Computational Linguistics.
- [45] M. Mahoney. Data Compression Explained. 2013.
- [46] G. Miller. Note on the Bias of Information Estimates. In Information Theory in Psychology: Problems and Methods, pages 95–100. Free Press, 1955.
- [47] V. Nair and G. E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, pages 807–814, Madison, WI, USA, 2010. Omnipress.
- [48] L. Paninski. Estimation of Entropy and Mutual Information. Neural Computation, 15(6):1191–1253, June 2003.
- [49] R. Pasco. Source Coding Algorithms for Fast Data Compression (Ph.D. Thesis Abstr.). IEEE Transactions on Information Theory, 23(4):548–548, 1977.
- [50] T. Pelkonen, S. Franklin, J. Teller, P. Cavallaro, Q. Huang, J. Meza, and K. Veeraraghavan. Gorilla: a Fast, Scalable, In-Memory Time Series Database. Proc. VLDB Endow., 8(12):1816–1827, aug 2015.
- [51] O. Press, N. Smith, and M. Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In International Conference on Learning Representations, 2022.
- [52] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research (JMLR 2020), 21(140):1–67, 2020.
- [53] J. J. Rissanen. Generalized Kraft Inequality and Arithmetic Coding. IBM Journal of Research and Development, 20(3):198–203, 1976.
- [54] A. Roberts, H. W. Chung, G. Mishra, A. Levskaya, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, C. Hawthorne, A. Lewkowycz, A. Salcianu, M. van Zee, J. Austin, S. Goodman, L. B. Soares, H. Hu, S. Tsvyashchenko, A. Chowdhery, J. Bastings, J. Bulian, X. Garcia, J. Ni, A. Chen, K. Kenealy, K. Han, M. Casbon, J. H. Clark, S. Lee, D. Garrette, J. Lee-Thorp, C. Raffel, N. Shazeer, M. Ritter, M. Bosma, A. Passos, J. Maitin-Shepard, N. Fiedel, M. Omernick, B. Saeta, R. Sepassi, A. Spiridonov, J. Newlan, and A. Gesmundo. Scaling Up Models and Data with t5x and seqio. Journal of Machine Learning Research, 24(377):1–8, 2023.
- [55] R. Schaeffer, B. Miranda, and S. Koyejo. Are Emergent Abilities of Large Language Models a Mirage? In Thirty-Seventh Conference on Neural Information Processing Systems, 2023.
- [56] R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, Aug. 2016. Association for Computational Linguistics.
- [57] C. E. Shannon. A Mathematical Theory of Communication. The Bell System Technical Journal, 27:379–423, 1948.
- [58] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
- [59] N. Shazeer and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, July 2018.
- [60] S. Stanton, P. Izmailov, P. Kirichenko, A. Alemi, and A. G. Wilson. Does Knowledge Distillation Really Work? 2021.
- [61] Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. In International Conference on Learning Representations, 2022.
- [62] D. Tito Svenstrup, J. Hansen, and O. Winther. Hash Embeddings for Efficient Word Representations. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
- [63] C. S. K. Valmeekam, K. Narayanan, D. Kalathil, J.-F. Chamberland, and S. Shakkottai. LLMZip: Lossless Text Compression using Large Language Models, June 2023.
- [64] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural Discrete Representation Learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NeurIPS’17, page 6309–6318, Red Hook, NY, USA, 2017. Curran Associates Inc.
- [65] G. Van Rossum and F. L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009.
- [66] D. Varis and O. Bojar. Sequence Length is a Domain: Length-based Overfitting in Transformer Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8246–8257, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
- [67] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc., 2017.
- [68] L. Vilnis, Y. Zemlyanskiy, P. Murray, A. T. Passos, and S. Sanghai. Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 35120–35136. PMLR, 23–29 Jul 2023.
- [69] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020.
- [70] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-Attention with Linear Complexity, 2020.
- [71] M. L. Waskom. Seaborn: Statistical Data Visualization. Journal of Open Source Software, 6(60):3021, 2021.
- [72] B. L. Welch. The Generalization of “Student’s” Problem when Several Different Population Variances are Involved. Biometrika, 34(1/2):28–35, 1947.
- [73] I. H. Witten, R. M. Neal, and J. G. Cleary. Arithmetic Coding for Data Compression. Communications of The Acm, 30(6):520–540, June 1987.
- [74] L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022.
- [75] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33, 2020.
- [76] G. K. Zipf. The Psycho-Biology of Language. Houghton Mifflin, 1935.
- [77] J. Ziv and A. Lempel. A Universal Algorithm for Sequential Data Compression. IEEE Transactions on Information Theory, 23(3):337–343, 1977.
- [78] A. Zvonkin and L. Levin. The Complexity of Finite Objects and the Development of the Concepts of Information and Randomness by Means of the Theory of Algorithms. Russian Mathematical Surveys, 25:83, 10 2007.
Appendix A Numerical Values
Dataset | Size | bits/byte | FLOPs/byte |
---|---|---|---|
Bytes | m | M | |
m | M | ||
m | M | ||
b | M | ||
uniform | - | ||
SentencePiece | m | M | |
m | M | ||
m | M | ||
b | M | ||
uniform | - | ||
AC | m | M | |
m | M | ||
m | M | ||
b | M | ||
uniform | - | ||
AC | m | M | |
m | M | ||
m | M | ||
b | M | ||
StaticAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
uniform | - | ||
StaticAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
uniform | - | ||
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
GZip | m | M | |
m | M | ||
m | M | ||
b | M | ||
uniform | - | ||
GZip | m | M | |
m | M | ||
m | M | ||
b | M |
Dataset | Size | bits/byte | FLOPs/byte |
---|---|---|---|
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M | ||
EqualInfoAC | m | M | |
m | M | ||
m | M | ||
b | M |
Dataset | Compression Ratio | Size | bits/byte | FLOPs/byte |
---|---|---|---|---|
EqualInfoAC | m | M | ||
m | M | |||
m | M | |||
b | M | |||
EqualInfoAC Zero | m | M | ||
m | M | |||
m | M | |||
b | M | |||
EqualInfoAC | m | M | ||
m | M | |||
m | M | |||
b | M | |||
EqualInfoAC Zero | m | M | ||
m | M | |||
m | M | |||
b | M |
Appendix B Variance
Sampling from the validation set was seeded. For a given seed, the same batches are sampled at each evaluation step within a training run. Similarly, when models of a different size are trained on the same compressed data, the same evaluation batches are sampled, allowing for fair comparison. As the Bytes and SentencePiece baselines use deterministic datasets, the validation seed is not used. Instead the “start_step” is incremented by to get a new sample of batches.
Model initialization and the order of the training data is controlled by the training seed. This seed was also changed during variance testing. During training, the dataset is checkpointed and therefore each example is seen exactly once. The exact order of the training data is determined by the seed. As the Bytes and SentencePiece baselines use deterministic datasets, the training order is fixed.
models with m parameters were trained with different seeds (both validation and training) for each compression method and the two baselines. The mean and standard deviation can be found in Table 12. The variance is so low that we only report single values for most other experimental settings, such as larger models.
Method | bits/byte |
---|---|
Bytes | |
SentencePiece | |
AC | |
StaticAC | |
EqualInfoAC | |
EqualInfoAC | |
EqualInfoAC | |
EqualInfoAC | |
GZip |
Training models of size m and b over data compressed with EqualInfoAC and EqualInfoAC, as well as a b model with EqualInfoAC, occasionally diverged, collapsing to a simple model that just output the uniform distribution. The numbers for these settings exclude these divergent runs. This resulted in re-runs in the most problematic case.
Appendix C The Amount of Raw Text Bytes Seen by M2
Table 13 shows the number of tokens and bytes found in the training dataset for each compression method. During the data generation process, sequences of —generated by concatenating C4 byte-tokenized documents together—are compressed. Some of these sequences, namely the final sequence created from the tail of the concatenated docs, are too short to be compressed to the target length of . Thus, the exact number of tokens in the dataset can vary slightly. With no padding, each dataset would have been trained on tokens, we see all settings are close to this value, with the maximum deviation being EqualInfoAC with fewer tokens. All compression datasets are created from the same source sequences, thus the underlying byte sequences compressed by weaker methods are prefixes of the underlying sequences compressed by stronger methods.
Method | Compression Ratio | Tokens | Bytes |
---|---|---|---|
Bytes | |||
SentencePiece | |||
AC | |||
StaticAC |