The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
What does cross-entropy actually measure, and why is it connected to compression? It measures how much probability a model assigns to outcomes that actually occur: surprising outcomes cost more, predictable ones cost less. Averaged over data, that cost is the model’s expected coding length under an idealized code. This makes cross-entropy a useful measure of predictive fit—not a standalone measure of intelligence.
How cross-entropy scores a prediction
For a single observed outcome, a model’s cost is the negative logarithm of the probability it assigned to that outcome. If the model gives an event probability 1/8, its idealized code length is -log₂(1/8) = 3 bits. An event assigned probability 1/2 costs 1 bit; one assigned probability 1/16 costs 4 bits. A model is penalized more when the outcome it encounters was less likely according to its predictions.
Let P be the distribution that generates examples and Q the distribution predicted by a model. For discrete outcomes, cross-entropy is the expected negative log probability assigned by Q to outcomes drawn from P:
H(P,Q) = Ex∼P[-log₂ Q(x)]
With base-2 logarithms, the result is in bits. Using natural logarithms gives nats instead. Cross-entropy is an average over a distribution or dataset, not the cost of any one prediction.
#1 Best Overall
From a class label to a sequence
In classification, the target is often represented as a one-hot label: the observed class is the correct outcome, and the loss is the negative log of the probability assigned to that class. In autoregressive language modeling, the same calculation is applied repeatedly to next-token predictions. The model’s probability for a sequence is factorized into conditional probabilities given the preceding context; summing their negative logarithms gives the sequence’s model-based cost.
Reported averages depend on what is being counted. A language-model result may be normalized per token or per character, and those denominators are not interchangeable. Tokenizers divide text differently, while the logarithm base determines whether the loss is in bits or nats. A loss value needs its unit and averaging convention to be interpretable.
Why cross-entropy includes a mismatch penalty
Cross-entropy is related to the source’s own uncertainty by a precise identity:
H(P,Q) = H(P) + DKL(P || Q)
H(P) is the entropy of the source: its irreducible uncertainty under the distribution P. DKL(P || Q), the Kullback–Leibler divergence, is the extra expected cost of using model Q rather than the source distribution itself. Because this divergence is nonnegative, cross-entropy is at least the source entropy. It reaches that minimum when Q matches P almost everywhere on outcomes the source can produce.
This is why cross-entropy is not simply “the entropy of the text.” The source entropy is generally not known; a model’s measured cross-entropy also reflects how well that particular model matches the data. Better probability assignments reduce the mismatch penalty.
How prediction becomes lossless compression
A probability model can guide an entropy coder, such as arithmetic coding. The coder uses the model’s probabilities to turn a sequence into a bitstream. With suitable coding and shared decoding procedures, the bitstream can be made to approach the model’s summed negative log probability in length, subject to implementation details and message-termination overhead.
The decoder must have the model and coding procedure needed to interpret the stream, and it reconstructs the original sequence exactly. This is lossless compression—not summarization, paraphrasing, or deletion of information. If the model assigns higher probability to the sequence that actually occurs, its idealized code is shorter.
Why model loss is not the same as file size
Lower held-out cross-entropy means shorter idealized codes on average only when the text, tokenization, context protocol, probability convention, and normalization are held constant. An actual compressed file can also include framing, coder overhead, and information needed to identify or transmit the model. The model itself may be shared in advance or may need to be sent. Consequently, a reduction in language-model loss does not automatically translate into an equal reduction in an end-user’s file size.
What language-model loss tells you—and what it does not
Training a language model commonly means minimizing negative log-likelihood, the same basic quantity as cross-entropy loss over observed next tokens. Evaluation on held-out examples estimates how well the model assigns probabilities to data from that evaluation distribution. A lower score is evidence of better average prediction under that setup; it does not, by itself, establish broader reasoning ability or intelligence.
OpenAI’s January 23, 2020 publication, “Scaling laws for neural language models”, studies empirical scaling relationships for cross-entropy loss and reports that loss varied with model size, dataset size, and training compute. That makes cross-entropy a useful performance measure in scaling research, but does not make it a complete account of what a model can do.
Perplexity is another way to express average language-model loss: it is commonly calculated by exponentiating average natural-log loss, or by raising 2 to average loss in bits. Comparisons need to specify the convention. They also need the evaluation corpus and split, tokenization or character unit, context protocol, sequence-boundary handling, and whether the data are in-distribution or shifted. Without those details, two scores may not be comparable.
Recommended Free Tools
What published entropy-rate estimates mean
A 2020 paper by Takata, Kaji, and Utsuro, “Cross Entropy of Neural Language Models at Infinity—A New Bound of the Entropy Rate,” reports an English entropy-rate estimate of 1.12 bits per character. The authors obtained it by extrapolating the effects of training-data size and context length toward infinity using neural language models. It is a model-based extrapolation, not a universal constant or a directly measured file-compression rate.
In the paper’s specific model and dataset experiments, the authors also report observed minimum cross-entropies of 1.21 bits per character for English and 4.43 bits per character for Chinese. These are historical experimental results tied to that study’s setup; they should not be treated as general values for all text, models, or languages.
How to compare cross-entropy results fairly
To tell whether one probability model assigns better predictions than another, compare them under the same conditions:
- Data: Use the same evaluation corpus and split, and identify whether it is in-distribution or shifted.
- Representation: Match tokenization, or use the same character or byte definition. Per-token results from different tokenizers do not share a denominator.
- Protocol: Match available context, sequence boundaries, and handling of the first prediction in a sequence.
- Units and averaging: State the logarithm base and whether the result is averaged per token, character, byte, or sequence.
- Evidence type: Distinguish measured results from extrapolated estimates.
For compression comparisons, also account for the coder, framing, model overhead, and any side information sent to the decoder. Cross-entropy describes expected model-based coding cost; a real compression ratio describes a complete encoding setup.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
