iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Entropy measures the average uncertainty in a probability distribution; cross-entropy measures how costly it is to use one distribution’s probabilities to describe outcomes generated by another. Their relationship is exact: cross-entropy equals entropy plus a nonnegative mismatch penalty called KL divergence.
A small example: scoring a known class
Suppose the true class is definitely A. Represent that target with a distribution p that assigns probability 1 to A. A model’s predicted distribution q assigns A probability 0.8 in one case and 0.1 in another.
| Model probability for A | Cross-entropy for this target | Interpretation |
|---|---|---|
| q(A) = 0.8 | −log₂(0.8) ≈ 0.322 bits | The model gave the observed class substantial probability. |
| q(A) = 0.1 | −log₂(0.1) ≈ 3.322 bits | The model assigned the observed class much less probability, so its score is worse. |
These values are direct calculations from the definition, not measurements from a model benchmark. This example also shows why cross-entropy is not the uncertainty of the target alone: it depends on the probabilities assigned by q.
What entropy measures
For a discrete random variable X with probability mass function p(x), Shannon entropy is the expected surprisal of its outcomes:
#1 Best Overall
H(X) = −Σₓ p(x) log₂ p(x)
The surprisal of one outcome x is −log₂ p(x). A rare outcome has greater surprisal than a common one; entropy averages surprisal across outcomes according to their probabilities. It describes a distribution, not the information content of one particular observation.
With base-2 logarithms, entropy is measured in bits. Natural logarithms give nats; choosing another base changes the unit and rescales the numerical value. If a variable has m possible outcomes, its entropy is zero when one outcome is certain and no more than log₂(m) bits. The upper bound is reached by a uniform distribution. Carnegie Mellon’s information theory lecture notes explain these properties and entropy’s connection to coding.
What cross-entropy measures
For two distributions over the same outcomes, cross-entropy is the expected cost of scoring outcomes drawn from p using the probabilities assigned by q:
Recommended Free Tools
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
H(p,q) = −Σₓ p(x) log₂ q(x)
The roles matter: p supplies the outcomes and the weights in the average, while q supplies the probabilities inside the logarithm. If p and q match, cross-entropy is the same as entropy. If they do not, cross-entropy reflects both the uncertainty in p and the penalty for using a mismatched q.
How entropy, cross-entropy, and KL divergence fit together
For discrete distributions, KL divergence is defined as:
DKL(p‖q) = Σₓ p(x) log₂[p(x)/q(x)]
Rearranging the terms gives the central relationship:
Rank #3
H(p,q) = H(p) + DKL(p‖q)
Entropy is the cost associated with the target distribution itself. KL divergence adds the cost of using q instead of p. The mismatch penalty is nonnegative and is zero when the distributions match. Carnegie Mellon’s lecture notes on information theory define cross-entropy and KL divergence and establish this relationship.
KL divergence is generally asymmetric: DKL(p‖q) need not equal DKL(q‖p). It is therefore misleading to call it a distance without qualification. The order also determines which distribution weights the outcomes: both H(p,q) and DKL(p‖q) use p as the weighting distribution.
What happens when a probability is zero?
Under the standard discrete definition, if q(x) is zero for an outcome where p(x) is positive, cross-entropy and DKL(p‖q) are infinite: q treats a possible target outcome as impossible. When p(x) is zero, its term contributes zero by the limiting convention 0 log(0/q) = 0.
Rank #4
Why cross-entropy is used in classification
A classifier produces predicted probabilities over classes. For a one-hot target—where the observed class has probability 1 and the others have probability 0—the per-example multiclass cross-entropy reduces to −log q(y), where y is the observed class. This is why assigning a low probability to the correct class incurs a larger loss.
Across a fixed dataset, the mean cross-entropy for observed labels is the average negative log-likelihood. Thus, minimizing that loss is equivalent to maximizing the likelihood of the observed labels under the model, for this setup. If targets are distributions rather than one-hot labels, the per-example loss is −Σₓ p(x) log q(x). With the target distribution fixed, the entropy term H(p) does not change as the model changes, so minimizing cross-entropy also minimizes DKL(p‖q). The LMU Statistical Learning and Data Science chapter on cross-entropy and KL discusses this connection in machine learning.
Free tools Windows power users keep installed
One-click scans. No signup required.
This equivalence concerns the stated probability distributions and fixed targets; it does not mean every loss function or every training setup is interchangeable with cross-entropy.
Best Value
How the coding interpretation makes the difference intuitive
Entropy is the lower bound on expected code length for a source when its probabilities are known. If observations actually come from p but a code is optimized for q, the expected description length corresponds to cross-entropy H(p,q). The overhead compared with an ideal code based on p is DKL(p‖q). A mismatched probability model makes descriptions longer on average. Carnegie Mellon’s lecture notes explain this coding interpretation.
A caveat for continuous variables
For continuous variables, entropy is expressed using a density integral and is called differential entropy. Unlike discrete entropy, differential entropy can be negative, so discrete intuitions and bounds should not be carried over automatically. MIT OpenCourseWare’s Spring 2016 information theory lecture notes distinguish differential entropy from discrete entropy.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →
