Free tools Windows power users keep installed
One-click scans. No signup required.
Information entropy is the average information you gain when you learn the outcome of an uncertain event. A rare result is more surprising than a likely one; entropy averages that surprise across all possible results. For a fair coin, the average is one bit.
What does information entropy mean?
Imagine a fair coin before it is tossed. Either heads or tails could occur, and neither is more likely than the other. Learning the result resolves uncertainty: if the coin lands heads, that outcome was one of two equally likely possibilities.
Shannon’s information theory quantifies the surprise of an individual outcome using its probability. An outcome with probability p has self-information log₂(1/p) bits. A less likely outcome carries more self-information because learning it rules out more possibilities. For a fair coin, p = 1/2, so either outcome carries log₂(2) = 1 bit. MIT’s lecture notes attribute the concise description “Information is the resolution of uncertainty” to Claude Shannon (MIT OpenCourseWare, Fall 2012).
Entropy is the average self-information across the outcomes of a random variable. In plain language, it is the average uncertainty before the outcome is revealed. It belongs to the probability distribution, not to one result that has already happened. MIT describes it as the average information received when learning the value of a discrete random variable (MIT OpenCourseWare, Spring 2017).
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsHow do I calculate Shannon entropy?
For a discrete random variable X whose possible outcomes have probabilities pᵢ, Shannon entropy is:
H(X) = −Σᵢ pᵢ log₂(pᵢ)
The probabilities must sum to 1. The equivalent form, Σᵢ pᵢ log₂(1/pᵢ), makes the averaging especially clear: multiply each outcome’s self-information by its probability, then add the results. The base-2 logarithm gives the answer in bits, also called shannons in formal usage. Using natural logarithms instead gives nats. If an outcome has probability zero, its contribution is defined by the limiting convention 0 log 0 = 0.
Fair and biased coins
For a coin with probability p of heads and 1−p of tails, entropy is h(p) = −p log₂(p) − (1−p) log₂(1−p). At p = 1/2, the outcomes are equally likely and entropy reaches its maximum of 1 bit. As p approaches 0 or 1, one result becomes nearly certain and entropy tends toward zero. A predictable coin toss leaves less uncertainty to resolve.
Identifying a card’s suit
Suppose a card is selected uniformly from a standard 52-card deck, and you learn only that it is a spade. There are 13 spades, so the probability of that information is 13/52 = 1/4. Its self-information is log₂(52/13) = 2 bits. This is the information in learning that the card belongs to the spade subset, not the information in identifying which particular spade it is. MIT uses this example in its lecture notes (MIT OpenCourseWare, Fall 2012).
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #3
A distribution with unequal probabilities
Entropy is an average, so outcomes with different probabilities contribute different amounts. For a four-outcome source with probabilities 1/3, 1/2, 1/12 and 1/12, MIT calculates an entropy of 1.626 bits (MIT OpenCourseWare, 2017 course notes). The less likely outcomes have more self-information, but they count less often in the weighted average.
Why does entropy matter for compression?
Entropy connects uncertainty to the amount of data needed to represent a source. If symbols come from a known probability distribution, a lossless code can assign shorter representations to common symbols and longer ones to rare symbols. For that modeled source, entropy sets a lower bound on the average number of bits per symbol for unambiguous lossless representation. Suitable practical codes can approach that limit under the assumptions of the source and coding method (MIT OpenCourseWare, Spring 2017).
Rank #4
That lower bound is an average, not a rule that every symbol must use exactly the entropy number of bits. Actual codeword lengths are discrete, and a coding scheme may use different lengths for different symbols. The entropy value describes the distribution’s theoretical average limit; it does not by itself specify a complete code.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What is the difference between information entropy and thermodynamic entropy?
Information entropy describes uncertainty in a probability distribution over outcomes. Thermodynamic entropy belongs to statistical mechanics and is connected to the number and probabilities of microscopic states (microstates) consistent with a larger-scale description (a macrostate). The concepts share a deep mathematical relationship, but they arise in different models and have distinct physical interpretations and units. “Disorder” can be a rough intuition for some discussions of physical entropy, but it is not a complete definition of either concept. For an introductory physics treatment of microstates and macrostates, see the University of Massachusetts Amherst Open Books chapter on entropy.
Where to learn more
For a free, structured next step, MIT OpenCourseWare’s archived Spring 2008 Information and Entropy course includes Professor Paul Penfield’s open textbook and materials on bits and codes, compression, probability, communications, inference, maximum entropy, physical systems and quantum information. It is an archived course edition, rather than a current class.
For a book-length introduction, James V. Stone’s Information Theory: A Tutorial Introduction (published 2015) is presented by its author as a novice primer with accessible examples and online MATLAB and Python programs. The author’s page does not establish current price or availability.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
