Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
An n-gram language model predicts a word using counts of short sequences in a text collection. A unigram uses no preceding word as context, a bigram uses one, and a trigram uses two. This made n-grams an important statistical approach to language modeling, but their fixed context and sparse counts limit what they can predict.
What an n-gram language model is
An n-gram is a sequence of n adjacent items. In language modeling, the items are usually words or characters. An n-gram language model uses the frequency of word sequences in a training corpus to estimate the probability of the next word.
The order describes the sequence length, not how many words the model will generate. In next-word prediction, the most recent n−1 words form the context:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitches| Model | Sequence length | Preceding words used to predict the next word |
|---|---|---|
| Unigram | 1 | 0 |
| Bigram | 2 | 1 |
| Trigram | 3 | 2 |
For example, a trigram model predicting the next word after “I want to” uses the two most recent words, “want to.” It estimates possible continuations from sequences observed in its corpus. Which word it favors depends on that training data and on how the model handles sequences that were not observed.
#1 Best Overall
How the next-word estimate works
Let wᵢ be the word to predict and let the preceding n−1 words be its history. The model estimates P(wᵢ | wᵢ₋ₙ₊₁ … wᵢ₋₁): the probability of the next word given that recent history.
A basic estimate divides the count of the complete sequence by the count of its preceding context. For a bigram, for instance, the estimate for a word after “want” is the number of times the pair “want [word]” occurs divided by the number of times “want” occurs as the context. This is a Markov-style approximation: the model conditions on a selected recent window rather than the entire document.
Why raw counts fail on unseen phrases
Even a large corpus cannot contain every possible word sequence. If a sequence never appears in training, its raw count is zero; a direct count-based estimate can therefore assign it zero probability. Rare sequences can also produce unreliable estimates because a small number of examples may not represent how language is generally used.
Free tools Windows power users keep installed
One-click scans. No signup required.
This is the sparse-data problem. It becomes more demanding as the context grows: longer sequences can capture more local detail, but any particular sequence is less likely to have appeared often in the training sample.
Rank #3
- Language fundamentals grade 1
- Language skills
- Grammar practice
How smoothing helps
Smoothing adjusts count-based estimates so that unseen or infrequent sequences do not automatically receive unusable probabilities. Some approaches discount observed counts and redistribute probability mass; backoff approaches can fall back to evidence from shorter contexts. Interpolation combines estimates from multiple orders, such as trigram and bigram evidence.
These approaches are not interchangeable, and no one method wins in every setting. In a 1998 Harvard technical report, Stanley F. Chen and Joshua Goodman compared smoothing methods across training-data sizes, corpora, count cutoffs, and bigram and trigram models. Training-data size was the most significant factor in their experiments. The report also described a variation of Kneser–Ney smoothing motivated by its empirical analysis. These findings describe the study’s conditions, not a universal ranking of methods. Read Chen and Goodman’s 1998 technical report.
Microsoft Research’s record for the authors’ June 1996 ACL proceedings paper reports speech-recognition results associated with improved smoothing. That result is evidence for the paper’s experimental setting, not a guarantee that a particular smoothing method will improve every system. See the Microsoft Research publication record.
Choosing an order and evaluating a model
Higher-order models use more preceding words and can represent longer local patterns, but they require enough training examples to estimate those patterns reliably. Increasing the order alone does not ensure a better model. Performance depends on the corpus, its size, tokenization and vocabulary, count cutoffs, smoothing, and the task being measured.
- Choose the order for the available data and task. A longer context may help when the relevant patterns recur often enough; sparse data can undermine its estimates.
- Compare smoothing methods under the same conditions. Use the same corpus, tokenization, cutoffs, and held-out evaluation rather than treating a method’s name as a performance guarantee.
- Match the metric to the use. Cross-entropy or perplexity evaluates probability estimates. For a speech application, task-level measures such as speech-recognition performance may also matter.
Where n-grams fit in language-model history
N-grams are part of statistical language modeling: estimating the likelihood of sequences using patterns observed in data. Markov chains and Claude Shannon’s information-theoretic treatment of statistical sources are relevant foundations for this line of work; they should not be mistaken for today’s exact terminology or modern NLP system design. Carnegie Mellon’s overview discusses this statistical and information-theoretic context. Read Carnegie Mellon’s language-modeling overview.
N-gram models became a major practical approach before neural language models. Their fixed context and dependence on observed sequence counts make them useful for explaining the progression from statistical methods to neural approaches, but smoothing does not give them the broader ability to generalize from unseen patterns associated with neural models. An n-gram model estimates sequence probabilities; it does not understand meaning.
Further reading
The n-gram chapter in the Stanford-hosted third edition of Speech and Language Processing develops the model, smoothing, and its limitations in more detail. Read the n-gram chapter in the Stanford textbook.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
