iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Underfitting means a model is too limited or constrained to capture useful patterns, so its training and validation performance are both weak. Overfitting means it fits the training examples too closely, including noise or quirks, so training performance is strong but validation performance is materially worse. The practical goal is not the highest training score; it is reliable performance on data that represents the model’s intended use.
Overfitting vs. underfitting: what is the difference?
| Pattern | Training performance | Validation performance | What it suggests |
|---|---|---|---|
| Underfitting | Poor | Generally poor | The model or its constraints may be too simple to capture the useful signal. |
| Overfitting | Strong | Materially weaker | The model may be tracking training-specific noise or quirks that do not carry over to unseen examples. |
| Useful fit | Strong enough | Strong and reasonably close to training performance | The model captures patterns that transfer beyond the training examples. |
The student analogy is a useful shorthand: an overfitting model “memorizes,” while an underfitting model “doesn’t study at all.” It describes the difference in behavior, not a literal process inside a model. In practice, the signs are score patterns measured with a suitable metric.
How to diagnose the problem from model scores
Choose a metric that matches the task, then compare scores on the training data with scores on validation data or across cross-validation folds. Scikit-learn’s documentation summarizes the basic heuristic: “If the training score and the validation score are both low, the estimator will be underfitting.” It also says: “If the training score is high and the validation score is low, the estimator is overfitting and otherwise it is working very well.” These are useful starting points, not universal numeric thresholds.
Recommended Free Tools
- Both scores are weak: investigate underfitting, but also check whether the metric, data, and evaluation setup are appropriate.
- Training is strong and validation is notably weaker: overfitting is a reasonable concern. Check whether the validation examples fairly represent the model’s intended use.
- The scores look reasonable: compare variability across folds or repeated splits and consider how performance changes as the training set grows. A single favorable split can hide instability.
A score pattern is not a diagnosis by itself. Data quality, metric choice, leakage, split design, and the relationship between validation examples and real deployment can all affect the result. For example, a validation set that is much easier than the data the model will encounter may give misleading reassurance.
#1 Best Overall
Use learning and validation curves to narrow down the cause
Validation curves: test one setting at a time
A validation curve plots training and validation scores as you vary one hyperparameter. For example, changing regularization strength can show whether the model performs poorly because it is too constrained or because it is too flexible. Stronger regularization limits model flexibility and may reduce overfitting; too much can suppress useful signal and cause underfitting. Scikit-learn’s validation and learning curve documentation describes these curves and how to inspect the score patterns.
Learning curves: see whether more examples help
A learning curve compares training and validation performance as the amount of training data changes. If validation performance improves as more representative examples are added and the gap narrows, additional data may help. If both scores settle at poor levels, simply adding data may not solve the problem: the model may be unable to represent the useful relationship, or another part of the setup may be limiting performance.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
What to change for overfitting or underfitting
| Observed issue | Experiments to try | Trade-off to watch |
|---|---|---|
| Overfitting: training score is strong, validation is weaker | Try a less flexible model, increase regularization, or add more representative training data when feasible. | Too much regularization or too little flexibility can turn overfitting into underfitting. More data helps only when it is relevant and the model can represent the underlying relationship. |
| Underfitting: training and validation scores are both weak | Try a more expressive model or relax overly strong regularization. | Greater flexibility can improve the training fit but may widen the gap between training and validation performance. |
Treat each change as an experiment rather than a guaranteed fix. Change one major factor at a time, keep the metric and validation design consistent, and record both scores and their gap. Scikit-learn’s regularization example illustrates how stronger regularization constrains coefficients and how an overly strong constraint can limit useful fit.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Compare models without contaminating the final test
- Fit candidate models on training data. Use this data to estimate the model’s parameters.
- Compare choices with validation data or cross-validation. Use the same task-appropriate metric and a split design that reflects the intended use. Scikit-learn’s cross-validation guide explains the role of cross-validation in evaluating model choices.
- Keep a final test set for an evaluation after choices are made. Use it to assess the selected approach, not as another tuning target.
Repeatedly changing a model in response to its test score lets decisions adapt to that test set, so it no longer provides an untouched estimate of generalization. The same caution applies to a validation score after extensive hyperparameter tuning: it has guided choices and should not be presented as an independent final assessment.
Rank #3
For time-dependent, grouped, or otherwise structured observations, the split should preserve the structure that matters for deployment. A random split can make evaluation too easy if closely related observations land on both sides. The appropriate split depends on the data and intended use; no single splitter is correct for every problem.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What to compare when choosing a model
Evaluate candidate models using a consistent metric and split strategy. Look beyond the best training score:
Rank #4
- Training score: how well the model fits the examples used to train it.
- Validation or cross-validation score: how well it performs on held-out examples used for model selection.
- Train-validation gap: whether fit on training examples carries over to held-out examples.
- Variation across folds or repeated splits: whether performance depends heavily on which examples were held out.
- Performance as sample size grows: whether additional representative data appears to improve validation results.
- Complexity and regularization: whether the model’s flexibility is worth the validation performance it achieves.
Scikit-learn’s polynomial model illustration shows the bias–variance intuition: a simple model can have high bias, while a more flexible one can become sensitive to changes in its training sample. That framework helps explain the ends of the fit spectrum, but real-world generalization also depends on data noise and evaluation design.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsQuick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
