iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
If both your training and validation scores are poor, the model may be underfitting because its regularization is too strong. Relax the penalty in small, documented steps, compare every trial on the same validation design, and stop if validation performance no longer improves. A weak fit can also come from inadequate features, bad labels, scaling errors, failed optimization, or an undersized model, so diagnose those causes before assuming regularization is responsible.
The key evidence is the gap between training and validation performance. Scikit-learn’s validation-curve documentation puts it plainly: “If the training score and the validation score are both low, the estimator will be underfitting.” A high training score with a low validation score points toward overfitting instead.
First, verify that the problem is underfitting
Do not change a penalty after looking at one score. Compute the same task-appropriate metric on the training data and on data that was not used to fit the model. For classification, that might be balanced accuracy, precision-recall, log loss, or another metric that reflects the cost of errors. For regression, choose a suitable loss or error measure. The metric must match how the model will be used in deployment.
| Training result | Validation result | Likely interpretation | Next investigation |
|---|---|---|---|
| Low | Low | Underfitting is plausible | Check regularization, features, model capacity, optimization, and data quality |
| High | Low | Overfitting is more likely | Increase regularization, simplify the model, add data, or improve validation design |
| High | High | Useful fit | Confirm once on an untouched test set and monitor after deployment |
| Unstable across splits | Unstable across splits | High variance or an unrepresentative split | Inspect fold variability, group or time structure, and sample size |
The training score is not a generalization estimate. Use a validation set or cross-validation while making choices, and keep a separate test set untouched until the end. If rows from the same person, device, customer, or time period are related, ordinary random folds can leak structure or misrepresent deployment. Use group-aware or time-aware splits when those conditions apply.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Establish a trustworthy baseline before tuning
Confirm the split and metric
Write down the split rule, random seed where applicable, metric, and preprocessing steps. Run a baseline with the current regularization value and save both training and validation results. For cross-validation, record the score from every fold, not only the mean; a small mean difference can be meaningless when fold variation is large.
Keep preprocessing inside the fit pipeline
Transformations such as imputation, scaling, feature selection, and dimensionality reduction must be fitted only on each training fold. In scikit-learn, place them in a Pipeline so validation and test observations cannot influence learned transformation parameters. This is especially important for support-vector machines: they are not scale invariant, and scikit-learn recommends scaling their input features.
Check that training actually converged
An optimization warning, an iteration limit, exploding values, or a solver that stops early can look like excessive regularization. Inspect convergence messages and solver diagnostics. Confirm that labels are aligned with rows, categorical values are encoded as intended, missing-value handling is consistent, and the metric implementation is correct. A broken data path cannot be repaired by changing a penalty.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesRank #2
Relax regularization in controlled steps
Once the baseline and diagnosis are sound, search a range of weaker and stronger settings rather than making one dramatic change. Use logarithmically spaced values when the parameter acts as a scale factor. Compare every candidate with the same folds, preprocessing, metric, and random-state policy.
| Estimator or family | Parameter | Stronger regularization | Weaker regularization | Important qualification |
|---|---|---|---|---|
| Scikit-learn SVM | C |
Decrease C |
Increase C |
C controls the penalty trade-off; confirm the convention for the exact estimator and version |
| Scikit-learn logistic regression | C |
Decrease C |
Increase C |
The documentation describes larger C as less regularization; supported penalties and defaults vary by release |
| Lasso | alpha |
Increase alpha |
Decrease alpha |
alpha multiplies the L1 penalty |
| Elastic Net | alpha and the L1 ratio |
Increase overall alpha for a stronger penalty |
Decrease alpha for a weaker overall penalty |
The L1 ratio controls the mix of L1 and L2 behavior; inspect the estimator’s versioned API |
These directions are not universal across libraries. A parameter named “regularization,” “weight decay,” or “strength” may increase the penalty when its numeric value increases. Read the documentation for the installed estimator and version before mapping a search range.
Use a validation curve, not a single comparison
A validation curve plots a score against one selected parameter. It shows whether the model is on the under-regularized or over-regularized side of the useful range and whether the curve has flattened. Search, for example, several values of C around the baseline for an SVM or logistic model, and several values of alpha for Lasso. Select using validation results, then evaluate the chosen configuration once on the untouched test set.
Use a learning curve to test whether more data could help
A learning curve varies the number of training examples and plots training and validation scores. If both curves remain low and close together as the sample grows, the problem is likely bias, representation, optimization, or capacity rather than a shortage of data. If the validation score is still rising and the training and validation curves are converging, additional representative data may help. Do not use the test set to draw this tuning curve.
Understand what the penalty is doing
L1, L2, and Elastic Net in linear models
L1 regularization can drive coefficients exactly to zero, producing a sparse model that may be easier to inspect and deploy. L2 regularization shrinks coefficients toward zero without generally making them exactly zero, which can stabilize a model when many features contribute small effects. Elastic Net combines L1 and L2 behavior and can retain sparsity while handling correlated predictors differently from pure L1.
When a linear model underfits, lowering an L1 or Elastic Net alpha allows larger coefficients and a less constrained fit. Examine whether useful coefficients are being eliminated and whether features have comparable scales. Scikit-learn documents LassoCV and LassoLarsCV for selecting Lasso’s alpha by cross-validation; these can replace a manually chosen single value, provided the cross-validation design matches the deployment problem.
SVMs
For scikit-learn SVMs, a smaller C imposes stronger regularization and a larger C weakens it. If both scores are low, increase C gradually while keeping feature scaling in the pipeline. A very large C can reduce bias but may increase variance, so the validation curve—not the training score alone—decides whether the change helped.
Logistic regression
Scikit-learn describes C as inversely related to regularization strength: larger C means less regularization. The library supports L1, L2, and Elastic Net combinations subject to solver and version constraints. Check those combinations in the documentation for your installed release; a setting accepted in one release may have different defaults or restrictions in another.
Free tools Windows power users keep installed
One-click scans. No signup required.
Neural networks
The same diagnosis applies to neural networks: compare training and validation behavior before changing dropout, weight decay, or early-stopping patience. There is no framework-independent numeric rule for how much to change any of them. Use the documentation for the framework and version you run, and change one control at a time while holding the data split, optimizer, learning-rate schedule, and random-seed policy steady.
Best Value
A reproducible tuning workflow
- Freeze the evaluation design. Choose the deployment-representative metric and split. Reserve the final test set; do not use it to select parameters.
- Record the baseline. Fit the current pipeline and capture training score, validation scores for every fold, mean, spread, fit warnings, and training time.
- Inspect the data path. Verify labels, feature construction, missing-value handling, encoding, scaling, and convergence before touching regularization.
- Define a directional search. Include the current setting, several weaker settings, and at least one stronger setting so the curve has context. For example, increasing
Cweakens the penalty for the scikit-learn SVM and logistic-regression cases; decreasing Lassoalphaweakens its L1 penalty. - Run consistent cross-validation. Use the same folds for each candidate. Keep group or time boundaries intact when random splitting would leak information.
- Compare more than the mean score. Record the training-to-validation gap, fold or seed stability, fit time, model size, and—when relevant—the number of nonzero coefficients.
- Select the simplest setting that meets the objective. If several candidates perform similarly, prefer the one with better stability, fewer features, or easier interpretation when those properties matter.
- Refit and test once. Refit the selected pipeline on the permitted training data, then report its performance on the untouched test set. Treat that result as the final estimate for this experiment.
Example scikit-learn pattern
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
pipe = Pipeline([
('scale', StandardScaler()),
('model', SVC(kernel='rbf'))
])
grid = {'model__C': [0.01, 0.1, 1, 10, 100]}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
search = GridSearchCV(pipe, grid, scoring='balanced_accuracy', cv=cv, return_train_score=True)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)This pattern keeps scaling inside each fold and retains training scores for diagnosing the gap. Replace the metric, splitter, estimator, and parameter range with choices appropriate to your task. Do not treat the example’s five folds, seed, or parameter values as universal requirements.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.If weaker regularization does not improve validation performance
Improve the representation
Underfitting can mean the features do not express the relationship the target depends on. Check units, aggregation windows, categorical encoding, interaction terms, transformations for skewed variables, and whether important signals are missing. For time-dependent tasks, ensure features use only information available at prediction time.
Increase model capacity deliberately
Compare a richer model family or a controlled increase in flexibility after establishing the baseline. More capacity can reduce bias but also increase variance. Evaluate it with the same split and metric, and keep regularization in the search rather than removing it entirely.
Recommended Free Tools
Repair data and objective problems
- Audit label definitions, class mapping, duplicates, and mislabeled examples.
- Check whether the training distribution represents the deployment population.
- Inspect class imbalance and whether the chosen metric hides poor minority-class performance.
- Verify that the loss function and target transformation match the business objective.
- Look for leakage in feature creation, aggregation, and preprocessing.
Revisit optimization and scaling
Non-convergence, poor learning-rate choices, numerical overflow, or unscaled features can leave a model with a weak training fit. For SVMs, scaling is a specific requirement because the algorithm is not scale invariant. For other estimators, follow their solver guidance and inspect warnings rather than silently accepting the fitted result.
How to choose among candidate settings
| Comparison criterion | What to inspect | Why it matters |
|---|---|---|
| Validation score | Mean score using the deployment-appropriate metric | Measures performance on data not used for that fold’s fit |
| Training-to-validation gap | Difference between training and validation scores | Reveals whether added flexibility is creating overfitting |
| Stability | Spread across folds or random seeds | Shows whether the apparent improvement is dependable |
| Simplicity | Feature count, coefficient sparsity, latency, and interpretability | Determines operational cost and how easily behavior can be explained |
| Final test result | One score on untouched test data | Provides the unbiased final check after choices are complete |
For linear models, include coefficient sparsity in the decision when it is operationally important. L1 and Elastic Net can produce sparse solutions, but a slightly denser model may be preferable if it is substantially more stable or accurate. Make that trade-off explicit instead of selecting solely by a rounded score.
Common mistakes and their fixes
- Tuning from the training score: A higher training score after weakening regularization does not prove better generalization. Compare validation results.
- Changing the wrong direction: Increasing
Cweakens regularization in the scikit-learn SVM and logistic-regression conventions, while decreasing Lassoalphaweakens its penalty. Verify the estimator API. - Scaling before cross-validation: Fit scalers inside the pipeline so validation information is not used to learn preprocessing parameters.
- Repeating experiments on the test set: This turns the test set into another tuning set. Keep it untouched until the final report.
- Ignoring split structure: Random folds can be misleading for grouped or chronological data. Preserve those boundaries.
- Removing regularization entirely: A model that needs less penalty can still overfit at an extreme setting. Search a range and use the validation curve.
- Assuming every weak fit is a penalty problem: If training performance remains poor after sensible changes, investigate features, labels, optimization, scaling, and model family.
A practical decision rule
When training and validation scores are both low, first make the evaluation and pipeline trustworthy. Then relax the estimator’s penalty according to its documented parameter convention and measure the result with consistent cross-validation. If validation improves without an unacceptable increase in the training-to-validation gap, keep the weaker setting and verify it once on the untouched test set. If validation stays flat or worsens, stop reducing regularization and address representation, data quality, optimization, or capacity instead.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
