When a grid search is too expensive or too rigid, try randomized search, Bayesian optimization, or resource-adaptive search with successive halving or Hyperband. They solve different problems: randomized search gives you a simple, controllable trial budget; Bayesian optimization uses earlier results to choose later trials; and successive halving spends fewer resources on candidates that look weak early. The right choice depends on your search space, evaluation cost, compute budget, and whether early scores predict final performance.
Why look beyond grid search?
Grid search evaluates every combination in a specified set of parameter values. That is easy to reason about, but the number of runs grows as you add parameters or choices: with several parameters, the total is the product of the number of values specified for each one. A grid can therefore spend substantial compute evaluating combinations that are not promising. Scikit-learn’s hyperparameter-tuning documentation describes grid search and alternatives.
These alternatives change how candidates are selected, how much resource each candidate receives, or both. None guarantees a better model: each method searches according to the space, objective, and evaluation procedure you define.
1. Randomized search: set a trial budget and sample candidates
Randomized search draws a fixed number of configurations from distributions or discrete choices instead of evaluating every point in a Cartesian grid. You choose the number of trials independently of how many possible values the space contains. This makes it a practical baseline when you want predictable search effort and can specify plausible ranges.
Recommended Free Tools
#1 Best Overall
Define useful distributions
For continuous parameters, scikit-learn recommends using continuous distributions rather than listing a few arbitrary values. For a parameter whose scale matters, a log-uniform distribution can allocate trials across orders of magnitude instead of concentrating them in one part of the range. The distribution is part of the search design: an unrealistic range can waste the trial budget even when the sampling algorithm works as intended.
When it fits
- Use it when you need a straightforward baseline and want to cap the number of evaluations.
- It is easy to parallelize because candidate evaluations can be run independently.
- Adding a parameter that turns out to be irrelevant does not multiply the candidate count as it would in a full grid, though it can still consume search attention.
In scikit-learn, the relevant estimator is RandomizedSearchCV. Its results still depend on the chosen parameter distributions, score, and cross-validation setup. The official documentation explains the search interface and its inputs.
Rank #2
2. Bayesian optimization: use previous trials to guide the next ones
Bayesian optimization treats tuning as an adaptive loop. It evaluates initial configurations, fits a surrogate or probabilistic model of the objective from observed results, chooses a promising next configuration, observes its score, and updates the model. The intention is to make each expensive evaluation more informed by the trials already completed.
When it fits—and what it costs
Consider it when evaluations are expensive enough that choosing candidates intelligently could save meaningful compute. Its benefit depends on the objective, search-space representation, and evaluation conditions; it does not guarantee a global optimum or consistently beat randomized search. Noisy, high-dimensional, non-convex objectives make reliable optimization difficult. Because later choices use earlier outcomes, adaptive search is commonly sequential and can be harder to parallelize than independent random trials. Parallel variants trade off some of that feedback or add implementation complexity. These challenges are discussed in the 2021 hyperparameter-optimization survey and the Hyperband paper.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesKerasTuner lists Random Search, Bayesian Optimization, and Hyperband as built-in algorithms; that is one implementation option, not a claim that its APIs are interchangeable with other frameworks. See the KerasTuner overview.
3. Successive halving and Hyperband: allocate resources adaptively
Successive halving and Hyperband address the cost of evaluating candidates by changing how much resource each receives. Start many candidates with a small budget, keep the stronger performers, then give the survivors more resource while stopping others early. Resource might mean training samples or a numeric model control such as estimator count in scikit-learn; the Hyperband paper also discusses iterations, data samples, and features.
Rank #4
Why Hyperband is a family-level choice
Successive halving is the basic successive-allocation approach; Hyperband organizes such allocation across different resource-budget strategies. Treat them as one resource-adaptive family when comparing three broad tuning techniques: the central idea is to reserve full effort for candidates that perform well under smaller budgets.
The early-ranking caveat
Pruning only helps when early performance is informative about later performance. If a configuration starts slowly but would improve with more training, an early score can cause it to be discarded prematurely. Check whether learning curves at the proposed resource levels rank candidates usefully before relying on aggressive stopping.
Best Value
Scikit-learn provides HalvingRandomSearchCV and HalvingGridSearchCV. Its documentation currently marks successive-halving estimators experimental and says they require an explicit enable import; check the documentation for the scikit-learn version you use before adopting them. The current stable documentation describes the estimators and their resource options.
How the three approaches compare
| Decision factor | Randomized search | Bayesian optimization | Successive halving / Hyperband |
|---|---|---|---|
| How candidates are chosen | Samples independently from defined distributions or choices | Uses earlier trial results to guide later candidates | Typically combines candidate selection with increasing resource for survivors |
| How it addresses evaluation cost | Caps the number of sampled candidates | May reduce the number of expensive evaluations; results depend on the problem | Stops weaker candidates before they receive the full resource budget |
| Parallelism | Straightforward to parallelize across independent trials | Adaptive feedback makes some searchers sequential; parallel variants involve trade-offs | Candidates can run in parallel within a rung, subject to resource and scheduling limits |
| Main setup burden | Choose plausible distributions and a trial budget | Define the objective and search space, and select optimization and modeling choices | Choose comparable resource levels and establish a useful early stopping signal |
| Useful when | You want a simple, budget-controlled baseline | Each evaluation is costly enough to justify informed selection | You can identify poor trials early and reallocate their resources |
This is a conceptual comparison, not a benchmark ranking. The scikit-learn documentation, optimization survey, and Hyperband paper describe the relevant methods and trade-offs.
Choose by the shape and cost of your tuning problem
- Choose randomized search when you can define sensible parameter distributions and want to control the number of trials. It is often the clearest starting point for a broad space.
- Consider Bayesian optimization when full evaluations are expensive and the objective and space are suitable for an adaptive optimizer. Account for the difficulty of parallelizing feedback-driven choices.
- Consider successive halving or Hyperband when trials can be compared at smaller resource levels and early scores reliably indicate which candidates deserve more training.
Trial cost alone does not determine the best method. Also consider total compute, parameter-space shape, early learning-curve reliability, parallel hardware or scheduling constraints, and the implementation you can maintain. A method that sounds efficient can waste compute if its assumptions do not match the problem.
Make the comparison trustworthy
Define the evaluation before searching
Specify the estimator, parameter space, search or sampling method, cross-validation scheme, and score function. Scikit-learn summarizes a search in those terms in its tuning documentation. Choose a validation design appropriate to the data, and keep final test data out of the tuning loop; repeatedly selecting against the test set turns it into part of the optimization process.
Record enough to reproduce the result
- Save the parameter space and distributions, including bounds and scales.
- Record the random seed where applicable, trial budget, resource schedule, and stopping rules.
- Keep trial outcomes, score definitions, validation splits or cross-validation setup, and software versions.
- Report the best configuration as best under that objective and evaluation procedure, not as automatic proof of generalization.
The Hyperband authors reported a 5× to 30× speedup over state-of-the-art Bayesian optimization algorithms on a variety of deep-learning and kernel-based learning problems in their 2016 experimental settings. That is a result for those comparisons, not a performance promise for a different dataset, workload, or implementation. See the Hyperband paper.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
