autorank review
A focused Python package for choosing and reporting tests across paired populations.
Reviewed by iTechGuides Editors · Editorial team · Updated Oct 2026
autorank is an open-source Python package for comparing multiple paired populations, including classifier results. It suits analysts who work in Python and want help selecting statistical tests and presenting comparisons without choosing each frequentist test manually. The analysis starts from a pandas DataFrame and can produce rankings, confidence intervals, effect sizes, test results, plots, tables, and LaTeX reports. It is locally installed software available for Linux, Windows, and macOS, rather than a conventional desktop statistics application.
Its central strength is the test-selection workflow: autorank checks normality and homogeneity, then selects a frequentist test based on those assumptions and the number of populations. Normality testing uses Shapiro-Wilk, while homogeneity checks use Bartlett's or Levene's test. Users can instead run Bayesian signed-rank analysis and configure the region of practical equivalence (ROPE). Effect sizes and confidence intervals add context to the test results, and plots plus generated tables and reports support communicating comparisons. Integration with pandas, SciPy, statsmodels, and baycomp fits an existing Python analysis workflow.
The open-source package has a free plan, and its support channel is community-based. Its scope is deliberately narrower than general statistical software: it focuses on paired-population comparison and ranking, with no regression models or broad modeling capability. That makes autorank a fit for Python users comparing classifier or other paired results who value automated test choice and reporting. Analysts looking for regression, wider modeling, or a conventional point-and-click statistics application should choose a broader alternative.
autorank pros and cons
- Where it wins
- Selects frequentist tests using normality, homogeneity, and population count
- Supports Bayesian signed-rank analysis with configurable ROPE
- Generates rankings, plots, tables, and LaTeX reports
- Where it doesn't
- Focused on comparing multiple paired populations, not broad statistical analysis
- Does not provide regression models
- Requires data in a pandas DataFrame and Python scripting
autorank fact sheet, pricing and score →
Advertiser disclosure: iTechGuides is reader-supported. Vendors can pay for top positions in our rankings and for a place on other products' pages, and we may earn a commission when you click some links. Bought places stand at the number bought; this page and our disclosure say so. Paying vendors may get a fuller, more detailed review from our editors, and that review can change their score. How we rank.
Last updated · How we research and update