Heritrix review
A configurable crawler for teams operating their own web-archiving infrastructure.
Reviewed by iTechGuides Editors · Editorial team · Updated Oct 2026
Heritrix is an open-source web crawler developed by the Internet Archive for archival-quality crawling. It is aimed at technical teams and organizations that operate their own web-archiving infrastructure, rather than buyers seeking a hosted capture service. Operators define jobs with seed URLs and configuration files, then set the scope and rules that determine which content the crawler includes or excludes. Heritrix is primarily used on Linux, requires Java 17 or later, and is also listed for Windows and API use. Its Apache License 2.0 makes it available as open-source software.
The crawl controls are the central strength. Teams can set limits based on downloaded bytes, document count, or elapsed time, and configure politeness policies. Heritrix can respect robots.txt and META nofollow directives, follow links within configured scope, and write collected material using WARC writer components. Job management is available through a web-based user interface as well as a REST API, giving technical operators both a graphical option and a way to control jobs programmatically. These capabilities suit workflows where teams need to shape crawling behavior and produce WARC output.
Heritrix is a crawler engine, not a complete archive-management experience. It does not provide a hosted capture service, and recurring schedules and retention depend on how an organization deploys and configures it. That distinction matters for teams planning the surrounding infrastructure and ongoing operations: the crawler handles configurable collection, while the supplied product description does not position it as a turnkey system for managing an archive end to end. Choose Heritrix if your team can operate its own crawling environment and values scope, limits, API control, and WARC output. A team looking for hosted capture or a complete archive-management service should consider a different type of product.
Heritrix pros and cons
- Where it wins
- Set crawl scope, inclusion and exclusion rules, and resource limits
- Manage jobs through a web interface or REST API
- Write crawled content with WARC writer components
- Where it doesn't
- Requires teams to operate their own infrastructure
- Job configuration and crawl policies require technical oversight
- Recurring schedules and retention depend on deployment and configuration
Heritrix fact sheet, pricing and score →
Advertiser disclosure: iTechGuides is reader-supported. Vendors can pay for top positions in our rankings and for a place on other products' pages, and we may earn a commission when you click some links. Bought places stand at the number bought; this page and our disclosure say so. Paying vendors may get a fuller, more detailed review from our editors, and that review can change their score. How we rank.
Last updated · How we research and update