What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A Whoosh index is not usually one monolithic file. In Whoosh 2.7.4, it is organized around a master .toc file and one or more segment mini-indexes. Each segment can contain separate files for document data, stored values, terms, and postings; optional term-vector files depend on the schema. The exact directory contents therefore depend on both field configuration and how the index has been updated or merged.
Start with the master file and its segments
The Whoosh 2.7.4 file-layout documentation describes the .toc as the master file. It contains information about the index and its segments, so it is the place that tracks which mini-indexes make up the index. The number in its filename is a revision number, not a count of documents.
A segment is a mini-index. Adding documents can create a new segment; searches combine results from the segments, and Whoosh can later merge segments. Consequently, two indexes with the same schema and documents need not have the same segment count or identical directory contents: their indexing and merge histories can differ. See the Whoosh 2.7.4 file database documentation.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteWhat the segment files do
The segment files divide index work into distinct kinds of information. In the documented layout, the segment number is part of each segment filename.
#1 Best Overall
| File | Role in Whoosh 2.7.4 |
|---|---|
<segment_number>.dci |
Per-document information, such as field lengths. Field lengths are relevant to scoring for applicable fields; do not assume every field has them. |
<segment_number>.dcz |
Stored field values for documents. This supports retrieving document data and is separate from the term-to-document postings used for search. |
<segment_number>.tiz |
Per-term information. Its size varies with the number of unique terms. |
<segment_number>.pst |
Postings for terms: the information connecting terms to matching documents. Its size depends on the collection and field formats, including whether positions are retained. |
<segment_number>.fvz |
Term vectors, also called forward indexes, when the schema enables them. Whoosh 2.7.4 documents this file as conditional, not a required part of every index. |
These names describe file roles, not a promise about exact byte-level record layouts or fixed file sizes. The official file database page documents the layout; it does not establish a universal inventory for every index.
How schema choices shape the contents
The schema declares fields and their types, and a field can be indexed, stored, or both. These choices determine what information Whoosh needs to retain.
Indexed data and stored data are different
Indexing makes a field searchable; storing retains its value so it can be retrieved with a document. A field does not have to do both. In Whoosh, TEXT is not stored by default. Use TEXT(stored=True) when the original text should also be retained. A STORED field keeps a value without indexing it. The Whoosh schema documentation describes these field behaviors.
Posting formats control what search can use
An inverted index maps terms to documents. A posting format determines how much information is kept for each term: existence alone, term frequency, or frequency plus positions. In Whoosh 2.7.4, TEXT uses positional information by default to support phrase queries. Disabling phrase support allows frequency-only storage instead. Positions add information to postings, so the field format can also affect .pst size.
Field type matters too: an ID field treats a complete value, such as a path, as one term, while KEYWORD is intended for delimited keywords. Those indexing behaviors follow the schema, rather than being inferred from the file extension alone.
Term vectors reverse the usual direction
Postings are inverted: they connect terms to documents. A term vector, or forward index, instead connects a document to its terms. Whoosh does not use term vectors by default. The 2.7.4 documentation says a segment’s .fvz file is created only when at least one schema field stores term vectors. Its absence is therefore normal for schemas that do not request them.
Why your index directory may look different
- Different schemas: stored values affect document data files, posting formats determine whether term frequencies or positions are retained, and term-vector settings determine whether
.fvzappears. - Different update and merge histories: newly added documents can form additional segments, while merges can change how many segments remain.
- Different collections and formats: the number of unique terms and the information stored in postings affect file sizes; the documentation does not specify a single expected size.
Accordingly, the extensions are useful clues about a file’s role, but a directory listing alone does not reveal a universal layout or a fixed number of files.
Version matters when inspecting or migrating an index
The file details above are documented for Whoosh 2.7.4, not as a guarantee for every Whoosh release. The index API exposes a version tuple identifying both the release that created an index and its on-disk format version. For forensic inspection or migration, check the actual index version and consult documentation or source for the matching release. The cited documentation does not establish byte-level compatibility across versions. See the Whoosh index API documentation.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
