Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsAjit Jaokar’s 2021 taxonomy organizes transformer-based pretrained language models (TPTLMs) along four lenses: their pretraining corpus, architecture, self-supervised learning objective, and extensions. It is best read as a conceptual map of model families—not as a current ranking or guide to choosing a model for a particular task.
What the taxonomy is for
In a post published September 5, 2021, Jaokar presents a framework for describing how transformer-based pretrained language models differ. The four lenses help readers ask distinct questions: What data was used for pretraining? What parts of the transformer architecture are used? What self-supervised learning approach is involved? What additional design property or capability is emphasized? Read Jaokar’s taxonomy.
These lenses are not four competing labels from which a model receives exactly one. A model can be described by its corpus and architecture, for example, while also having an extension such as a long-sequence design. The extension list itself mixes engineering properties, representation choices, and intended capabilities.
1. Pretraining corpus: what data and language coverage?
The post distinguishes models trained on general corpora from those trained on social-media or language-specific data. Language-specific training can be monolingual or multilingual. Corpus is a useful lens because it points to the data and language context behind pretraining, rather than the model’s structure.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11#1 Best Overall
As examples in the 2021 post, GPT-1 is associated with BooksCorpus, while BERT and UniLM are associated with English Wikipedia and BooksCorpus. These are historical examples from that post, not a complete inventory of models or a statement about current training data.
2. Architecture: which transformer stack?
The architecture lens divides models by the transformer stack they use:
- Encoder-based: uses an encoder stack.
- Decoder-based: uses a decoder stack.
- Encoder-decoder-based: uses both an encoder and a decoder.
This category describes model structure. By itself, it does not establish which model will perform best for a particular task or how it must be adapted or deployed.
3. Self-supervised learning: what objective family?
The post groups self-supervised learning (SSL) approaches into four families:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Generative
- Contrastive
- Adversarial
- Hybrid
This lens concerns the learning approach used in pretraining. The taxonomy names these broad families; it does not provide a current comparative evaluation of models within them.
4. Extensions: what additional property or capability?
Jaokar’s post lists several extensions to the basic perspectives. They are best treated as overlapping descriptors, not as mutually exclusive model classes.
- Efficiency and size: compact models, efficient models, and green models. The post describes compression methods for compact models including pruning, parameter sharing, distillation, and quantization. DeBERTa is listed as an efficient-model example.
- Scale: large-scale models.
- Sequence length: long-sequence models.
- Representation and input: character-based models, tokenization-free models, and sentence-embedding models. CharacterBERT is given as a character-based example.
- Knowledge: knowledge-enriched models.
The examples and labels reflect the 2021 post’s organization. They should not be taken as a complete or up-to-date catalog of model designs.
How to use the map when comparing models
The taxonomy helps structure a first-pass description, but it does not answer which model to use. A practical comparison for a real project should also examine:
Recommended Free Tools
- The task and required output format.
- The training corpus and language coverage relevant to the intended use.
- The architecture and approach to context length.
- Prompting, adaptation, or other requirements for the task.
- Deployment cost and latency.
- Licensing and data-governance constraints.
Those are decision criteria to apply alongside the taxonomy, not factors evaluated in Jaokar’s post. The post is a conceptual explainer rather than a task-specific recommendation or model ranking.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What the linked AMMUS survey covers
Jaokar’s post points readers to the survey AMMUS: A Survey of Transformer-based Pretrained Models in Natural Language Processing. Its abstract describes coverage of pretraining, methods and tasks, embeddings, downstream adaptation, intrinsic and extrinsic benchmarks, useful libraries, and future research directions. The abstract offers an overview of the survey’s scope, but it does not by itself verify every detail in Jaokar’s taxonomy. See the AMMUS survey record on arXiv.
How current is this taxonomy?
The post was published in 2021, so its categories and examples are useful as a framework, not as a current assessment of model availability or suitability. It does not establish present-day benchmark leadership, licensing, or deployment requirements. Those details need to be checked against current sources for the specific models and use case being considered.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →
