iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Multi-token prediction (MTP) can speed up language-model generation by letting a smaller drafter propose several tokens and having the target model verify them together. Accepted draft tokens can be emitted after one verification pass; rejected tokens are replaced with the target model’s output. This reduces serial work without asking the target to trust an unverified guess.
How multi-token prediction works
In ordinary autoregressive decoding, a model generates one token, then uses it to calculate the next. Because each step depends on the previous one, the target model must run repeatedly in sequence.
- Draft: A smaller model or draft head proposes a short continuation.
- Verify: The target model checks the proposed positions in parallel.
- Accept or correct: If the target accepts draft tokens, several can be emitted from the verification work. At the first rejection, the target supplies the appropriate token and drafting continues from that point.
The goal is fewer expensive target-model passes, not bypassing the target model’s verification. The method is a form of speculative decoding: its speed benefit depends on how much draft work the target can accept and how much overhead the draft and verification process adds.
Which Qwen models have documented native MTP support?
The vLLM Speculators guide identifies Qwen3-Next and Qwen3.5 as examples with native MTP support. Its description says the draft head combines verifier hidden states with token embeddings, normalizes and projects them, then processes them through a decoder layer. Output states are fed forward recursively to draft later tokens. See the vLLM Speculators MTP guide.
#1 Best Overall
This is a framework’s documentation of supported models and its account of the architecture; it is not a claim that every Qwen checkpoint or serving stack supports the same workflow. The Qwen3 Technical Report, dated May 14, 2025, does not describe MTP. “Qwen3” by itself is therefore too broad a label for a support claim: check the exact model variant and serving framework.
How Gemma 4’s MTP drafter is built
Google’s MTP documentation is for Gemma 4. Its drafter shares the target model’s input embedding table and uses the target’s last-layer activations. Google describes combining activations with token embeddings and down-projecting them to the drafter’s dimension. These details should not be generalized to other Gemma releases.
For Gemma 4 E2B and E4B, Google also describes an efficient embedder that groups similar tokens and narrows final calculations to likely clusters. The implementation details and supported workflows are documented in Google’s Gemma 4 MTP guide.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →How much faster can MTP be?
Google’s May 5, 2026 announcement reports “up to a 3x speedup” for Gemma 4 MTP drafters. Google says it tested with LiteRT-LM, MLX, Hugging Face Transformers, and vLLM. That is a vendor-reported maximum across tested configurations, not a guaranteed result or a prediction for every model, prompt, or machine. The announcement also reports up to about 2.2x locally for Gemma 4 26B MoE on Apple Silicon at batch sizes of 4 to 8. See Google’s Gemma 4 MTP announcement.
Architecture and workload can change the outcome. Google notes that verifying drafts for the Gemma 4 26B A4B MoE model can require loading expert weights, potentially offsetting the saved decoding work. At batch size one, there may be less opportunity to overlap that work; using more sequences can increase expert reuse. Google’s guide states: “At batch size 1 this overlap is limited, which is why the 26B A4B drafter may not yield speedups on hardware platforms without good parallelism.”
In practice, the result depends on the model, hardware, backend, and batch size. A draft that is often rejected, or overhead that is large relative to the target-model work saved, can reduce or eliminate the gain.
How to compare Qwen and Gemma MTP fairly
The cited sources do not provide a controlled, same-hardware benchmark comparing Qwen and Gemma MTP. They therefore do not establish that one family is faster or produces better results than the other. For a useful comparison, keep these conditions constant:
Free tools Windows power users keep installed
One-click scans. No signup required.
- Target task and prompt
- Hardware, precision, and memory settings
- Serving framework and version
- Batch size and output length
Measure both tokens per second and end-to-end latency. If available, record accepted draft length or acceptance rate as well: throughput alone can hide how effectively each system’s drafter is being used.
Best Value
Where to find the models and frameworks
Google says Gemma 4 MTP weights are available through Hugging Face and Kaggle under the same Apache 2.0 license as Gemma 4. Its announcement lists Transformers, MLX, vLLM, SGLang, and Ollama as ways to experiment. The vLLM Speculators documentation describes MTP fine-tuning and deployment workflows for supported native-MTP models.
These options do not imply that every listed framework supports every MTP model or checkpoint in the same way. Check the specific model and framework documentation before choosing a deployment path; support can change over time.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
