Google Releases Multi-Token Prediction Drafters for Gemma 4 with 3x Speedup

Google AI released Multi-Token Prediction drafters for the Gemma 4 model family on May 6, 2026, using speculative decoding architecture to achieve up to 3x speedup in tokens-per-second for models like Gemma 4 31B without quality loss. The drafters address memory-bandwidth bottlenecks and are supported across LiteRT-LM, MLX, Hugging Face, and vLLM frameworks. The release follows Gemma 4's rapid adoption of over 60 million downloads in its first few weeks.

Memory-bandwidth bottleneck

• ▪During standard Large Language Model inference, the processor spends the majority of its operational time moving billions of parameters from Video RAM to the compute units just to generate a single token
• ▪Multi-Token Prediction drafters for Gemma 4 specifically target memory-bandwidth limitations that often hinder performance on consumer-grade hardware
• ▪Standard Large Language Model inference processes are often limited by the speed at which data can be moved rather than by the processor's calculation speed
• ▪The memory-bandwidth bottleneck leads to under-utilized compute resources and high latency, particularly on consumer-grade hardware where memory speeds may not match the demands of high-parameter models

Multi-token prediction architecture

• ▪The Multi-Token Prediction drafter is a specialized, smaller model that can perform predictions in significantly less time than it takes the larger target model to process a single token
• ▪Google released Multi-Token Prediction drafters for the Gemma 4 family of open models on May 6, 2026
• ▪The Multi-Token Prediction drafter utilizes idle compute cycles to predict several future tokens simultaneously
• ▪Multi-Token Prediction drafters for Gemma 4 allow models like Gemma 4 31B to achieve up to a 3x speedup in tokens-per-second

Speculative decoding mechanics

• ▪The Multi-Token Prediction drafters for Gemma 4 achieve speed increases with no degradation in reasoning logic or output quality
• ▪Google's Multi-Token Prediction drafters use a specialized speculative decoding architecture to address latency bottlenecks in AI inference
• ▪Speculative decoding with Multi-Token Prediction drafters decouples token generation from verification by pairing a heavy target model with a lightweight drafter model
• ▪In speculative decoding with Multi-Token Prediction drafters, once the drafter has proposed a sequence of tokens, the target model performs a verification step and accepts multiple tokens at once if the predictions are accurate

Framework compatibility

• ▪The broad framework compatibility of Multi-Token Prediction drafters allows developers to implement faster inference on diverse hardware, from Apple Silicon via MLX to cloud-based deployments via vLLM
• ▪Multi-Token Prediction drafters for Gemma 4 are supported across LiteRT-LM, MLX, and Hugging Face frameworks

Hardware deployment options

• ▪Google designed Multi-Token Prediction drafters for Gemma 4 to improve responsiveness across mobile devices, developer workstations, and the cloud
• ▪Multi-Token Prediction drafters for Gemma 4 make high-capability open models more responsive and viable for real-time applications without requiring specialized, high-end enterprise hardware for every use case
• ▪Multi-Token Prediction drafters for Gemma 4 enhance responsiveness for developers working on mobile devices, workstations, and cloud environments

Gemma 4 adoption metrics

• ▪The Gemma 4 family of models has seen over 60 million downloads in its first few weeks
• ▪Google describes Gemma 4 as delivering unprecedented intelligence-per-parameter
• ▪The release of Multi-Token Prediction drafters for Gemma 4 follows the model's rapid adoption of over 60 million downloads