Baidu releases ERNIE 5.5 Flash Base, a 280B open-source MoE foundation model

Baidu releases ERNIE 5.5 Flash Base, a 280-billion-parameter open-source Mixture-of-Experts foundation model that activates 14 billion parameters per token. Pre-trained on 33.5 trillion tokens, the model is distributed as an un-tuned raw checkpoint under a custom license that excludes usage in the EU, UK, and South Korea. Self-reported benchmarks show ERNIE 5.5 leading rival base models like DeepSeek-V4-Flash-Base on 24 of 30 tests, including MMLU-Pro and HumanEval+.

Model release and licensing terms

• ▪The ERNIE 5.5 Flash Base License v1.0 is free for entities with under US$50 million in revenue but explicitly excludes usage in the European Union, United Kingdom, and South Korea
• ▪Baidu released ERNIE 5.5 Flash Base, a 280-billion-parameter text-only sparse Mixture-of-Experts foundation model that activates 14 billion parameters per token
• ▪Baidu released the FP8 and BF16 weights for ERNIE 5.5 Flash Base on Hugging Face and ModelScope under ERNIE 5.5 Flash Base License v1.0

Architecture and technical design

• ▪ERNIE 5.5 Flash Base incorporates a tokenizer with 201,216 entries that groups numbers right-to-left in chunks of up to three digits
• ▪ERNIE 5.5 Flash Base implements ECHO speculative decoding, which reuses the final backbone layer as a draft model adding 0.36 billion parameters excluded from headline counts
• ▪ERNIE 5.5 Flash Base features a hybrid attention mechanism across 43 layers, using Heavily Compressed Attention in 37 layers and Multi-head Latent Attention with sparse attention in 6 layers
• ▪ERNIE 5.5 Flash Base utilizes a Multi-Router Latent MoE architecture containing 512 routed experts running in a 2048-dimensional latent space alongside one shared expert

Pre-training and hardware optimizations

• ▪Baidu pre-trained ERNIE 5.5 Flash Base on 33.5 trillion tokens using the Muon optimizer with a 128,000-token training context window
• ▪Baidu trained ERNIE 5.5 Flash Base on FlashMask-4 kernels, reporting speedups on NVIDIA SM100 GPUs of up to 175% in forward pass and 74% in backward pass over FlashAttention-4 MaskMod

Benchmark evaluation results

• ▪Baidu self-reported that ERNIE 5.5 Flash Base scored 78.4 on MMLU-Pro, surpassing DeepSeek-V4-Flash-Base at 69.2, Hy3-preview-Base at 66.9, and MiMo-V2.5-Base at 65.7
• ▪DeepSeek-V4-Flash-Base scored 60.0 on HELMET-Rerank, outperforming ERNIE 5.5 Flash Base which scored 56.2 in Baidu's evaluation pipeline
• ▪Baidu reported that ERNIE 5.5 Flash Base achieved the top score on 24 out of 30 base-model benchmarks in Baidu's unified evaluation pipeline