NVIDIA Releases Gated DeltaNet-2 Linear Attention Architecture

NVIDIA AI released Gated DeltaNet-2 on May 24, 2026, a linear attention architecture that decouples erase and write operations in the delta rule to address memory editing challenges in fixed-size recurrent states. The 1.3 billion parameter model was trained on 100 billion FineWeb-Edu tokens and outperforms Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across language modeling, commonsense reasoning, and long-context retrieval benchmarks.

Gated DeltaNet-2 architecture design

• ▪NVIDIA released Gated DeltaNet-2 on May 24, 2026.
• ▪NVIDIA's Gated DeltaNet-2 decouples erase and write operations into a channel-wise erase gate b_t on the key axis and a channel-wise write gate w_t on the value axis.
• ▪Gated DeltaNet-2 was trained on 100 billion FineWeb-Edu tokens.
• ▪Gated DeltaNet-2 has 1.3 billion parameters.

Memory management mechanism

• ▪Linear attention squeezes the unbounded KV cache into a fixed-size recurrent state.
• ▪Prior delta-rule models like Gated DeltaNet and KDA use one scalar gate to control both erasing old content and writing new content.

Performance benchmarks

• ▪Gated DeltaNet-2 outperforms Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across language modeling, commonsense reasoning, and long-context retrieval.
• ▪Gated DeltaNet-2 achieves the largest performance gains on RULER S-NIAH and multi-key needle retrieval benchmarks.

1 source