DeepSeek’s Engram model beats a matched MoE baseline on reasoning and code
A DeepSeek-AI and Peking University paper introduces Engram, a memory module that uses hashed N-gram lookups for static patterns alongside a model’s neural computation. In the authors’ tests, their 26.7-billion-parameter model outperformed a matched mixture-of-experts baseline across benchmarks, including reasoning, code and maths; the paper reports that a 100-billion-parameter memory table added less than 3% inference-throughput overhead when held in host memory.
Bottom line — The authors report a 5-point gain on BBH and a 3-point gain on HumanEval, but the results remain their own benchmark tests.
Go deeper 7
-
The ACL paper describes the method as Deep Sparse Embedding (DSE); the arXiv and other articles call it Engram.
-
In the 27B comparison, the team reallocated parameters from 72 routed experts to 55 and added a 5.7B memory module, keeping the total at 26.7B.
-
The authors report gains over the MoE baseline of 5 points on BBH, 3.7 on ARC-Challenge and 2.4 on MATH.
-
For long-context tests, the paper reports 97.0 versus 84.2 on Multi-Query NIAH in a comparison designed to match pre-training loss.
-
The paper says the 100B-parameter table was offloaded to host DRAM in a prototype inference test; throughput fell 1.9% on a 4B backbone and 2.8% on an 8B backbone.
-
The memory is learned during training and uses static hashed lookups; the paper does not present it as an editable live database.
-
The authors note that their experiments use a specific backbone and training recipe, and that results after supervised fine-tuning and reinforcement learning remain underexplored.