A Pretrained, Parametric Long-Term Memory
Memory Decoder
at Scale
Disentangle long-term memory and reasoning. Keep the base language model frozen, pretrain long-term memory as a standalone component, and scale memory independently.
1LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University
2Shanghai Artificial Intelligence Laboratory
3Electronic Engineering, Tsinghua University
Frozen Base ModelReasoninglanguage modeling & reasoning
Swappable MemoryLong-Term Memorygeneral or domain memory
Run SeparatelyCombine Once
01 · Core Idea
Long-term memory and reasoning need not remain entangled in the same parameters.
Decoder-only language models entangle long-term memory and reasoning within a single parameter set. We pretrain long-term memory as a standalone memory that can be scaled, swapped, and reused while the base model remains frozen.
02 · Swap Memory
Two base settings.
A wheel of memories.
General memory pairs with Pythia-410M. Biology, law, and finance memories all share the same frozen Qwen3-14B base.
language modeling
& reasoning
Frozen Parameters
Active MemoryGeneral6.9B parameters
1 / 4
Attached Memory ModuleGeneral MemoryPretrained on the deduplicated Pile for broad corpus knowledge.
→Pythia-410M + Memory17-task AVG37.34+7.48
Click a memory, use the arrows, or press ← / → on the wheel.
03 · Architecture
Two objectives.
One prediction.
The base model learns language modeling and reasoning; the standalone memory is trained to approximate retrieval-induced next-token distributions. At inference, both process the same context in parallel.
Figure 1Training and inference architecture
Base Model ParametersFrozen base model
Standard next-token pretraining supplies language modeling, computation, and reasoning.
Memory ParametersStandalone memory
A decoder learns the retrieval-induced distribution while retaining a next-token objective.
Plug-and-Play InferenceParallel composition
Exchange the active memory, then interpolate two next-token distributions.
Inference
pfinal = (1 − α) pbase + α pmemory
Two Forward Passes · One Final Distribution04 · Data Construction
Turn retrieval into
pretraining targets.
At pretraining scale, a corpus with N tokens yields N keys and N training queries, creating a joint indexing-and-search bottleneck. Compression, centroid-range sharding, and parallel GPU search enable memory pretraining over 300B training tokens.
Use the frozen Pythia-6.9B final hidden state as a 4096-d key for each context.
OPQ256 projects every key from 4096 to 256 dimensions before indexing.
HNSW selects IVF centroids, then maps each query to centroid-range shards.
GPU shards search independently; workers merge neighbors into a sparse next-token target.
Figure 2Distributed Faiss pipeline for constructing pretraining-scale kNN distributions
05 · Scale Memory
Scale memory,
not reasoning.
Our results show that a small frozen base model paired with a large memory offers a clear performance–efficiency advantage. Under matched training budgets, allocating capacity to memory consistently shifts the scaling frontier upward.
Figure 3General memory transfer across backbones
Open full size 06 · Results
The gain persists
across settings.
Switch between general and domain memory to compare their gains across backbone scales.
Pythia 1.4B+1.60
Pythia 2.8B+1.60
Pythia 6.9B+1.49
Average across 17 diverse benchmarks with matched-size memory.
general-memory task–scale combinations improve.
domain evaluations improve across all Qwen3 scales.
of the standard budget transfers memory across vocabularies.
07 · Resources
Paper, code, and models
08 · Citation
Cite this work.
If Memory Decoder at Scale supports your research, please use the following BibTeX entry.
@misc{wei2026memorydecoderscalepretrained,
title={Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory},
author={Rubin Wei and Jiaqi Cao and Jiarui Wang and Junming Zhang and Qipeng Guo and Bowen Zhou and Zhouhan Lin},
year={2026},
eprint={2607.27919},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2607.27919},
}