Your Agent Doesn't Need a Memory Layer. It Needs Its History Back.

Last Updated: 2026-10-24

Adding a memory layer to an AI agent is usually the wrong first spend. A plain retrieval baseline over raw conversation history scored highest among the compared memory systems in six of seven benchmark settings, beating the strongest of them by 0.13 to 5.53 points, and it ingested a one-million-token history in 0.33 hours against up to 26.95 for the slowest system tested. The raw record is not the problem. Preserve it, measure against it, and only add structure where a measured gap remains.

What the Raw-RAG study actually found

A study published on arXiv on 22 October 2026 built the simplest possible alternative to agent memory: dense retrieval plus a reranker over the untouched conversation history, feeding the selected passages to a reader model. Across four public benchmarks, that baseline scored highest among the compared memory systems in six of seven benchmark-model settings, beating the strongest memory system by between 0.13 and 5.53 points. Ingesting a one-million-token history took it 0.33 hours, against 3.25 to 26.95 hours for the memory systems timed on the same workload.

The paper, "Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought" ^[1], tested its Raw-RAG baseline against the memory systems compared across LoCoMo, LongMemEval, MEME and MemoryAgentBench. The results table in the study's official repository ^[2] shows the pattern at its widest on MemoryAgentBench with DeepSeek-V4-Flash as the reader: Raw-RAG scored 53.41 against 47.88 for Memora, the strongest memory system in that setting, a 5.53-point lead. The systems compared include MemOS, EverMemOS, MemMachine and Memora, each of which rewrites conversation history into facts, summaries or graphs before the agent ever reads it.

The ingestion gap

The ingestion gap is the part an owner feels directly. On a cumulative one-million-token ingestion workload, the study timed Raw-RAG at 0.33 hours, the next-fastest completed run at 3.25 hours, and the slowest at 26.95 hours ^[1]. That is the difference between indexing a history in minutes and running an overnight, or multi-day, processing job before the agent can answer anything at all.

Why structured memory loses: the evidence it throws away

Adding structure to the raw baseline makes it worse

The paper Answering a question about what a customer decided after a price change needs the value, the change, the date and the link between them, and extracting facts or writing summaries tends to keep the value and drop the links. The study measured this directly: on questions whose answer value was present in the retrieved context, raw retrieval also carried a complete supporting path 62.34 percent of the time, against 56.70 percent for the strongest memory system.

The study ran an ablation that makes the point brutally. Starting from its own raw baseline on MemoryAgentBench, which scored 48.32, adding atomic-fact extraction produced 28.70, adding summary augmentation 42.51, and adding graph expansion 42.54 ^[2]. Every "improvement" made the baseline worse. Removing the cross-encoder reranker also cut the score to 44.40, which shows the machinery that does the work here is retrieval quality, not memory structure.

Better retrieval cannot repair lost information

The sharpest experiment in the paper asks whether better retrieval over a memory store can repair what ingestion lost. It cannot. Even when a memory system was given complete access to its own transformed store, an accuracy gap against raw evidence remained ^[1]. Information discarded during ingestion stays discarded, no matter how well you search for it afterwards.

Where memory still earns its place

What memory is still for

This is a baseline result, not a proof that memory products are useless. One of the seven benchmark-model settings went the other way, with MemMachine leading Raw-RAG by 1.52 points, and feeding the whole history to the reader beat everything in that setting, at 63.17 against Raw-RAG's 43.19 ^[1]. The study's own limitations section is explicit that the evidence covers offline conversation on public benchmarks, not interactive or operational systems, and that raw-history approaches raise storage and privacy concerns ^[1].

Three independent teams, one conclusion

The finding lands on receptive ground. A second peer study published the same week compared six explicit memory systems against raw-history retrieval on two benchmarks and found no consistent advantage for memory construction, despite construction costs reaching hours per conversation ^[3]. A third, reproducing the LightMem memory system, found that changing only the retriever over a fixed memory store moved answer accuracy from 58.1 to 75.5 percent, and that raw-turn retrieval generally matched constructed memory as the answering-token budget increased ^[4]. Three independent teams, one conclusion: the complexity is arriving before the evidence.

Memory layers still have a defensible role. Where deletion obligations apply, a raw transcript that must be produced on request is a liability rather than an asset. Where context budgets are tight, a compressed store earns its keep. And where a measured gap persists after the baseline runs, that is evidence, not fashion. What the study removes is the default: memory as an automatic line item in every agent architecture.

The decision this hands an owner

The practical decision is sequencing, not refusal: run the cheap baseline before buying the memory layer. Concretely, it looks like this. Export the conversation logs the agent already produces. Index them with ordinary dense retrieval and a reranker, the same commodity components the study used. Pull twenty real questions the business actually needed answered from past conversations, run them against the baseline, and record the accuracy. Only where that number falls short is there evidence that structure would help, and the failed questions themselves say which structure.

In our own agent builds on a self-hosted server, the raw record is treated the same way this study treats it: the unprocessed log is the asset, and every layer added on top of it has to justify itself against a measurement rather than an architecture diagram. It is the same instinct this site's method applies everywhere. In discovery workshops, owners rarely arrive asking for a baseline; they arrive asking which component to add. The useful reframe is that the cheapest component, retrieval over what already exists, is the one worth measuring first.

The artefact this produces costs almost nothing: a dated baseline run, with its question set and its measured accuracy, kept next to the architecture decision it informed. It answers a question that otherwise has no answer on record: how did you know the memory layer was necessary? With a measurement instead of an invoice.

Sources

1. [1] "Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought", arXiv 2610.30755, published 22 October 2026: https://arxiv.org/abs/2610.30755
2. [2] Official Raw-RAG repository, result tables and ablations: https://github.com/Litmeb/RAW-RAG
3. [3] "Do Conversational Agents Need Explicit Memory? A Controlled Comparison with Raw-History Retrieval", arXiv 2610.24671, published 17 October 2026: https://arxiv.org/abs/2610.24671
4. [4] "Reproducing LightMem: Naive RAG Is Just as Good for Memory Management", arXiv 2607.29104, published 17 October 2026: https://arxiv.org/abs/2607.29104