Number 2 of 12 · November 22, 2026

How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents

The right amount of agent memory depends on the model: a strong model keeps gaining from more guidelines, while a weaker one did better with a small selected subset.

What they found

The authors built a pipeline where an SI agent learns from its own past runs. It mines one to three short lessons per episode, then merges them into fewer guidelines. Each guideline records how many lessons it merged. On the AppWorld benchmark, with models from 40-92% baseline task completion, the central finding is that the right amount of memory is a property of the model. DeepSeek-V3.2 gained +9.5 points of task completion and +16.1 points of scenario completion with its full 191-guideline set. gpt-oss-120b started at 39.9%, reached 51.2% with all 199 guidelines, and reached 55.4% (mean of two runs) with a selected subset of about 29.

Why it matters

Many teams give agents memory and assume more is better. This paper says that is not always true, and the best amount may differ by model. The authors also share what went wrong in practice: a high-support guideline that backfired, provenance silently dropped during consolidation, a swallowed API error that failed every task while the log looked plausible, and a baseline confound that nearly gave a misleading comparison with a prior method. They turn these into recommendations for teams deploying agent memory.

How it changes the race

This is a practical, deployment-focused result rather than a new capability. It suggests teams should tune how much memory each model gets, and it may save effort for those building agents on weaker models. The pipeline is open source in altk-evolve and is deployed in IBM's CUGA agent and watsonx Orchestrate.

Safety and control

The reported failures, such as an API error that was swallowed and a log that looked fine while every task failed, show that agent evaluations can mislead quietly. Careful checking of logs and baselines matters for trusting agent results.

The abstract, by the authors

LLM agents on long-horizon tasks fail in recurring, procedural ways. They mis-paginate an API, resolve the wrong entity, or answer when no answer was asked. We built a pipeline that lets an agent learn from its own history. It first mines short lessons from the agent's own trajectories, one to three per episode. It then consolidates those lessons into a smaller set of guidelines, where each guideline records how many lessons it merged. At inference the agent receives either all guidelines or a selected subset. The pipeline ships in the open-source altk-evolve library and is deployed in IBM's CUGA agent and watsonx Orchestrate platform. We report what we learned building, measuring, and productizing it. Our primary instrument is the AppWorld benchmark, across models spanning 40-92% baseline task completion. The central finding is that the right amount of memory is a property of the model, not the memory. A strong model keeps improving as guidelines are added. DeepSeek-V3.2 gains +9.5 points of task completion and +16.1 points of scenario completion with its full 191-guideline set. A weak model gains less from the same full set than from a small selected subset. gpt-oss-120b starts at 39.9% task completion. It reaches 51.2% with all 199 of its guidelines but 55.4% (mean of two runs) with a selected subset of about 29. Around that result we report what went wrong: a high-support guideline that backfired, provenance silently dropped during consolidation, a swallowed API error that failed every task while producing a plausible log, and a baseline confound that nearly produced a misleading comparison with a prior method. We distill these into recommendations for teams deploying agent memory.

Quoted from arXiv:2611.26361. The research and its words belong to its authors.

The authors

K. R. Jayaram, Vatche Isahagian, Aswath Vaithinathan Aravindan, Vinod Muthusamy, Punleuk Oum, Gaodan Fang, Michal Ulewicz, Gegi Thomas, Evelyn Duesterwald

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.