Anthropic's Memory and Dreaming for Continuous Agent Learning

The capabilities of Large Language Models (LLMs) have improved rapidly over the past few years, and AI agents are now capable of executing complex tasks that run for hours or even days. We have witnessed the birth of the Model Context Protocol (MCP), tool-calling capabilities, and various Agent SDKs. However, as pointed out by Mahesh, a product manager on Anthropic’s platform team, during a recent presentation, the next indispensable underlying primitives to move agent systems toward true “continuous self-learning” and long-horizon context management are Memory and the Dreaming mechanism.

When developers begin deploying thousands of parallel agents within enterprise environments, they face the same bottleneck: each agent acts as a one-off computational node, unable to learn from its peers’ mistakes or effectively accumulate knowledge about the project environment beyond a single session. With the newly launched Memory API and the Dreaming feature (in research preview), Anthropic proposes a systematic solution to this pain point.

Why Do We Need a Robust Memory Mechanism?

For continuously running agents, memory is the key to their evolution. Through memory, agents can understand task success criteria, record common mistakes, and evaluate the effectiveness of different strategies. Simultaneously, they can learn about their environments, such as the codebase they interact with, file states, and various dependencies.

More importantly, it enables cross-agent learning: agents can share their experiences within the same environment. For example, Rakuten reported a 90% drop in “first pass mistakes” among their internal knowledge agents after deploying this memory system, as agents could catch and share error information with the next iteration of agents. This not only improved intelligence but also significantly reduced token consumption and system latency.

Three Design Principles of the Frontier Memory System

To support the scaling of multi-agent systems, Anthropic adhered to three core design principles when designing the Memory API for Claude Managed Agents:

1. Maximize Intelligence by Default

Early agent memory (such as Claude.md from a year and a half ago) was often constrained by rigid tool calls or manual user annotations. However, the latest models (like Claude Opus 4.7) possess outstanding “file system-based memory” capabilities. Anthropic chose to delegate control by modeling memory as a virtual file system, allowing Claude to use its most proficient bash and grep tools to autonomously decide what content is worth recording, how to split files, and how to organize directory structures.

2. Scale with Multi-Agent Systems

Enterprise environments typically have hundreds of agents running concurrently and accessing shared states. This system introduces two critical properties:

• Permission Scopes: Agents can be given read-only access to an “organization-wide knowledge base” (like SOPs or best practices) and read-write access to a “task-specific workspace”, preventing core knowledge from being accidentally overwritten.
• Optimistic Concurrency: Using a content hash, an agent checks whether the state has been modified by other agents before updating memory, avoiding data conflicts.

3. Enterprise Control & Standalone API

To meet production environment standards, developers must have complete control. The API provides a detailed Version History and attribution metadata (recording which agent, in which session, and at what time the memory was modified). Furthermore, a standalone API design ensures enterprises can intervene outside the Managed Agents system—for instance, to perform PII (Personally Identifiable Information) scanning or to clone memory into external governance pipelines.

What is the Dreaming Mechanism?

Although the synchronous Memory API solves memory access for individual agents, Anthropic found efficiency bottlenecks in large-scale multi-agent systems: individual agents are often restricted by their own task perspectives, struggling to notice macro patterns across the system, and prone to creating redundant records.

To address this, Anthropic introduced the Dreaming process. This is a batch asynchronous process. You can configure Dreaming to trigger at specific times (e.g., via cron) or after tasks finish. It comprehensively reviews transcripts from recent agent sessions, identifies common mistakes and successful strategies, and automatically cleans, deduplicates, and verifies knowledge in the memory store.

The Real-World Benefits and Operation of Dreaming

In a legal scenario test by Harvey, deploying the Dreaming mechanism increased the task completion rate by 6 times. In an SRE (Site Reliability Engineering) automated debugging scenario, the power of Dreaming is even more evident:

When a system triggers consecutive CPU load alerts, different SRE agents investigate and record their findings separately. The overnight Dreaming job analyzes the logs from the past 7 days and discovers that multiple agents encountered a “60-second retry latency pattern following an upstream CPU spike”. Since a single agent only observes its own delay, it cannot deduce the 60-second rule; however, Dreaming can consolidate these cross-session patterns and automatically write a verified system debugging guide. The next day, when a new agent encounters the same alert, it can directly read this conclusion, avoiding repeated investigations.

Practical Recommendations for Engineering and Enterprises

1. Separate Task and Memory Objectives: Utilizing out-of-band mechanisms like Dreaming allows task-executing agents to focus on solving problems, delegating the objective of Memory Quality to background processes, and preventing prompt interference.
2. Scaling with Compute: Just as test-time compute improves reasoning quality, Dreaming allows agents to trade additional tokens for a stable, long-term cognitive foundation for the entire system. This is a necessary investment for large-scale enterprise agents.
3. Transition from Storage to Knowledge Networks: As systems run, the Memory Store should be viewed as a dynamically updating knowledge base rather than a static workspace buffer. Well-designed version control and attribution mechanisms will be your final defense when debugging agent behavior.

Related Reading

• AI Agent Development Guide
• Enterprise RAG Practice Guide
• Enterprise Agentic AI Governance Strategy