DEV Community

Cover image for AML Cycle 2 Results Are Approaching. Which Memory Failure Will You Look For First?
Agent Memory Leaderboard
Agent Memory Leaderboard

Posted on

AML Cycle 2 Results Are Approaching. Which Memory Failure Will You Look For First?

A memory leaderboard gives you scores. Your application gives you a reason to care about them.

Perhaps your agent keeps using a decision your team reversed last week. Perhaps it retrieves a successful fix from the same repository—but applies it to a task where that fix no longer belongs. Or perhaps it remembers a screenshot’s caption while missing the visual detail the next question depends on.

These failures lead to different questions when a new leaderboard arrives.

With Agent Memory Leaderboard (AML) Cycle 2 results approaching, here are three questions we suggest bringing to the results.

1. Can the system handle a changing history?

Imagine a customer changes their delivery address, but only for one order.

A useful memory system needs to preserve the update’s scope. Applying the temporary address to every future order would be wrong. Ignoring it would also be wrong.

When examining textual memory results, consider the failure your application actually faces: missing facts, conflicting updates, temporal relationships, or instructions applied outside their intended scope.

A strong overall score is useful. The capability profile helps you decide what to investigate next.

2. Does past engineering experience help the next task?

A coding agent finds a previous workaround in the repository’s history. That workaround succeeded before an API changed.

Should it reuse the solution, adapt it, or reject it?

The practical value of coding memory depends on selecting experience that fits the current task. Returning more history can also introduce more distractions.

AML Cycle 2’s Coding Memory track examines the retrieval, filtering, and reuse of historical engineering experience under relevant and noisy conditions. When reading those results, ask which conditions resemble the history your own agents encounter.

3. Can the system retrieve the evidence a question depends on?

Suppose a user shares a screenshot alongside a written description. Later, a question depends on something visible in the image but absent from the description.

Finding the conversation is only part of the task. The needed evidence must survive storage and retrieval, and remain available for downstream use.

The Multimodal Memory track adds this image–text perspective to the evaluation.

What AML holds consistent

AML Cycle 2 evaluates Textual, Coding, and Multimodal Memory in separate tracks, with separate Open-source Methods and Commercial Products leaderboards.

Participating systems provide Add and Search interfaces. AML controls the downstream answer generation or agent execution and scoring under the relevant evaluation contract.

That shared setup supports comparison within each track. Choosing a system for a particular application still requires examining the conditions and capabilities behind its score.

Bring one failure case to the leaderboard

Before the results arrive, write down one memory failure you want your agent to stop making.

Then use it to guide your reading:

  • Which published capability is closest to that failure?
  • What does the evaluation test—and what would still need testing in your application?
  • Which result would make you investigate a system further?

AML Cycle 2 results are approaching.

Explore the competition and release updates:

https://agentmemoryleaderboard.ai/competition/

If you could check only one capability first, what would it be—and what happened in your own workflow that made it matter?

Top comments (0)