ALTK-Evolve: learned memory tops τ²-bench leaderboard for customer service tasks without frontier models
In brief
- An agent using GPT-4.1 achieves 67.5% Pass^4 on τ²-bench retail, outperforming specialized models.
- ALTK-Evolve generates behavioral rules from agent errors and successes without fine-tuning.
- Learned rules reduce response time and improve reliability, even with frontier models like GPT-5.5.
IBM Research unveils ALTK-Evolve, a method that generates behavioral rules for customer service agents from their own successes and failures, without updating model weights. According to the announcement, an agent using GPT-4.1 achieves 67.5% Pass^4 on the τ²-bench retail benchmark, outperforming all specialized models in a snapshot taken on October 12, 2026.
The technique relies on analyzing the agent's attempts during training tasks. GPT-4.1 extracts rules such as customer authentication or return handling, then embeds them into the model's prompt for subsequent conversations. Three rule configurations are tested: no rules, a high-confidence core (support ≥ 3), and all extracted rules. Gains vary by domain: retail benefits from the full set, while aviation favors a compact core for reliability.
This approach also improves frontier models like GPT-5.5, with a 15-point reliability increase on aviation tasks. Learned rules reduce response time and compute costs without requiring fine-tuning or human annotation. They offer a lightweight alternative to traditional methods for adapting agents to sector-specific policies.
What it changes for you
- Retail & E-commerceCustomer service agents can be optimized without costly fine-tuning by integrating learned rules to comply with sector policies and improve reliability.
- Transport & LogisticsLearned rules enhance the handling of complex tasks like refunds or rebookings, with measurable reliability gains on benchmarks.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face - Blog.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error