New Research Calls Out “Paused-Time” Benchmarks, Pushing Real-Time Standards for AI Agents

In a growing share of the tasks developers want artificial-intelligence agents to handle, being right is no longer enough. Customer-service bots are expected to respond while a user keeps typing; cyber-defence tools must act before an intrusion spreads; robotics systems have to move as the world moves. Yet the dominant way the industry evaluates “LLM agents” still treats time as optional: the environment is paused while the model reasons, calls tools, and crafts an answer.

New research introduces a benchmark designed to make that assumption impossible. Called RealtimeBench, it evaluates language-model agents in interactive game-like environments that keep running during inference, forcing systems to trade off deliberation against speed. The benchmark contains 180 tasks spanning six game types, and its premise is straightforward: in a live setting, a correct move delivered late can be indistinguishable from a wrong one.

When latency becomes the main failure mode

LLM agents typically operate in loops: observe the state, plan, potentially invoke tools (search, code execution, memory), then act. Each step adds latency. In “paused-world” evaluation, the agent is rewarded for eventual correctness regardless of how long it took to decide. RealtimeBench flips that by continuing the game clock while the model thinks.

That shift matters because the time cost of “better reasoning” is not abstract. In many agent designs, a more capable model uses longer chains of thought, makes more tool calls, or explores more alternatives—improving accuracy but increasing response time. RealtimeBench treats latency as part of the task, not a side metric.

A telling head-to-head: smarter loses to faster

To illustrate the impact, the researchers ran paired evaluations comparing two systems with different profiles: a higher-performing but slower model (described as GPT-6.1-sol) and a faster model (Jev). When the game is paused—mirroring traditional evaluation—GPT-6.1-sol outperforms Jev. But when the environment runs in real time, that advantage can evaporate or reverse: the faster system gains ground because it can act while the other is still computing.

This result highlights a growing tension in agent development. Many teams have focused on maximizing benchmark scores on static or turn-based tasks, where the optimal approach is often to think longer. RealtimeBench suggests that this can mislead practitioners about real-world performance, where delays are punished by the environment itself.

Designing agents “under a live clock”

The benchmark’s broader message is that agent training and evaluation need to incorporate clock time explicitly. In real deployments, systems must manage not only what to do but when to do it—sometimes opting for a decent action now rather than a perfect one later.

Researchers point to several strategies suggested by the live-clock framing:

• Adaptive compute: spending less time on easy moments and reserving deeper reasoning for critical states.
• Action while thinking: issuing low-risk “keepalive” moves (or safety-preserving actions) while planning higher-value steps.
• Tool-call budgeting: limiting external calls or parallelising them, because each call can become a delay that the environment exploits.
• Latency-aware policies: explicitly learning thresholds where speed dominates accuracy.

In effect, RealtimeBench pushes the field toward a more realistic conception of intelligence in interactive systems: competence includes the ability to operate within deadlines.

Why games are a useful proving ground

Games offer controllable, repeatable environments with clear success metrics and a natural notion of time pressure. By using multiple game types and a sizable task set, the benchmark aims to prevent “overfitting” to a single timing pattern, and to capture diverse real-time dynamics—situations where waiting changes the state, removes options, or increases risk.

The research also underscores an uncomfortable reality for AI product builders: improvements that look impressive in offline evaluations may not translate to time-sensitive settings. An agent that can solve a complex task in principle might still fail in practice if it cannot keep up with the pace of the environment.

Why it matters

As AI agents move from chat windows into security consoles, logistics systems, financial workflows and robotics, latency will increasingly define whether an action is useful. Benchmarks that freeze the world during inference can reward behaviours that break the moment the clock starts. RealtimeBench is part of a broader shift toward measuring AI the way it will be used: in motion, under pressure, and without timeouts.