StarBench:面向智能体多模态决策与信息寻求的回合制 RPG 基准

📊 论文图表(共 4 张)

展开查看 4 张图

‹ ›

TLDR

现有VLM在真实游戏客户端中能否像人类一样从截图生成连贯的低层按键操作并在受挫时主动寻求信息仍是开放问题。本文基于《崩坏：星穹铁道》构建StarBench基准，涵盖8项战斗任务，在统一任务与指标下设立直接控制与工具辅助控制两种评测模式，并加入ask-or-act诊断衡量是否/何时请求简短指引。实验显示当前VLM在直接控制模式下感知到控制的保真度存在显著差距，而合理的信息寻求行为与成功率提升正相关。

Abstract

Human players do more than press buttons: they ground what they see on screen into precise keyboard-mouse actions and, when stuck, they seek information before trying again. We ask whether current vision-language models (VLMs) can do the same. Despite encouraging results under simplified control or tool scaffolds, human-like play in a real client - mapping raw screenshots to temporally coherent low-level actions while deciding when to ask for guidance - remains an open challenge. We introduce StarBench, a turn-based RPG benchmark derived from Honkai: Star Rail that targets these two human-like competencies: multimodal decision-making from pixels to actions and agentic information seeking. StarBench standardizes evaluation across eight combat tasks and two regimes with shared tasks and metrics: (i) direct control, where agents receive only screenshots and must emit low-level primitives (click and keypress) with no semantic hints; and (ii) tool-assisted control, where higher-level intents can be mapped to primitives by detectors and OCR outputs provide optional textualized observations to ease UI grounding. To mirror human practice, StarBench also includes an ask-or-act diagnostic that measures whether and when agents choose to request brief guidance before proceeding, and how that choice affects subsequent performance. We report reference baselines for contemporary VLMs and a human reference. Results expose sizable gaps in perception-to-control fidelity in the direct regime, while showing that judicious information seeking correlates with improved success, establishing StarBench as a reproducible yardstick for agentic information seeking and multimodal decision-making in real-client play.

动机

真实游戏客户端中从原始截图映射到时序连贯的键盘鼠标底层操作，且能在受挫时主动寻求信息，是VLM尚未被验证的人类级游戏能力。

方法

在《崩坏：星穹铁道》真客户端中定义8项战斗任务，设立直接控制（截图→点击/按键）与工具辅助控制（detector+OCR文本化观察）两种统一协议，并加入ask-or-act诊断量化何时请求简短指引。

结果

直接控制模式下VLM的感知到控制保真度存在显著差距，而合理的信息寻求行为与后续任务成功率正相关，具体基线数字原文未在摘录中给出。

结论

StarBench为真实客户端中的智能体信息寻求与多模态决策提供了可复现的衡量基准，揭示了VLM在底层控制与主动求助决策上的不足。