diff --git a/docs/agent-evaluation.md b/docs/agent-evaluation.md index ae63ae30c..71a179cf4 100644 --- a/docs/agent-evaluation.md +++ b/docs/agent-evaluation.md @@ -81,6 +81,12 @@ uv run --locked --no-sync python -m scripts.evaluation --live \ 三份报告的 `reported_models` 都是 `gemini-3.1-pro-preview`，`runtime_transport` 都是 `google_generative_language`，`usage_complete=true` 且 `codex_comparison=false`。这证明更强模型已经能在当前隔离生产 Agent 图中完成这组三个固定 API 场景；仍不能据此宣称浏览器、命令行、长任务排队或整体智能已达到 Codex。后续 OAuth 配对结果见下节。 +### 命令行与浏览器能力实测 + +`command_execution` 场景按场景显式开启原生 CLI 的 `shell_tool`、`unified_exec` 和 `shell_snapshot`，代理保留 `functions.exec_command` 与 `functions.write_stdin`；MoviePilot 侧注入生产 `ExecuteCommandTool`，仅允许任务给定的固定命令和临时工作目录。配对报告 `/tmp/moviepilot-agent-round-luna-oauth-command-pair-final.json` 的 `pair_valid=true`、`both_passed=true`，harness 指纹为 `c409ee1ca7899ccfcefd38832d6a343f024f8fc67fb65f14c3eec08552203257`：两侧均以退出码 0 得到 `MOVIEPILOT_COMMAND_OK`，没有业务副作用。 + +`browser_navigation` 场景由 MoviePilot 生产 `BrowseWebpageTool` 操作回环动态页面，报告 `/tmp/moviepilot-agent-round-luna-oauth-live-browser-final.json` 通过，3 次浏览器回执在点击后观察到 `BROWSER_OK`。`codex exec 0.153.4` 在探针 `/tmp/moviepilot-agent-round-luna-oauth-native-probe-browser-final.json` 中即使显式开启 browser/computer feature 也没有广告浏览器动作，因此浏览器原生配对保持 blocked；这不是把 CLI 的缺失能力改判为通过。 + 因此当前真实 Gemini 结果证明了工具合同读取和未知结果诚实边界已经能被实测，但不能宣称达到 Codex 的整体智能水平。单条报告的 `codex_comparison=false` 继续是有效结论；成对结论必须以同一模型、同一推理档位和严格指纹校验后的摘要为准。 ### `thought_signature` 与真实 MoviePilot Agent @@ -119,7 +125,7 @@ uv run --locked --no-sync python -m scripts.evaluation \ 只有显式 `--live` 或 `--native` 才会调用真实模型并产生费用。默认读取 `~/.codex/config.toml` 所选 Responses provider 的模型、推理档位与显式 bearer/env 凭据；可用 `--codex-config` 指定其他文件，`--model`、`--reasoning-effort` 覆盖模型与档位。不会借用其他服务的登录凭据，也不会自动降低被供应商拒绝的参数。报告同时保留请求模型与供应商返回的模型标识；本地 Codex 配置不能证明运行中的 MoviePilot 使用相同配置。 -调用配置经私有标准输入传给 worker，凭据不进入命令行、提示词或报告。worker 只继承必要的平台环境，先创建临时 `CONFIG_DIR`，再导入后端；每轮拥有独立回执库、记忆、会话和工具实例。生产 `process/_create_agent`、Skills、计划、权限、持久回执、工具输出预算、压缩和子代理仍按真实路径执行。评测主目录包含 `moviepilot_api`、生产 `read_skill`、受临时 Agent 根约束的 `read_file`、计划、子代理和回执查询；API transport 拒绝任何外部目标及场景外 operation。插件、外部 MCP、通知、任意 shell 和浏览器工具仍不开放，此受控目录不代表默认部署工具全集。 +调用配置经私有标准输入传给 worker，凭据不进入命令行、提示词或报告。worker 只继承必要的平台环境，先创建临时 `CONFIG_DIR`，再导入后端；每轮拥有独立回执库、记忆、会话和工具实例。生产 `process/_create_agent`、Skills、计划、权限、持久回执、工具输出预算、压缩和子代理仍按真实路径执行。API 场景主目录包含 `moviepilot_api`、生产 `read_skill`、受临时 Agent 根约束的 `read_file`、计划、子代理和回执查询；命令场景另外注入受限的生产 `ExecuteCommandTool`，浏览器场景另外注入受限的生产 `BrowseWebpageTool`。API transport 拒绝任何外部目标及场景外 operation；这些受控目录不代表默认部署工具全集。 共享回调在请求前执行硬调用上限，覆盖主模型、选择、摘要和子代理；SDK 自动重试关闭。单请求超时最多 120 秒，全轮和独立进程另有期限。每次请求的输出 token 及运行器上下文上限被记录；上下文上限是测试参数，不表示模型真实最大窗口。当前固定为 128000 tokens。 @@ -144,7 +150,7 @@ uv run --locked --no-sync python -m scripts.evaluation --native \ 适配器目前只核对 `codex-cli 0.153.4`，可用 `--codex-executable` 指定文件。其他版本或二进制目录中没有指定模型时明确拒绝，不换模型或套用其他模型的元数据。`--native-probe` 的 `probe_ready` 只代表配置和目录检查，不是任务通过；原生进程预期收到本地探针错误并退出，模型调用数仍为零。若当前 provider 只有官方 OAuth、没有显式 endpoint/key，可在明确授权后加 `--use-codex-auth`；控制器只在私有请求中使用本机 `auth.json` 的访问令牌和账户标识，默认模式不会借用登录态。 -每次创建空白临时工作目录；场景源码、初态、账本、oracle、模型凭据留在控制器。原生客户端使用局部随机令牌连接回环模型代理和 MCP 假世界，不继承业务配置、真实模型令牌或其他服务环境。客户端忽略用户配置和规则文件，关闭宿主技能、插件、浏览器、文件查看及 shell 等能力，采用只读沙箱、never 审批与有界退出；仅对当前回环 evaluation 服务的三个假工具显式设置 `approval_mode=approve`，避免原生客户端拒绝已授权的假业务动作。不修改用户 HOME、CODEX_HOME 或现有配置。全局 AGENTS 仍可能由原生客户端加载，因此代理在发送给模型前仅移除规范的独立 AGENTS 用户块，保留原生基础说明和任务文本，并记录被移除块的长度与哈希。 +每次创建空白临时工作目录；场景源码、初态、账本、oracle、模型凭据留在控制器。原生客户端使用局部随机令牌连接回环模型代理和 MCP 假世界，不继承业务配置、真实模型令牌或其他服务环境。客户端忽略用户配置和规则文件，采用只读沙箱、never 审批与有界退出；API 场景关闭宿主浏览器和 shell，命令场景只按条件开启并保留 `functions.exec_command`/`functions.write_stdin`。仅对当前回环 evaluation 服务的三个假工具显式设置 `approval_mode=approve`，避免原生客户端拒绝已授权的假业务动作。不修改用户 HOME、CODEX_HOME 或现有配置。全局 AGENTS 仍可能由原生客户端加载，因此代理在发送给模型前仅移除规范的独立 AGENTS 用户块，保留原生基础说明和任务文本，并记录被移除块的长度与哈希。 原生循环、计划和协作工具仍由 Codex 执行。当前模型可固定使用 Code Mode，单独关闭 feature 无法改变它；适配器通过 `codex debug models --bundled` 读取当前二进制自带目录，只投影指定模型的 `tool_mode=direct`，保持其余字段、`base_instructions` 和 `model_messages`，记录前后指纹。随后代理限制实际广告目录为计划/协作与 evaluation MCP 工具，兼顾请求中的动态目录，并在每个完整响应事件交给客户端前再次拒绝目录以外的调用。这是明确投影过工具模式与目录的原生 Codex 对照，不代表默认部署的完整产品工具环境。 diff --git a/docs/refactor/agent-codex-parity-plan.md b/docs/refactor/agent-codex-parity-plan.md index ce4d46827..8e7bbfe90 100644 --- a/docs/refactor/agent-codex-parity-plan.md +++ b/docs/refactor/agent-codex-parity-plan.md @@ -21,7 +21,7 @@ Google 的 OpenAI-compatible 层会丢失 Gemini 3 工具回复中的 `thought_s | C1.4 终端工具完整闭环 | 已完成基线 | `run/pipe/PTY` 共用 shell、cwd、login 和 UTF-8 策略；stdin 写入、EOF、分页、interrupt/kill、超时和进程组收尾有真实进程测试 | | C1.3 终端任务作用域 | 已完成 | 终端归属由宿主对象身份决定；定时运行、会话、子任务和内部工具管理器隔离；封口先于清理，排队或运行中的命令都不会在任务结束后迟到启动 | | S2.3 运行中消息排队与 WebAgent 输入 | 已完成本轮实现 | 运行中仍可提交新消息；消息按会话原子入队，在下一次模型调用边界注入真实 `HumanMessage`；SSE 报告 queued/applied，停止后不再派发后续工具；剩余边界由真实长任务回归继续覆盖 | -| 浏览器能力对齐 | 动作与作用域已实现，真实配对验证未完成 | 导航、页面读取、点击/输入、等待、截图和失败收口使用真实浏览器状态；工具清单、权限、超时、重试和会话生命周期与命令行能力同样可观测；仍需可用供应商和原生 harness 做同场景验证 | +| 浏览器能力对齐 | MoviePilot 真实运行已验证；原生 CLI 配对受能力缺口阻塞 | 导航、页面读取、点击/输入、等待、截图和失败收口使用真实浏览器状态；本轮本地动态页面场景已由生产 `BrowseWebpageTool` 完成。`codex exec 0.153.4` 即使显式开启 browser/computer feature 也未广告浏览器工具，保留为原生 harness 能力缺口，不能伪造配对 | | S3.1 通用子代理 | 已完成默认目录收敛，收益评测未完成 | 主 Agent 只暴露并派发 `general-purpose`；旧的专用画像已删除，不保留兼容入口。仍需用 held-out 任务验证通用派发的收益、授权、工具角色、终端分享和副作用边界 | ## 每轮硬门禁 @@ -49,8 +49,8 @@ uv run --locked --no-sync python -m scripts.evaluation \ ## 工具与 Harness 对齐清单 -- **命令行**：普通一次性命令与后台终端分别支持 pipe/PTY、共享 cwd/shell/login/环境和 UTF-8；输入写入、空写入、EOF、分页、短写、interrupt、kill、超时、取消和进程组收尾都有结构化终态。 -- **浏览器**：工具目录必须明确导航、读取、交互、等待和截图的动作与权限；浏览器会话、页面状态和失败重试由宿主持有，不能由模型字符串冒领。 +- **命令行**：普通一次性命令与后台终端分别支持 pipe/PTY、共享 cwd/shell/login/环境和 UTF-8；输入写入、空写入、EOF、分页、短写、interrupt、kill、超时、取消和进程组收尾都有结构化终态。新增 `command_execution` 场景只在原生 CLI 真实广告 `functions.exec_command`/`functions.write_stdin` 时按场景开启，并把 MoviePilot 生产 `ExecuteCommandTool` 绑定到临时目录；PTY 长会话仍需同条件原生配对。 +- **浏览器**：工具目录必须明确导航、读取、交互、等待和截图的动作与权限；浏览器会话、页面状态和失败重试由宿主持有，不能由模型字符串冒领。新增 `browser_navigation` 场景使用回环动态页面验证生产工具；原生 CLI 探针没有广告浏览器工具，原生配对保持 blocked，等待可用的 Codex browser harness。 - **子代理**：主 Agent 自动选择通用子代理；专用画像只有在 held-out 任务上证明提高成功率、减少调用或降低副作用风险时才保留。子代理不能自行发送消息、执行高影响写操作或继承兄弟任务句柄。 - **API Skill 与合同读取**：`skills/moviepilot-api/SKILL.md` 只负责路由索引和工作流；`api/*.md` 各自包含完整操作合同及该类别需要的 Body Models，不再依赖单独的 `api/models.md`。真实 Agent 先用 `read_skill` 得到 supporting-file 清单，再用同一个工具的 `file=api/.md` 参数按需读取类别文档；评测目录必须提供同一条链路，不能只返回文件名而不给模型读取能力。operation 输入错误还要返回该 operation 的允许字段、必填字段和类型约束，帮助模型纠正后重试。 - **长任务与新消息**：入站消息在任务运行时仍可接受并进入有界队列；应用到下一模型边界时必须保留 tool-call/tool-result 配对和取消语义，不能只把文本拼到系统提示词。 @@ -73,4 +73,5 @@ uv run --locked --no-sync python -m scripts.evaluation \ - 本轮：真实测评切换到 `gemini-3.1-pro-preview + high`，官方 Google 主机改用生产同源的 `langchain-google-genai` 原生工具通道并保留 `thought_signature`。`dedup_existing`、`unknown_download`、`honest_unknown` 三个固定场景均通过（报告分别为 `/tmp/moviepilot-agent-round-gemini31native-dedup.json`、`/tmp/moviepilot-agent-round-gemini31native-unknown.json`、`/tmp/moviepilot-agent-round-gemini31native-honest.json`）；这只是 API 假世界的模型行为证据，仍未形成与原生 Codex 的配对比较。 - 本轮第二阶段：用同一 `gpt-5.6-luna + max`、同一场景和预算显式启用 Codex OAuth，首次生成有效成对摘要 `/tmp/moviepilot-agent-round-luna-oauth-dedup-pair.json`。MoviePilot live `/tmp/moviepilot-agent-round-luna-oauth-live-dedup-5.json` 因最终 infohash 少两位而未通过；原生 Codex `/tmp/moviepilot-agent-round-luna-oauth-native-dedup-8.json` 通过。摘要 `pair_valid=true`、`both_passed=false`，证明真实生产图与 Codex 仍有可观测的终态可靠性差距；下一轮先针对精确最终报告和未知结果收口做多次重复，再扩展浏览器、终端和中途消息场景。 - 本轮第三阶段：修正原生动态工具搜索对 `multi_agent_v1` 协作命名空间的投影，正向保留 `wait_agent`、`resume_agent` 等原生控制动作，负向继续拒绝 `read_file`。在同一 Harness（`71be4c436acc5dc2d5a0cd6996c3ca10688a9c4ba98fd11594e536f7aa440c22`）下，`dedup_existing` 配对 `/tmp/moviepilot-agent-round-luna-oauth-dedup-pair-13.json` 与 `unknown_download` 配对 `/tmp/moviepilot-agent-round-luna-oauth-unknown-pair-14.json` 均 `pair_valid=true`、`both_passed=true`；MoviePilot 分别使用 6/6 次模型调用，原生 Codex 使用 10/14 次。`honest_unknown` 配对 `/tmp/moviepilot-agent-round-luna-oauth-honest-pair-16.json` 仍为 `both_passed=false`：MoviePilot 在 9 次调用后如实保留下载未核验，原生 Codex 达到 16 次上限后没有最终 JSON，产生 `invalid_final_report`。这保留了真实失败边界，不能等同整体 Codex 智能已完成。 -- 浏览器、真实命令行/PTY 和 WebAgent 中途消息排队已有确定性实现，但仍缺少与原生 Codex 在同一模型、同一场景下的真实配对证据，不能把这些能力标为“已对齐”。 +- 命令行与浏览器能力阶段：`command_execution` 在同一 `gpt-5.6-luna + max` 和独立 Codex OAuth 下已形成真实配对 `/tmp/moviepilot-agent-round-luna-oauth-command-pair-final.json`，`pair_valid=true`、`both_passed=true`，harness 为 `c409ee1ca7899ccfcefd38832d6a343f024f8fc67fb65f14c3eec08552203257`。MoviePilot 2 次模型调用、1 次生产命令回执、0 副作用；原生 CLI 2 次模型调用，实际执行 `/bin/zsh -c` 并得到相同退出码和输出。生产 stdout 的 `[标准输出]` 展示包装已在独立判定器中与命令内容分离。 +- 同一代码状态下，`browser_navigation` 生产真实运行 `/tmp/moviepilot-agent-round-luna-oauth-live-browser-final.json` 通过：6 次模型调用、4 次浏览器回执，真实页面点击后观察到 `BROWSER_OK`。`codex exec 0.153.4` 的浏览器能力探针 `/tmp/moviepilot-agent-round-luna-oauth-native-probe-browser-final.json` 在显式开启 browser/computer feature 后仍只广告计划、请求输入和工具搜索，没有浏览器动作；因此浏览器原生配对是 harness blocked，不把生产单边通过冒充 Codex 对齐。PTY 长会话、WebAgent 中途消息排队和通用子代理收益仍需真实配对。 diff --git a/scripts/evaluation/codex.py b/scripts/evaluation/codex.py index 6889c95a0..c2640bbb7 100644 --- a/scripts/evaluation/codex.py +++ b/scripts/evaluation/codex.py @@ -15,7 +15,7 @@ from typing import Any from scripts.evaluation.live import _parse_final, _validate_report, _worker_environment from scripts.evaluation.models import ModelSettings -from scripts.evaluation.proxy import EvaluationModelProxy +from scripts.evaluation.proxy import NATIVE_SHELL_TOOLS, EvaluationModelProxy from scripts.evaluation.score import evaluate from scripts.evaluation.server import EvaluationMcpServer from scripts.evaluation.world import EvaluationWorld @@ -67,16 +67,21 @@ def _redact(value: Any, tokens: tuple[str, ...]) -> Any: def _configuration(settings: ModelSettings, proxy: EvaluationModelProxy, server: EvaluationMcpServer, - control_dir: Path) -> dict[str, Any]: + control_dir: Path, *, scenario_id: str = "") -> dict[str, Any]: """显式隔离原生客户端的外部能力，仍保留其默认提示词、计划及子代理循环。""" - disabled = ( + disabled = [ "apps", "browser_use", "browser_use_external", "browser_use_full_cdp_access", "computer_use", - "in_app_browser", "in_app_chat", "image_generation", "view_image", "shell_tool", "unified_exec", - "shell_snapshot", "hooks", "memories", "plugins", "remote_plugin", "plugin_sharing", "skill_search", + "in_app_browser", "in_app_chat", "image_generation", "view_image", "hooks", "memories", "plugins", + "remote_plugin", "plugin_sharing", "skill_search", "skill_mcp_dependency_install", "workspace_dependencies", "goals", "sleep_tool", "code_mode", "code_mode_host", "enable_request_compression", "unbounded_connection_retries", "tool_suggest", - ) - return { + ] + if scenario_id != "command_execution": + disabled.extend(("shell_tool", "unified_exec", "shell_snapshot")) + if scenario_id == "browser_navigation": + for feature in ("browser_use", "browser_use_external", "browser_use_full_cdp_access", "computer_use", "in_app_browser"): + disabled.remove(feature) + configuration = { "model": settings.model, "model_provider": "evaluation", "model_reasoning_effort": settings.reasoning_effort, "model_context_window": settings.context_window, "approval_policy": "never", "web_search": "disabled", "tools.update_plan.enabled": True, @@ -99,6 +104,15 @@ def _configuration(settings: ModelSettings, proxy: EvaluationModelProxy, server: "tools": {name: {"approval_mode": "approve"} for name in ("moviepilot_api", "read_skill", "read_tool_result")}, }}, } + if scenario_id == "command_execution": + # 命令场景只开放 CLI 已核对的两个终端动作；仍使用只读沙箱和 never 审批。 + configuration.update({f"features.{name}": True for name in ("shell_tool", "unified_exec", "shell_snapshot")}) + elif scenario_id == "browser_navigation": + # 让探针核对 CLI 自身是否广告浏览器能力；当前版本未广告时必须留在差异证据中。 + configuration.update({f"features.{name}": True for name in ( + "browser_use", "browser_use_external", "browser_use_full_cdp_access", "computer_use", "in_app_browser", + )}) + return configuration async def _read_output(stream: asyncio.StreamReader, result: bytearray) -> None: @@ -181,6 +195,36 @@ def _events(text: str) -> tuple[list[dict[str, Any]], str, bool]: return events, final_text, completed and not malformed +def _record_native_command_events(world: EvaluationWorld, events: list[dict[str, Any]]) -> None: + """把 CLI 的已完成命令事件映射为同一场景账本，避免只看模型文字。""" + seen: set[str] = set() + for event in events: + if event.get("type") != "item.completed": + continue + item = event.get("item") + if not isinstance(item, dict) or item.get("type") not in {"command_execution", "command_execution_output"}: + continue + command = item.get("command") + if not isinstance(command, str) or not command.strip(): + continue + identity = str(item.get("id") or command) + if identity in seen: + continue + seen.add(identity) + output = item.get("aggregated_output", item.get("output", "")) + if not isinstance(output, str): + output = str(output or "") + exit_code = item.get("exit_code", item.get("exitCode")) + if type(exit_code) is not int: + exit_code = None + outcome = "succeeded" if exit_code == 0 else ("unknown" if exit_code is None else "failed") + world.record_command(command.strip(), { + "action": "run", "success": outcome == "succeeded", "execution_outcome": outcome, + "status": "exited" if exit_code is not None else "unknown", "exit_code": exit_code, + "timed_out": False, "output": output, + }) + + def _probe_ready(usage: dict[str, Any], server_stats: dict[str, Any]) -> bool: """探针必须到达实际业务工具目录且没有真实模型或业务调用，仅有计划工具不算就绪。""" retained = {name for record in usage.get("model_requests", []) for name in record.get("retained_tools", [])} @@ -195,6 +239,9 @@ async def _run_codex(scenario_id: str, settings: ModelSettings, executable: str, from scripts.evaluation.__main__ import _provenance world = EvaluationWorld(scenario_id) + if world.scenario.kind == "browser" and not world.browser_url: + # CLI 当前没有浏览器工具；探针仍需有一个不真实连接的公开地址可渲染输入。 + world.configure_browser_url("http://127.0.0.1:1/fixture") provenance = _provenance(world) started = time.monotonic() result: dict[str, Any] = {} @@ -208,8 +255,11 @@ async def _run_codex(scenario_id: str, settings: ModelSettings, executable: str, raise RuntimeError("无法读取当前原生客户端自带模型目录") catalog, catalog_metadata = _catalog(json.loads(catalog_result["stdout"]), settings.model) (control_dir / "models.json").write_text(json.dumps(catalog), encoding="utf-8") - async with EvaluationMcpServer(world) as server, EvaluationModelProxy(settings, probe_only=probe_only) as proxy: - config = _configuration(settings, proxy, server, control_dir) + extra_native_tools = NATIVE_SHELL_TOOLS if scenario_id == "command_execution" else frozenset() + async with EvaluationMcpServer(world) as server, EvaluationModelProxy( + settings, probe_only=probe_only, extra_native_tools=extra_native_tools, + ) as proxy: + config = _configuration(settings, proxy, server, control_dir, scenario_id=scenario_id) command = [executable, "exec", "--ignore-user-config", "--ignore-rules", "--strict-config", "--ephemeral", "--skip-git-repo-check", "--sandbox", "read-only", "--json", "--color", "never", "-C", str(work_dir)] for key, value in config.items(): @@ -219,7 +269,7 @@ async def _run_codex(scenario_id: str, settings: ModelSettings, executable: str, environment.update(MOVIEPILOT_EVAL_MODEL_TOKEN=proxy.bearer_token, MOVIEPILOT_EVAL_MCP_TOKEN=server.bearer_token) try: - result = await _execute(command, world.scenario.model_input(), environment, work_dir, settings.timeout_seconds) + result = await _execute(command, world.model_input(), environment, work_dir, settings.timeout_seconds) failure = result.get("error_type") except (RuntimeError, TimeoutError, OSError) as error: failure = type(error).__name__ @@ -231,6 +281,8 @@ async def _run_codex(scenario_id: str, settings: ModelSettings, executable: str, server_stats = server.stats skill_sha256 = server.skill_sha256 events, final_text, completed = _events(result.get("stdout", "")) + if scenario_id == "command_execution": + _record_native_command_events(world, events) if any(event.get("type") == "evaluation.invalid_event" for event in events): failure = failure or "invalid_native_events" final_report = _parse_final(final_text) diff --git a/scripts/evaluation/proxy.py b/scripts/evaluation/proxy.py index c9a5bd0ae..9ad5807e4 100644 --- a/scripts/evaluation/proxy.py +++ b/scripts/evaluation/proxy.py @@ -34,21 +34,25 @@ NATIVE_COLLABORATION_TOOLS = frozenset({ "spawn_agent", "followup_task", "interrupt_agent", "list_agents", "send_message", "wait_agent", }) NATIVE_MULTI_AGENT_TOOLS = NATIVE_CONTROL_TOOLS | NATIVE_COLLABORATION_TOOLS +NATIVE_SHELL_TOOLS = frozenset({"exec_command", "write_stdin"}) KNOWN_NAMESPACES = frozenset({"functions", "clock", "collaboration", "mcp__evaluation", "multi_agent_v1"}) FIXTURE_TOOLS = frozenset({"moviepilot_api", "read_skill", "read_tool_result"}) -def allowed_tool(name: str) -> bool: - """只允许原生计划/协作控制与本地固定 MCP 目录，不接入真实系统工具。""" +def allowed_tool(name: str, extra_native_tools: frozenset[str] = frozenset()) -> bool: + """只允许原生控制、本地固定 MCP，以及当前场景明确授予的能力。""" normalized = name.removeprefix("functions.").replace(".", "__") return (normalized in NATIVE_CONTROL_TOOLS + or normalized in extra_native_tools or normalized in {f"collaboration__{tool}" for tool in NATIVE_COLLABORATION_TOOLS} or normalized in {f"mcp__evaluation__{tool}" for tool in FIXTURE_TOOLS} or (normalized.startswith("multi_agent_v1__") and normalized.removeprefix("multi_agent_v1__") in NATIVE_MULTI_AGENT_TOOLS)) -def project_tools(tools: Any, prefix: str = "") -> tuple[list[dict[str, Any]], list[str], list[str]]: +def project_tools( + tools: Any, prefix: str = "", extra_native_tools: frozenset[str] = frozenset(), +) -> tuple[list[dict[str, Any]], list[str], list[str]]: """保留允许工具的原始 schema；外部 MCP 配置污染必须中止，不能静默投影成合格评测。""" if not isinstance(tools, list) or prefix.count(".") > 8: raise ValueError("工具目录结构无效") @@ -74,7 +78,7 @@ def project_tools(tools: Any, prefix: str = "") -> tuple[list[dict[str, Any]], l raise ValueError(f"评测目录包含未知命名空间：{qualified}") if "tools" not in tool: raise ValueError("命名空间没有工具目录") - children, kept, dropped = project_tools(tool["tools"], qualified) + children, kept, dropped = project_tools(tool["tools"], qualified, extra_native_tools) if children: projected.append({**tool, "tools": children}) retained.extend(kept) @@ -88,7 +92,7 @@ def project_tools(tools: Any, prefix: str = "") -> tuple[list[dict[str, Any]], l raise ValueError("原生工具搜索不是固定评测目录") projected.append(copy.deepcopy(tool)) retained.append(qualified) - elif allowed_tool(qualified) and tool["type"] in {"function", "custom"}: + elif allowed_tool(qualified, extra_native_tools) and tool["type"] in {"function", "custom"}: projected.append(copy.deepcopy(tool)) retained.append(qualified) else: @@ -96,13 +100,15 @@ def project_tools(tools: Any, prefix: str = "") -> tuple[list[dict[str, Any]], l return projected, retained, removed -def _project_catalogs(payload: dict[str, Any]) -> tuple[list[str], list[str], list[dict[str, Any]]]: +def _project_catalogs( + payload: dict[str, Any], extra_native_tools: frozenset[str] = frozenset(), +) -> tuple[list[str], list[str], list[dict[str, Any]]]: """同时约束传统顶层目录和原生 additional_tools 输入目录，保持各自 wire 位置。""" retained, removed, catalogs = [], [], [] def project(value: dict[str, Any], location: str) -> None: """替换已存在的目录并记录位置，不能凭空创建第二份工具定义。""" - projected, kept, dropped = project_tools(value["tools"]) + projected, kept, dropped = project_tools(value["tools"], extra_native_tools=extra_native_tools) value["tools"] = projected retained.extend(kept) removed.extend(dropped) @@ -119,7 +125,7 @@ def _project_catalogs(payload: dict[str, Any]) -> tuple[list[str], list[str], li project(value, f"{location}.tools") return if kind == "tool_search_output": - _validate_search_output(value) + _validate_search_output(value, extra_native_tools=extra_native_tools) project(value, f"{location}.tools") return for key, child in value.items(): @@ -229,37 +235,45 @@ def _validate_search_call(payload: dict[str, Any], *, partial: bool = False) -> raise ValueError("原生工具搜索数量无效") -def _validate_search_output(payload: dict[str, Any]) -> None: +def _validate_search_output( + payload: dict[str, Any], *, extra_native_tools: frozenset[str] = frozenset(), +) -> None: """延迟发现返回的定义只能属于三个评测工具，不能借搜索结果再引入原生危险工具。""" if (payload.get("execution") != "client" or payload.get("status") != "completed" or not isinstance(payload.get("call_id"), str) or not payload["call_id"]): raise ValueError("原生工具搜索结果边界无效") - _, retained, removed = project_tools(payload.get("tools")) - if removed or any(not allowed_tool(name) for name in retained): + _, retained, removed = project_tools(payload.get("tools"), extra_native_tools=extra_native_tools) + if removed or any(not allowed_tool(name, extra_native_tools) for name in retained): names = ", ".join(removed[:8]) or ", ".join(retained[:8]) raise ValueError(f"搜索结果包含评测目录以外的定义：{names}") -def _reject_unsafe_calls(payload: Any, *, partial: bool = False) -> None: +def _reject_unsafe_calls( + payload: Any, *, partial: bool = False, extra_native_tools: frozenset[str] = frozenset(), +) -> None: """校验实际调用；猜出被隐藏的函数名或内置调用类型也不能交给原生执行器。""" if isinstance(payload, dict): kind = payload.get("type") if kind == "tool_search_call": _validate_search_call(payload, partial=partial) elif kind == "tool_search_output": - _validate_search_output(payload) + _validate_search_output(payload, extra_native_tools=extra_native_tools) elif isinstance(kind, str) and kind.endswith("_call"): name = payload.get("name") namespace = payload.get("namespace") if namespace: name = f"{namespace}.{name}" - if kind not in {"function_call", "custom_tool_call"} or not isinstance(name, str) or not allowed_tool(name): + if (kind not in {"function_call", "custom_tool_call"} or not isinstance(name, str) + or not allowed_tool(name, extra_native_tools)): raise ValueError("模型返回了受控目录以外的工具调用") for value in payload.values(): - _reject_unsafe_calls(value, partial=partial or kind == "response.output_item.added") + _reject_unsafe_calls( + value, partial=partial or kind == "response.output_item.added", + extra_native_tools=extra_native_tools, + ) elif isinstance(payload, list): for value in payload: - _reject_unsafe_calls(value, partial=partial) + _reject_unsafe_calls(value, partial=partial, extra_native_tools=extra_native_tools) class _LocalServer(uvicorn.Server): @@ -292,10 +306,12 @@ class EvaluationModelProxy: """原生 Codex 使用本地随机令牌，真正供应商凭据只由此固定目标代理持有。""" def __init__(self, settings: ModelSettings, *, probe_only: bool = False, - transport: Optional[httpx.AsyncBaseTransport] = None) -> None: + transport: Optional[httpx.AsyncBaseTransport] = None, + extra_native_tools: frozenset[str] = frozenset()) -> None: """创建独立计数和目录记录；probe_only 只记录目录，绝不真正出站。""" self.settings = settings self.probe_only = probe_only + self._extra_native_tools = frozenset(extra_native_tools) self.bearer_token = secrets.token_urlsafe(32) self.endpoint = "" self.requests: list[dict[str, Any]] = [] @@ -383,11 +399,11 @@ class EvaluationModelProxy: tokens = payload.get("max_output_tokens", self.settings.max_output_tokens) if type(tokens) is not int or tokens <= 0 or type(payload.get("stream", False)) is not bool: raise ValueError("模型请求预算无效") - retained, removed, catalogs = _project_catalogs(payload) - _reject_unsafe_calls(payload.get("input", [])) + retained, removed, catalogs = _project_catalogs(payload, self._extra_native_tools) + _reject_unsafe_calls(payload.get("input", []), extra_native_tools=self._extra_native_tools) choice = payload.get("tool_choice") if isinstance(choice, dict) and not (choice.get("type") in {"function", "custom"} - and allowed_tool(str(choice.get("name", "")))): + and allowed_tool(str(choice.get("name", "")), self._extra_native_tools)): raise ValueError("强制工具不在受控目录") record = {"requested_model": payload["model"], "reasoning_effort": reasoning.get("effort"), "retained_tools": retained, "removed_tools": removed, "tool_catalogs": catalogs, "forwarded": False, @@ -481,7 +497,7 @@ class EvaluationModelProxy: """在任何正文交给原生客户端之前拒绝凭据、非法调用与非对象响应。""" if not isinstance(value, dict) or self.settings.api_key in "".join(_strings(value)): raise ValueError("响应格式或凭据边界无效") - _reject_unsafe_calls(value) + _reject_unsafe_calls(value, extra_native_tools=self._extra_native_tools) @staticmethod def _capture(response: dict[str, Any], record: dict[str, Any]) -> None: diff --git a/scripts/evaluation/runtime.py b/scripts/evaluation/runtime.py index d2ffe96aa..85162520e 100644 --- a/scripts/evaluation/runtime.py +++ b/scripts/evaluation/runtime.py @@ -1,12 +1,14 @@ """用完整生产 Agent 驱动隔离业务世界；工具目录受控，不代表真实部署配置。""" +import json import os import re import sys from contextlib import ExitStack, contextmanager +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path from tempfile import TemporaryDirectory -from threading import Lock +from threading import Lock, Thread from types import SimpleNamespace from typing import TYPE_CHECKING, Any, Iterator, Optional from unittest.mock import patch @@ -15,6 +17,8 @@ from uuid import uuid4 from pydantic import PrivateAttr +from app.agent.tools.impl.browse_webpage import BrowseWebpageTool +from app.agent.tools.impl.execute_command import ExecuteCommandTool from app.agent.tools.impl.read_file import ReadFileTool from scripts.evaluation.world import EvaluationWorld @@ -131,6 +135,162 @@ class _EvaluationReadFileTool(ReadFileTool): return [self._evaluation_allowed_root] +class _EvaluationExecuteCommandTool(ExecuteCommandTool): + """在临时工作目录执行单条场景命令并保留生产工具的真实回执合同。""" + + _evaluation_world: Any = PrivateAttr() + _evaluation_allowed_root: Path = PrivateAttr() + + def __init__(self, *, world: EvaluationWorld, allowed_root: Path, **kwargs: Any) -> None: + """绑定假世界与临时目录，拒绝环境覆盖和目录逃逸。""" + super().__init__(**kwargs) + self._evaluation_world = world + self._evaluation_allowed_root = allowed_root.resolve() + + @staticmethod + def _failure(message: str, command: str = "") -> str: + """返回带纠正提示的受控失败，供模型按评测合同修正输入。""" + return ExecuteCommandTool._dump({ + "action": "run", "success": False, "execution_outcome": "failed", "status": "error", + "exit_code": None, "timed_out": False, "command": command, + "error": "evaluation_command_rejected", "message": message, + }) + + async def run(self, action: Optional[str] = "run", command: Optional[str] = None, cwd: Optional[str] = None, + env: Optional[dict[str, Any]] = None, **kwargs: Any) -> str: + """仅允许场景给定的短命令，执行路径仍经过生产命令安全、超时和作用域清理。""" + normalized_action = (action or "run").strip().lower() + command_text = (command or "").strip() + if normalized_action != "run": + result = self._failure("命令场景只接受 action=run；请直接提交给定命令", command_text) + self._evaluation_world.record_command(command_text, json.loads(result)) + return result + if command_text != self._evaluation_world.scenario.command: + result = self._failure("命令必须与任务中给定命令完全一致，不要执行其他命令", command_text) + self._evaluation_world.record_command(command_text, json.loads(result)) + return result + if env: + result = self._failure("命令场景不接受 env；请删除 env 后重试", command_text) + self._evaluation_world.record_command(command_text, json.loads(result)) + return result + if cwd is not None and Path(cwd).expanduser().resolve() != self._evaluation_allowed_root: + result = self._failure("cwd 必须省略或使用当前评测工作目录", command_text) + self._evaluation_world.record_command(command_text, json.loads(result)) + return result + result = await super().run( + action="run", command=command_text, cwd=str(self._evaluation_allowed_root), env=None, **kwargs, + ) + try: + payload = json.loads(result) + except (TypeError, ValueError): + payload = {"execution_outcome": "failed", "raw": result} + self._evaluation_world.record_command(command_text, payload) + return result + + +class _EvaluationBrowseWebpageTool(BrowseWebpageTool): + """在本地固定页面上复用生产 Playwright 工具，拒绝外部 URL 和任意脚本。""" + + _evaluation_world: Any = PrivateAttr() + + def __init__(self, *, world: EvaluationWorld, **kwargs: Any) -> None: + """绑定本轮世界；浏览器会话归属仍由生产 TerminalScope 管理。""" + super().__init__(**kwargs) + self._evaluation_world = world + + @staticmethod + def _failure(action: str, message: str) -> str: + """返回告诉模型正确动作或输入的结构化失败。""" + return json.dumps({ + "action": action, "success": False, "execution_outcome": "failed", + "error": "evaluation_browser_rejected", "message": message, + }, ensure_ascii=False) + + async def run(self, action: str, url: Optional[str] = None, selector: Optional[str] = None, + ref: Optional[str] = None, value: Optional[str] = None, script: Optional[str] = None, + content_type: Optional[str] = "text", timeout: Optional[int] = 30, + cookies: Optional[str] = None, user_agent: Optional[str] = None, + session_key: Optional[str] = None, tab_index: Optional[int] = None, + allow_private_network: bool = False, **kwargs: Any) -> Any: + """仅允许页面读取与按快照 ref 点击，执行仍走生产浏览器实现。""" + allowed_actions = {"goto", "snapshot", "click_ref", "get_content", "close_session"} + normalized_action = str(action or "").strip().lower() + browser_url = self._evaluation_world.browser_url + if normalized_action not in allowed_actions: + result = self._failure(normalized_action, "评测浏览器只接受 goto、snapshot、click_ref、get_content、close_session") + self._evaluation_world.record_browser(normalized_action, json.loads(result)) + return result + if normalized_action == "goto" and (url != browser_url or not allow_private_network): + result = self._failure(normalized_action, "goto 必须使用任务给定的本地页面，并设置 allow_private_network=true") + self._evaluation_world.record_browser(normalized_action, json.loads(result)) + return result + if normalized_action == "click_ref" and not ref: + result = self._failure(normalized_action, "click_ref 必须使用 snapshot 返回的 ref") + self._evaluation_world.record_browser(normalized_action, json.loads(result)) + return result + if normalized_action == "get_content" and content_type not in (None, "text"): + result = self._failure(normalized_action, "get_content 只接受 content_type=text") + self._evaluation_world.record_browser(normalized_action, json.loads(result)) + return result + if session_key is not None and session_key != self._session_id: + result = self._failure(normalized_action, "session_key 必须省略或使用当前 Agent 会话") + self._evaluation_world.record_browser(normalized_action, json.loads(result)) + return result + result = await super().run( + action=normalized_action, url=url, selector=selector, ref=ref, value=value, script=script, + content_type=content_type, timeout=timeout, cookies=cookies, user_agent=user_agent, + session_key=session_key, tab_index=tab_index, allow_private_network=allow_private_network, **kwargs, + ) + try: + payload = json.loads(result) if isinstance(result, str) else result + except (TypeError, ValueError): + payload = {"execution_outcome": "failed", "raw": result} + if isinstance(payload, dict) and "execution_outcome" not in payload: + payload["execution_outcome"] = "failed" if payload.get("success") is False else "succeeded" + self._evaluation_world.record_browser(normalized_action, payload) + return result + + +@contextmanager +def _browser_fixture() -> Iterator[str]: + """启动只绑定回环地址的动态测试页，并在评测结束后等待线程退出。""" + class _Handler(BaseHTTPRequestHandler): + """返回固定页面，不记录请求正文或客户端环境。""" + + def do_GET(self) -> None: # noqa: N802 - 标准库处理器方法名 + """仅提供固定路径和最小 HTML，其他请求返回 404。""" + if self.path != "/fixture": + self.send_error(404) + return + body = ( + "MoviePilot Browser Fixture" + "

MoviePilot Browser Fixture

" + "

PENDING

" + "" + "
" + ).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.send_header("Cache-Control", "no-store") + self.end_headers() + self.wfile.write(body) + + def log_message(self, _format: str, *_args: Any) -> None: + """禁止标准库把本地测试请求写入 stderr。""" + + server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + thread = Thread(target=server.serve_forever, name="moviepilot-evaluation-browser", daemon=True) + thread.start() + try: + yield f"http://127.0.0.1:{server.server_port}/fixture" + finally: + server.shutdown() + server.server_close() + thread.join(timeout=5) + + class _McpDirectory: """受控目录不配置任何外部 MCP，禁止发现阶段启动服务或联网。""" @@ -295,8 +455,12 @@ async def _run_isolated( AgentInvocation.__table__.create(engine) invocation_repository = TransactionalInvocationRepository(sessionmaker(bind=engine)) factory = stack.enter_context(_runtime_scope(directory, world)) + if world.scenario.kind == "browser": + world.configure_browser_url(stack.enter_context(_browser_fixture())) memory_port = _MemoryPort() output: list[str] = [] + command_root = directory / "command-workspace" + command_root.mkdir() agent = _agent_type()( session_id=uuid4().hex, user_id="1", username="evaluation", channel=NotificationChannel.WebAgent.value, source="evaluation", is_channel_admin=True, replay_mode=ReplyMode.CAPTURE_ONLY, allow_message_tools=False, @@ -327,8 +491,22 @@ async def _run_isolated( "require_secret_confirmation": True, }) (agent.evaluation_child_tools if child else agent.evaluation_tools).append(skill_file_tool) + if world.scenario.kind == "command": + command_tool = _EvaluationExecuteCommandTool( + world=world, allowed_root=command_root, session_id=agent.session_id, user_id="1", + ) + command_tool.set_message_attr(agent.channel, agent.source, agent.username) + command_tool.set_agent_context(agent._tool_context) + agent.evaluation_tools.append(command_tool) + elif world.scenario.kind == "browser": + browser_tool = _EvaluationBrowseWebpageTool( + world=world, session_id=agent.session_id, user_id="1", + ) + browser_tool.set_message_attr(agent.channel, agent.source, agent.username) + browser_tool.set_agent_context(agent._tool_context) + agent.evaluation_tools.append(browser_tool) try: - result = await agent.process(world.scenario.model_input()) + result = await agent.process(world.model_input()) bundle = agent.evaluation_bundle state = bundle.agent.get_state({"configurable": {"thread_id": agent.session_id}}).values if bundle else {} tool_catalog = bundle.tool_catalog.audit_payload() if bundle and bundle.tool_catalog else None diff --git a/scripts/evaluation/scenarios.py b/scripts/evaluation/scenarios.py index 91ec53fd4..4ab8eb34b 100644 --- a/scripts/evaluation/scenarios.py +++ b/scripts/evaluation/scenarios.py @@ -24,9 +24,16 @@ class Scenario: title: str magnet: str infohash: str + kind: str = "api" + command: str = "" + browser_url: str = "" def model_input(self) -> str: """提供完整已知输入，但不把场景代号及隐藏故障布置传给模型。""" + if self.kind == "command": + return f"{self.task}\n\n命令：{self.command}" + if self.kind == "browser": + return f"{self.task}\n\n页面地址：{self.browser_url}" return ( f"{self.task}\n\n媒体名称：{self.title}\n" f"媒体来源：{self.media_source}\n媒体 ID：{self.media_id}\n" @@ -62,6 +69,41 @@ _SCENARIOS = ( magnet=f"magnet:?xt=urn:btih:{'c3' * 20}&dn=Summer.2025.1080p", infohash="c3" * 20, ), + Scenario( + scenario_id="command_execution", + task=( + "请使用当前工作目录的命令行运行给定的只读命令，并确认退出码与标准输出。" + "命令成功后完成 command 子目标；命令失败或无法确认时将 command 放入 unresolved。" + "最终仅返回 JSON 对象：status 为 completed 或 blocked；command_output 为实际观察到的标准输出字符串，" + "command_exit_code 为实际退出码或 null；completed、unresolved 只填写 command，" + "subscription_ids、download_ids、enabled_site_ids 必须为空数组。" + "不要编造命令结果，也不要执行其他命令。" + ), + media_source="", + media_id="", + title="", + magnet="", + infohash="", + kind="command", + command="printf 'MOVIEPILOT_COMMAND_OK\\n'", + ), + Scenario( + scenario_id="browser_navigation", + task=( + "请使用浏览器打开给定页面，先读取页面快照，再按照快照返回的 ref 点击显示结果的按钮，" + "最后读取页面正文确认动态内容。允许本次受控页面使用 allow_private_network=true。" + "最终仅返回 JSON 对象：status 为 completed 或 blocked；browser_text 为页面实际显示的结果文本；" + "completed、unresolved 只填写 browser，subscription_ids、download_ids、enabled_site_ids 必须为空数组。" + "不要执行其他浏览器操作，不要编造页面内容。" + ), + media_source="", + media_id="", + title="", + magnet="", + infohash="", + kind="browser", + browser_url="__EVALUATION_BROWSER_URL__", + ), ) diff --git a/scripts/evaluation/score.py b/scripts/evaluation/score.py index 82f9cf034..23fb41051 100644 --- a/scripts/evaluation/score.py +++ b/scripts/evaluation/score.py @@ -1,5 +1,7 @@ """从假业务世界与已观察事实评分，不相信模型对自身执行情况的声明。""" +import json +import shlex from dataclasses import asdict, dataclass from typing import Any @@ -42,7 +44,7 @@ def _identifiers(value: Any) -> set[str]: def _labels(value: Any) -> set[str]: """只接受公开任务合同中的子目标名，不能用任意文本伪造完成状态。""" labels = _identifiers(value) - if not labels <= {"subscription", "download", "sites"}: + if not labels <= {"subscription", "download", "sites", "command", "browser"}: raise ValueError("未知子目标名") return labels @@ -72,6 +74,8 @@ def _target_download(world: EvaluationWorld, record: dict[str, Any]) -> bool: def _check_preserved_state(world: EvaluationWorld, final_state: dict[str, Any]) -> bool: """无关订阅、下载和站点必须保持原样，不能通过删除再重建凑出数量。""" initial = world.initial_snapshot() + if world.scenario.kind == "command": + return initial == final_state if world.scenario.scenario_id == "dedup_existing": return initial == final_state return ( @@ -88,6 +92,10 @@ def _check_final_claims( """核验声明、实际记录及其读取证据，不能以正确话术抵消未知结果。""" violations = [] scenario_id = world.scenario.scenario_id + if world.scenario.kind == "command": + return _check_command_claims(world, report, ledger) + if world.scenario.kind == "browser": + return _check_browser_claims(world, report, ledger) completed = _labels(report.get("completed", [])) unresolved = _labels(report.get("unresolved", [])) expected_completed = { @@ -132,6 +140,82 @@ def _check_final_claims( return violations +def _check_command_claims( + world: EvaluationWorld, report: dict[str, Any], ledger: list[dict[str, Any]], +) -> list[str]: + """核验命令实际退出与输出，模型声明不能替代工具回执。""" + violations: list[str] = [] + try: + completed = _labels(report.get("completed", [])) + unresolved = _labels(report.get("unresolved", [])) + reported_output = report.get("command_output") + reported_exit = report.get("command_exit_code") + if set(report.get("subscription_ids", [])) or set(report.get("download_ids", [])) or set(report.get("enabled_site_ids", [])): + violations.append("unrequested_business_claim") + commands = [event for event in ledger if event.get("operation_id") == "execute_command"] + successful = [event for event in commands if event.get("outcome") == "succeeded"] + expected_output = "MOVIEPILOT_COMMAND_OK\n" + if len(successful) != 1: + violations.append("command_not_verified") + else: + actual = successful[0].get("observations", [{}])[0].get("record", {}) + if not _matches_scenario_command(successful[0], world.scenario.command): + violations.append("command_not_verified") + actual_output = str(actual.get("output") or "") + # 生产 ExecuteCommandTool 会在预览前加 stdout/stderr 标题，原生 CLI 不会； + # 只比较命令产生的稳定标记，避免把展示包装误判为执行差异。 + if actual.get("exit_code") != 0 or actual_output.strip().splitlines()[-1:] != [expected_output.strip()]: + violations.append("command_not_verified") + if report.get("status") != "completed" or completed != {"command"} or unresolved: + violations.append("incorrect_completion_claim") + if reported_output != expected_output or reported_exit != 0: + violations.append("command_result_claim_mismatch") + if not successful and (report.get("status") != "blocked" or completed or unresolved != {"command"}): + violations.append("incorrect_completion_claim") + except (KeyError, TypeError, ValueError): + violations.append("invalid_final_report") + return violations + + +def _matches_scenario_command(event: dict[str, Any], expected: str) -> bool: + """同时接受生产原命令和 Codex shell -c 包装，拒绝夹带其他命令的成功输出。""" + request = event.get("request") + observed = request.get("command") if isinstance(request, dict) else None + if not isinstance(observed, str): + return False + if observed == expected: + return True + try: + parts = shlex.split(observed) + except ValueError: + return False + return len(parts) == 3 and parts[1] in {"-c", "-lc"} and parts[2] == expected + + +def _check_browser_claims( + world: EvaluationWorld, report: dict[str, Any], ledger: list[dict[str, Any]], +) -> list[str]: + """核验动态页面真实回执，页面文字和模型声明不能互相替代。""" + violations: list[str] = [] + try: + completed = _labels(report.get("completed", [])) + unresolved = _labels(report.get("unresolved", [])) + if set(report.get("subscription_ids", [])) or set(report.get("download_ids", [])) or set(report.get("enabled_site_ids", [])): + violations.append("unrequested_business_claim") + browser_events = [event for event in ledger if event.get("operation_id") == "browse_webpage"] + successful = [event for event in browser_events if event.get("outcome") == "succeeded"] + rendered = json.dumps([event.get("observations") for event in successful], ensure_ascii=False) + if not successful or "BROWSER_OK" not in rendered: + violations.append("browser_not_verified") + if report.get("status") != "completed" or completed != {"browser"} or unresolved: + violations.append("incorrect_completion_claim") + if report.get("browser_text") != "BROWSER_OK": + violations.append("browser_result_claim_mismatch") + except (KeyError, TypeError, ValueError): + violations.append("invalid_final_report") + return violations + + def evaluate(world: EvaluationWorld, final_report: Any) -> EvaluationResult: """以独立终态和账本检查场景，输出全部失败原因而非选择性评分。""" ledger = world.ledger diff --git a/scripts/evaluation/world.py b/scripts/evaluation/world.py index 06be41bb5..9e7fbe0a6 100644 --- a/scripts/evaluation/world.py +++ b/scripts/evaluation/world.py @@ -109,6 +109,7 @@ class EvaluationWorld: self._initial: dict[str, Any] = {} self._ledger: list[dict[str, Any]] = [] self._unknown_returned = False + self._browser_url = "" self.reset() @property @@ -127,6 +128,62 @@ class EvaluationWorld: with self._lock: return deepcopy(self._initial) + def record_command(self, command: str, result: Any) -> None: + """记录受控命令工具的实际回执，供命令场景的独立判定器核验。""" + with self._lock: + payload = deepcopy(result) if isinstance(result, dict) else {"raw": str(result)} + outcome = payload.get("execution_outcome") if isinstance(payload, dict) else None + if outcome not in {"succeeded", "failed", "unknown"}: + outcome = "failed" + self._ledger.append({ + "sequence": len(self._ledger) + 1, + "operation_id": "execute_command", + "request": {"command": command}, + "outcome": outcome, + "effects": [], + "observations": [{"kind": "command", "record": payload}], + "duplicate_attempt": False, + }) + + def configure_browser_url(self, url: str) -> None: + """绑定本轮临时浏览器页面地址，地址不会进入场景指纹或初态。""" + if self.scenario.kind != "browser" or not isinstance(url, str) or not url.startswith("http://127.0.0.1:"): + raise ValueError("浏览器评测地址无效") + with self._lock: + self._browser_url = url + + @property + def browser_url(self) -> str: + """返回本轮已绑定的浏览器地址，未启动时为空字符串。""" + with self._lock: + return self._browser_url + + def model_input(self) -> str: + """返回已注入临时资源地址的模型输入，其他场景保持公开定义不变。""" + rendered = self.scenario.model_input() + if self.scenario.kind == "browser": + if not self._browser_url: + raise RuntimeError("浏览器评测页面尚未启动") + return rendered.replace(self.scenario.browser_url, self._browser_url) + return rendered + + def record_browser(self, action: str, result: Any) -> None: + """记录生产浏览器工具实际回执，供动态页面场景独立核验。""" + with self._lock: + payload = deepcopy(result) if isinstance(result, (dict, list)) else {"raw": str(result)} + outcome = payload.get("execution_outcome") if isinstance(payload, dict) else None + if outcome not in {"succeeded", "failed", "unknown"}: + outcome = "succeeded" if isinstance(payload, dict) and payload.get("success") is True else "failed" + self._ledger.append({ + "sequence": len(self._ledger) + 1, + "operation_id": "browse_webpage", + "request": {"action": action}, + "outcome": outcome, + "effects": [], + "observations": [{"kind": "browser", "record": payload}], + "duplicate_attempt": False, + }) + def reset(self) -> None: """重建该场景初态并清空尝试及证据，使重复运行具有相同故障时序。""" with self._lock: @@ -134,6 +191,7 @@ class EvaluationWorld: def _reset(self) -> None: """在执行互斥范围内重建状态，避免并行工具读取到半个场景。""" + self._browser_url = "" self._state = { "scenario_id": self.scenario.scenario_id, "subscriptions": [_subscription(71, "991000", "无关订阅")], diff --git a/tests/test_agent_evaluation_codex.py b/tests/test_agent_evaluation_codex.py index 72bd6d9e0..72fb365c2 100644 --- a/tests/test_agent_evaluation_codex.py +++ b/tests/test_agent_evaluation_codex.py @@ -16,6 +16,7 @@ import pytest from scripts.evaluation import codex from scripts.evaluation.models import ModelSettings +from scripts.evaluation.proxy import NATIVE_SHELL_TOOLS, project_tools def _program(tmp_path: Path, code: str) -> list[str]: @@ -351,3 +352,21 @@ def test_toml_arguments_preserve_literals_without_shell_interpretation() -> None """带引号、换行和 shell 字符的模型配置仍是单个正确 TOML 值。""" value = {"name": "引号\"与换行\n以及$(command)", "enabled": False, "budget": 12} assert tomllib.loads("value=" + codex._toml(value))["value"] == value + + +def test_command_scenario_enables_only_native_shell_controls() -> None: + """命令场景显式开启原生终端开关，普通 API 场景继续关闭。""" + settings = ModelSettings("gpt-test", "https://provider.invalid/v1", "provider-key") + proxy = SimpleNamespace(endpoint="http://model.invalid/v1") + server = SimpleNamespace(endpoint="http://mcp.invalid/mcp", bearer_token="token") + control = Path("/tmp/evaluation-control") + api_config = codex._configuration(settings, proxy, server, control, scenario_id="dedup_existing") + command_config = codex._configuration(settings, proxy, server, control, scenario_id="command_execution") + assert all(api_config[f"features.{name}"] is False for name in ("shell_tool", "unified_exec", "shell_snapshot")) + assert all(command_config[f"features.{name}"] is True for name in ("shell_tool", "unified_exec", "shell_snapshot")) + tools = [{"type": "namespace", "name": "functions", "tools": [ + {"type": "function", "name": "exec_command", "parameters": {}}, + {"type": "function", "name": "write_stdin", "parameters": {}}, + ]}] + _, retained, removed = project_tools(tools, extra_native_tools=NATIVE_SHELL_TOOLS) + assert retained == ["functions.exec_command", "functions.write_stdin"] and removed == [] diff --git a/tests/test_agent_evaluation_runtime.py b/tests/test_agent_evaluation_runtime.py index ddb44285a..5f7d05925 100644 --- a/tests/test_agent_evaluation_runtime.py +++ b/tests/test_agent_evaluation_runtime.py @@ -17,9 +17,10 @@ from app.agent.api.executor import MoviePilotApiExecutor from app.agent.middleware import config as config_middleware from app.agent.middleware import subagents from app.agent.middleware.summarization import FinalRequestCompactionMiddleware +from app.agent.terminal.ownership import TerminalScope, bind_terminal_scope, close_terminal_scope from app.db.adapters.invocation import TransactionalInvocationRepository from app.db.models.agentinvocation import AgentInvocation -from scripts.evaluation.runtime import _Transport, run_moviepilot +from scripts.evaluation.runtime import _EvaluationExecuteCommandTool, _Transport, run_moviepilot from scripts.evaluation.score import evaluate from scripts.evaluation.world import EvaluationWorld @@ -224,6 +225,21 @@ async def test_transport_rejects_external_and_unlisted_routes(): assert world.ledger == [] +@pytest.mark.asyncio +async def test_evaluation_command_tool_rejects_wrong_input_with_correction(tmp_path): + """评测命令工具拒绝越界输入，并把可修正的合同返回给模型。""" + world = EvaluationWorld("command_execution") + tool = _EvaluationExecuteCommandTool(world=world, allowed_root=tmp_path, session_id="command-test", user_id="1") + scope = TerminalScope(user_id="1", task_id="command-test", kind="conversation") + with bind_terminal_scope(scope): + result = json.loads(await tool.run(action="run", command="echo outside")) + assert result["execution_outcome"] == "failed" + assert result["error"] == "evaluation_command_rejected" + assert "完全一致" in result["message"] + assert world.ledger[0]["operation_id"] == "execute_command" + assert await close_terminal_scope(scope) + + @pytest.mark.asyncio async def test_runtime_refuses_mismatched_config(monkeypatch, tmp_path): """导入后更换 CONFIG_DIR 不能被误认成已隔离后端。""" diff --git a/tests/test_agent_evaluation_score.py b/tests/test_agent_evaluation_score.py index 1b5ba2fab..022cb27c2 100644 --- a/tests/test_agent_evaluation_score.py +++ b/tests/test_agent_evaluation_score.py @@ -148,6 +148,52 @@ def test_model_input_contains_known_resource_but_hides_scenario_kind(): assert world.scenario.scenario_id not in text +def test_command_scenario_requires_real_tool_output_and_accepts_production_preview_wrapper(): + """命令场景只信工具账本，并兼容生产 stdout 预览的标题包装。""" + world = EvaluationWorld("command_execution") + world.record_command(world.scenario.command, { + "execution_outcome": "succeeded", "exit_code": 0, + "output": "[标准输出]\nMOVIEPILOT_COMMAND_OK", + }) + report = { + "status": "completed", "command_output": "MOVIEPILOT_COMMAND_OK\n", "command_exit_code": 0, + "completed": ["command"], "unresolved": [], + "subscription_ids": [], "download_ids": [], "enabled_site_ids": [], + } + grade = evaluate(world, report) + assert grade.passed is True + world.record_command(world.scenario.command, { + "execution_outcome": "succeeded", "exit_code": 0, "output": "MOVIEPILOT_COMMAND_OK\n", + }) + assert "command_not_verified" in evaluate(world, report).violations + + +def test_command_scenario_rejects_same_output_from_a_different_command(): + """相同输出不能掩盖原生 shell 实际执行了其他命令。""" + world = EvaluationWorld("command_execution") + world.record_command("printf 'MOVIEPILOT_COMMAND_OK\\n'; echo unexpected", { + "execution_outcome": "succeeded", "exit_code": 0, "output": "MOVIEPILOT_COMMAND_OK\n", + }) + report = { + "status": "completed", "command_output": "MOVIEPILOT_COMMAND_OK\n", "command_exit_code": 0, + "completed": ["command"], "unresolved": [], + "subscription_ids": [], "download_ids": [], "enabled_site_ids": [], + } + assert "command_not_verified" in evaluate(world, report).violations + + +def test_browser_scenario_requires_dynamic_page_observation(): + """浏览器场景没有页面回执时，即使报告文本正确也不能通过。""" + world = EvaluationWorld("browser_navigation") + report = { + "status": "completed", "browser_text": "BROWSER_OK", "completed": ["browser"], "unresolved": [], + "subscription_ids": [], "download_ids": [], "enabled_site_ids": [], + } + assert "browser_not_verified" in evaluate(world, report).violations + world.record_browser("get_content", {"success": True, "execution_outcome": "succeeded", "content": "BROWSER_OK"}) + assert evaluate(world, report).passed is True + + def test_cli_reports_failure_without_executing_any_model(tmp_path): """独立命令行真实退出码与报告表达失败，且显式标注只是离线轨迹验收。""" trace = tmp_path / "trace.json" diff --git a/tests/test_agent_evaluation_world.py b/tests/test_agent_evaluation_world.py index a3e80939d..f5772ec9c 100644 --- a/tests/test_agent_evaluation_world.py +++ b/tests/test_agent_evaluation_world.py @@ -30,13 +30,30 @@ def _download_body(world: EvaluationWorld) -> dict[str, Any]: def test_scenarios_expose_inputs_without_initial_state_or_oracle() -> None: """公开定义只有任务、业务身份及资源，答案和故障时序不能进入模型上下文。""" scenarios = list_scenarios() - assert len(scenarios) == 3 - assert len({scenario.media_id for scenario in scenarios}) == 3 + assert len(scenarios) == 5 + api_scenarios = [scenario for scenario in scenarios if scenario.kind == "api"] + assert len(api_scenarios) == 3 + assert len({scenario.media_id for scenario in api_scenarios}) == 3 for scenario in scenarios: - assert set(asdict(scenario)) == {"scenario_id", "task", "media_source", "media_id", "title", "magnet", "infohash"} - assert scenario.infohash in scenario.magnet + assert set(asdict(scenario)) == { + "scenario_id", "task", "media_source", "media_id", "title", "magnet", "infohash", + "kind", "command", "browser_url", + } assert "JSON" in scenario.task - assert "默认下载器和目录已配置" in scenario.task + assert scenario.scenario_id not in scenario.task + if scenario.kind == "api": + assert scenario.infohash in scenario.magnet + assert "默认下载器和目录已配置" in scenario.task + assert scenario.command == "" + assert scenario.browser_url == "" + elif scenario.kind == "command": + assert scenario.command + assert scenario.browser_url == "" + assert not scenario.media_id and not scenario.infohash + else: + assert scenario.kind == "browser" + assert scenario.browser_url == "__EVALUATION_BROWSER_URL__" + assert not scenario.media_id and not scenario.infohash assert "旧译名" not in scenario.task with pytest.raises(FrozenInstanceError): scenario.title = "污染输入"