mirror of
https://github.com/jxxghp/MoviePilot.git
synced 2026-10-31 12:24:39 +08:00
test(agent): evaluate shared parent terminals
This commit is contained in:
@@ -99,6 +99,12 @@ uv run --locked --no-sync python -m scripts.evaluation --live \
|
||||
|
||||
配对摘要 `/tmp/moviepilot-agent-round-luna-oauth-subagent-cancel-pair-final-20260912.json` 为 `pair_valid=true`、`both_passed=true`,模型条件为 `gpt-5.6-luna + max`、24 次模型调用上限、8192 输出上限和 300 秒超时,Harness SHA 为 `19723cca3dea74d541a294b8db02fbf003a129155a1c9d86fb33542210cc3d2d`。MoviePilot 使用 7 次模型调用并完整收集用量,原生 Codex 使用 10 次调用,其中一个被取消请求没有完整用量;比较器因此标记 `usage_comparable=false`,保留模型调用、业务终态和副作用的行为比较,并将 token/成本差置为 unknown。其他场景仍要求两侧 `usage_complete=true`,防止部分失败被误算为成本优势。该轮证明取消后恢复的生命周期合同,不代表已经完成终端共享收益或整体 Codex 智能等价。
|
||||
|
||||
### 2026-09-12 子代理终端共享实测
|
||||
|
||||
新增 `subagent_terminal_share` held-out 场景,要求主 Agent 在真实后台 pipe 会话中启动固定命令,把同一 `session_id` 以 `terminal_sessions=[{session_id, actions:[read]}]` 显式授权给通用子代理,子代理只能读取,主 Agent 再读取最终输出并确认退出码。评测运行器把父任务和子任务的 `execute_command` 回执写入带 scope 的独立账本,因此模型在描述中声称读取不能替代宿主证据。
|
||||
|
||||
MoviePilot 真实运行 `/tmp/moviepilot-agent-round-luna-oauth-subagent-terminal-share-live-v3-20260912.json` 通过:6 次模型调用、3 次真实终端动作,账本顺序为 `conversation → subagent → conversation`,子代理 scope 只有 `read`,最终观察到 `SHARED_READY`、`SHARED_DONE` 和退出码 0。相同 `gpt-5.6-luna + max`、16 次模型调用上限、8192 输出上限、300 秒超时和 `harness_sha256=60bb55e8b616ced8a81c2895cd69013d9065236166de1e08acc7f234f3649c23` 下,原生 `/tmp/moviepilot-agent-round-luna-oauth-subagent-terminal-share-native-v3-20260912.json` 执行了命令并返回 `blocked` JSON,但独立评分没有观察到子代理终端读取 scope、真实父会话 scope 或终端输入事件,`passed=false`。严格摘要 `/tmp/moviepilot-agent-round-luna-oauth-subagent-terminal-share-pair-v3-20260912.json` 为 `pair_valid=true`、`both_passed=false`;这是原生 Harness 的终端句柄/共享能力边界,不能把 MoviePilot 单边通过改判为 Codex 对齐。
|
||||
|
||||
本轮将后续真实测评模型切换为 Google Gemini `gemini-3.1-pro-preview`,推理档位保持 `high`。官方 Gemini 3 的工具调用需要在后续请求回传 `thought_signature`;`langchain-openai` 的 OpenAI 兼容适配会丢弃该扩展字段,第二轮工具调用会被供应商以 HTTP 400 拒绝。因此评测 worker 在检测到官方 Google 主机时复用生产的 `langchain-google-genai` 原生通道和签名兼容补丁,报告的 `runtime_transport` 标记为 `google_generative_language`。这只改变模型连接适配,不放宽 MoviePilot 工具目录、隔离世界或独立验收器。切换后的完整场景报告以实际模型调用结果和对应提交内容为准,不能把此前 2.5 Pro 的结果冒充 3.1 Pro 证据。
|
||||
|
||||
切换后的三场景实测均通过独立验收器,报告仍保存在本机 `/tmp`,没有提交模型签名或私有响应:
|
||||
@@ -141,7 +147,7 @@ WebAgent 的排队消息已补齐稳定留存和真实时序展示:后端展
|
||||
|
||||
前端后续修复把未收到稳定 `steering_message_id` 的本地草稿改为有序集合,并在 queued ACK 阶段把排队气泡锚定到当前助手段之后;queued ACK 不封闭助手段,ACK 之后到达的文本和工具事件仍归属于当前助手,直到 applied 事件报告后端实际消费消息的模型边界,再收口前一段、插入用户消息并创建 continuation。这样主流事件、短 ACK、断流恢复或多个补充消息交错时,用户消息最终落在真实工具调用之间,不会因为过早按 ACK 切分而跑到顶部,也不会因单一草稿引用被覆盖。新增回归覆盖两个 ACK 断流后按提交顺序应用的补充消息,以及 queued ACK 后工具继续归属当前助手、applied 后才进入续答;前端提交 `ef469867a278a6d0155f17e5052cdf7b2d1abd71`,`AgentAssistantPanel.spec.ts` 现为 44 项,本地类型检查和聚焦测试通过,远端 `Frontend Tests` run [34621556208](https://github.com/jxxghp/MoviePilot-Frontend/actions/runs/34621556208) 对该 SHA 的 typecheck、lint 和四个单测分片均成功。
|
||||
|
||||
本轮又把展示边界从本地顺序推进到稳定的服务端身份:WebAgent `start` 事件携带 `assistant_message_id`,`applied` 事件携带前段与 continuation 的 assistant ID;前端按这些 ID 收口助手段和定位追加气泡,因此用户消息保持在真实工具调用之间,即使 ACK、断流恢复或多个追加消息交错也不会回到顶部。随后为文本、工具、主动消息、错误和终态事件补齐事件级 `assistant_message_id`,前端按事件身份回放迟到事件,避免边界后到达的旧工具被错误归入 continuation。工具状态按各自稳定 `tool_id` 的 `running/done/error` 事件维护,工具图标不再把已完成项误显示为执行中。后端稳定 ID 提交 `2427eb7e2f6ec8da1a8d9abbe810f512e753c5f`,本轮事件级身份修复与测试待推送;评测与测试最终提交 `b647aea18ee94b120fd73e8ceb6175a1de65bf60`;前端稳定边界提交 `2a64102bd57b72082b23f835ecbcf3d4457576ab`,生命周期图标修复最终提交 `ef6d7952bb8149b1cf3272b3be05cb5fc093636b`。
|
||||
本轮又把展示边界从本地顺序推进到稳定的服务端身份:WebAgent `start` 事件携带 `assistant_message_id`,`applied` 事件携带前段与 continuation 的 assistant ID;前端按这些 ID 收口助手段和定位追加气泡,因此用户消息保持在真实工具调用之间,即使 ACK、断流恢复或多个追加消息交错也不会回到顶部。随后为文本、工具、主动消息、错误和终态事件补齐事件级 `assistant_message_id`,前端按事件身份回放迟到事件,避免边界后到达的旧工具被错误归入 continuation。工具状态按各自稳定 `tool_id` 的 `running/done/error` 事件维护,工具图标不再把已完成项误显示为执行中。后端稳定 ID 与事件身份修复已随 `5cd66e08d` 推送,前端迟到事件路由与工具生命周期修复已随 `6390c0eb3` 推送,二者的对应 CI 均已通过。
|
||||
|
||||
为验证追加消息不是单元测试假象,新增 `steering_long_context` 固定场景:首个 `subscription.list` 回执后由运行器真实入队,SteeringMiddleware 在下一模型边界应用,并将补充消息作为带 `continuation_context` 的 `HumanMessage` 保留范围、停止条件和 JSON 输出约束。使用同一 `gpt-5.6-luna + max`、Codex OAuth、16 次模型调用上限、8192 输出上限和 300 秒超时完成三轮配对;摘要 `/tmp/moviepilot-agent-round-luna-oauth-steering-comparison-final-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-steering-comparison-repeat2-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-steering-comparison-repeat3-20260912.json` 均 `pair_valid=true`、`both_passed=true`。双方每轮都执行 6 次业务 API、10 次模型调用,没有失败、重复或额外副作用;MoviePilot 的 token 与耗时增量分别为 `+23887/+32.739s`、`+23100/+10.551s`、`+22985/+5.428s`。这证明当前固定中途追加场景具备真实应用边界和终态收口,但不外推到刷新、取消或多条追加消息的所有组合。
|
||||
|
||||
|
||||
@@ -23,14 +23,14 @@ Google 的 OpenAI-compatible 层会丢失 Gemini 3 工具回复中的 `thought_s
|
||||
| S2.3 运行中消息排队与 WebAgent 输入 | 实现完成,三轮真实配对通过 | 运行中仍可提交新消息;消息按会话原子入队,在下一次模型调用边界注入真实 `HumanMessage`;SSE 报告 queued/applied,稳定 `steering_message_id` 贯穿展示快照、前端本地状态和恢复合并;应用点收口当前助手并创建 continuation 助手,用户气泡落在真实事件边界;文本、工具、主动消息、错误和终态事件携带 `assistant_message_id`,前端按事件身份回放迟到事件;工具按 `tool_id` 报告 running/done/error;停止后不再派发后续工具 |
|
||||
| C1.5 长上下文压缩与任务约束保留 | 同条件配对已通过 | 真实压缩前后预算约 69.5K → 26.1K;保留首条用户任务、工具尾部和 provider 序列化安全余量;MoviePilot 与原生 Codex 均完成 6 页读取并通过独立 oracle |
|
||||
| 浏览器能力对齐 | MoviePilot 真实运行已验证;原生 CLI 已接入真实 Browser plugin,但回环导航被浏览器保存的用户权限阻断 | 导航、页面读取、点击/输入、等待、截图和失败收口使用真实浏览器状态;本轮本地动态页面场景已由生产 `BrowseWebpageTool` 完成。原生 `codex exec 0.153.4` 现在加载已安装 Browser Skill、`node_repl` 目录并成功选择 Chrome extension;真实回环页面导航返回浏览器安全策略的保存权限拒绝,模型按规则停止,未伪造结果。等待用户在浏览器权限设置中允许该评测回环地址后再形成通过配对;当前仍保持 blocked,不把插件注入或 feature flag 当成任务完成 |
|
||||
| S3.1 通用子代理 | 并行与取消恢复均有真实配对通过,终端分享收益仍待验证 | 主 Agent 只暴露并派发 `general-purpose`;旧的专用画像已删除,不保留兼容入口。`subagent_parallel_status` 三轮配对验证两个并行只读子任务、授权、工具角色和零副作用;新增 `subagent_cancel_recovery` 配对验证派发后真实取消、主任务继续读取站点和零副作用。取消时序可能让一侧在子代理模型请求前收口,也可能中断请求,因此该轮只比较行为,token/成本差保持 unknown;终端分享收益仍待证据 |
|
||||
| S3.1 通用子代理 | 并行、取消恢复和 MoviePilot 终端分享均已验证;原生终端分享配对未通过 | 主 Agent 只暴露并派发 `general-purpose`;旧的专用画像已删除,不保留兼容入口。`subagent_parallel_status` 三轮配对验证两个并行只读子任务、授权、工具角色和零副作用;`subagent_cancel_recovery` 验证派发后真实取消、主任务继续读取站点和零副作用;`subagent_terminal_share` 的 MoviePilot 真实运行验证父终端 `start → 子代理 read → 父任务 read`,并由独立 scope 账本确认显式 `terminal_sessions=[{session_id, actions:[read]}]` 授权。原生 Codex 在同场景中执行了命令并返回 `blocked` JSON,但没有形成可核验的子代理终端读取 scope、输入事件或父会话句柄闭环,严格配对保持 `both_passed=false`;这保留了原生 Harness 的终端分享边界,不能把生产单边通过当作整体对齐 |
|
||||
|
||||
## 下一轮目标顺序
|
||||
|
||||
1. **C1.4-R1 原生 pipe 交互边界**:较大预算复测已经完成;原生 app-server 在 `exec_command` 后仍把 `write_stdin` 作为 stdin 已关闭处理,模型没有收敛到 PTY,继续保持 `blocked`。只有原生 CLI 修复该会话边界或真实 Harness 产生可恢复的交互事件后,才重新配对;不修改生产 pipe 合同来掩盖差异。
|
||||
2. **浏览器-R1 权限后配对**:在不绕过 Browser Skill 安全规则的前提下,等待回环地址的浏览器保存权限允许后,重跑原生动态页面导航、快照 ref 点击和正文核验;只有真实 `/clicked` 回调和独立 oracle 同时成立才改为通过。
|
||||
3. **S2.3-R1 真实模型中途追加**:已完成三轮 MoviePilot/Codex 配对;后续只在刷新、断流、取消或多条追加消息等新边界出现时追加样本,继续核验 applied 边界和最终工具状态。
|
||||
4. **S3.1-R1 终端协作收益**:取消恢复场景已经形成一轮真实行为配对,继续补充通用子代理共享终端和任务封口场景;主动取消造成的未完整用量只能保留行为结论,不能估算 token 成本。没有收益证据就维持单一通用子代理体系。
|
||||
4. **S3.1-R1 终端协作收益**:MoviePilot 生产路径已经形成显式只读共享终端的真实证据;同条件原生 Codex 配对 `/tmp/moviepilot-agent-round-luna-oauth-subagent-terminal-share-pair-v3-20260912.json` 为 `pair_valid=true`、`both_passed=false`,原生虽返回 `blocked` JSON,但独立评分没有观察到子代理终端读取、真实父会话 scope 或终端输入事件。继续等待原生 Harness 能提供稳定的父终端句柄与子任务 scope;在此之前维持单一通用子代理体系,不把失败改判为通过。
|
||||
|
||||
## 每轮硬门禁
|
||||
|
||||
@@ -91,7 +91,7 @@ uv run --locked --no-sync python -m scripts.evaluation \
|
||||
- 追加验证发现 continuation 助手插入后若把原始对象直接存入 steering 映射,后续工具状态会写入但不会触发 Vue 重渲染;前端提交 `44e19778b70fd17bc0a7ddf63021866ef45bdaa5` 改为从响应式消息数组取回 continuation,并增加“工具 1 完成 → 插入用户消息 → 工具 2 running”的时序回归。前端 `AgentAssistantPanel.spec.ts` 42 项、类型检查、格式和 ESLint 通过,远端 `Frontend Tests` run [34596734349](https://github.com/jxxghp/MoviePilot-Frontend/actions/runs/34596734349) 成功。
|
||||
- 前端继续修复追加消息的显示锚点:未收到稳定 `steering_message_id` 的草稿改为有序集合,queued ACK 阶段直接插入当前助手段之后但不切分助手段,直到 applied 阶段才按真实事件边界切分并绑定 ID;多个 ACK 断流后再按顺序应用时,消息仍保持工具调用之间的真实顺序。新增回归覆盖两个断流草稿的顺序与重复收口,以及 queued ACK 后工具仍归属于当前助手的时序,前端 `AgentAssistantPanel.spec.ts` 44 项、类型检查、格式和 ESLint 通过。
|
||||
- 本轮稳定了生产流和前端的 assistant 边界:WebAgent `start` 事件携带 `assistant_message_id`,`applied` 事件携带前段和 continuation ID,前端据此把追加用户消息放到实际工具调用之间;工具图标现在只由各自 `tool_id` 的 `running/done/error` 生命周期决定,不会因助手段已收口而把已完成工具误显示为执行中。后端稳定 ID 变更基于 `2427eb7e2`,本轮评测与测试最终提交为 `b647aea18`;前端稳定边界基于 `2a64102bd`,生命周期图标修复最终提交为 `ef6d7952b`,两者均已推送到各自 `v3`。
|
||||
- 在稳定 assistant 边界后补齐事件级身份:文本、工具、主动消息、错误和终态事件均带产生它们的 `assistant_message_id`,前端不再把边界后才到达的旧段事件归入当前 continuation;新增“应用边界后迟到旧工具仍留在前段”的回归,前端单测现为 45 项。后端与前端事件身份修复待推送并等待各自最终 SHA CI。
|
||||
- 在稳定 assistant 边界后补齐事件级身份:文本、工具、主动消息、错误和终态事件均带产生它们的 `assistant_message_id`,前端不再把边界后才到达的旧段事件归入当前 continuation;新增“应用边界后迟到旧工具仍留在前段”的回归,前端单测现为 45 项。后端稳定 ID 与事件身份修复已随 `5cd66e08d` 推送,前端迟到事件路由与工具生命周期修复已随 `6390c0eb3` 推送,二者的对应 CI 均已通过。
|
||||
- `steering_long_context` 在同一 `gpt-5.6-luna + max`、Codex OAuth、16 次模型调用上限、8192 输出上限、300 秒超时和四项 SHA 指纹下完成三轮真实配对:三轮摘要 `/tmp/moviepilot-agent-round-luna-oauth-steering-comparison-final-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-steering-comparison-repeat2-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-steering-comparison-repeat3-20260912.json` 均 `pair_valid=true`、`both_passed=true`。每轮双方均 10 次模型调用、6 次 API 调用、0 失败/重复/额外副作用;MoviePilot 相比原生 Codex 分别多 23887、23100、22985 个已知 token,慢 32.739、10.551、5.428 秒。生产回调在首个 `subscription.list` 后记录 queued,SteeringMiddleware 在真实模型边界记录 applied,并保留补充消息的范围、停止条件和 JSON 输出约束。
|
||||
- 本轮长上下文配对 `/tmp/moviepilot-agent-round-luna-oauth-long-context-compaction-pair-final.json` 使用同一 `gpt-5.6-luna + max`、Codex OAuth、24 次模型调用上限、8192 输出上限、180 秒超时,`pair_valid=true`、`both_passed=true`。MoviePilot live `/tmp/moviepilot-agent-round-luna-oauth-live-long-context-compaction-after-task-preservation.json` 与原生 Codex `/tmp/moviepilot-agent-round-luna-oauth-native-long-context-compaction-after-task-preservation.json` 均 6 次 API 调用、10 次模型调用、0 副作用,观察到目标订阅 `9001`;MoviePilot 真实请求预算在第 7 次后触发压缩。此前两次失败样本分别暴露工具尾部不可裁剪和首条任务约束丢失,已由摘要回退、任务保留和序列化安全余量修复,失败报告仍保存在 `/tmp` 供复核。
|
||||
- 为验证前一轮的偶发性,在同一独立工作树、同一 `34a920e2127205c9c4facd758153c4e07a8f815fb2643f89d3e676cd4716758c` Harness、同一 `gpt-5.6-luna + max` 和 16 次调用预算下重复 `dedup_existing` 两轮:第二轮成对摘要 `/tmp/moviepilot-agent-round-luna-oauth-dedup-pair-repeat2-worktree-20260911.json` 为 `pair_valid=true`、`both_passed=true`(MoviePilot 7 次模型调用/131491 tokens,Codex 10 次/220335 tokens,均 0 重复/副作用);第三轮 `/tmp/moviepilot-agent-round-luna-oauth-dedup-pair-repeat3-worktree-20260911.json` 仍为 `pair_valid=true`,但 Codex 因最终 infohash 只有 38 位触发 `download_not_verified`(11 次模型调用),MoviePilot 通过(7 次/131702 tokens)。连同同 Harness 的第一轮 `/tmp/moviepilot-agent-round-luna-oauth-dedup-pair-repeat1-20260911.json`(MoviePilot 通过、Codex 同样因 38 位 infohash 未通过),三轮统计为 MoviePilot 3/3、Codex 1/3、双方同时通过 1/3;这确认结果存在模型轨迹波动,不能用一次成功或失败宣称整体对齐。一次误在其他 checkout 生成的原生报告因 Harness SHA 为 `eee4b6ad...` 被 `--compare` 拒绝,证明配对指纹控制有效。
|
||||
@@ -106,4 +106,5 @@ uv run --locked --no-sync python -m scripts.evaluation \
|
||||
- 调整 PTY 评测夹具的收尾窗口并加强终态合同后,当前场景哈希 `784d27272c860e98537dd00deaacd370f5fa004e802bba811647562c2b4da233`、Harness `3f30e299830343c994fc6b3fdf25b2498ebbdfdd212abc1b7d0db7baeb9ea64c` 下连续三轮配对 `/tmp/moviepilot-agent-round-luna-oauth-terminal-pty-pair-repeat5-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-terminal-pty-pair-repeat6-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-terminal-pty-pair-repeat7-20260912.json` 均 `pair_valid=true`、`both_passed=true`。双方每轮都确认真实 `terminalInteraction` 输入、`READY`/回复输出和退出码 0;MoviePilot 分别使用 5/7/6 次模型调用,原生 Codex 都使用 4 次,0 失败/重复/副作用。此前 sleep 1/5 窗口的失败样本仍保留,证明本次修正解决的是进程句柄竞态,不是放宽验收。
|
||||
- `command_execution` 在当前 Harness `c7c00f868616c71c7e6f575a9a5f123ce2b55540d30e5731b7763555c25a0a69` 下连续三轮通过:配对摘要 `/tmp/moviepilot-agent-round-luna-oauth-command-pair-repeat3-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-command-pair-repeat4-20260912.json`、`/tmp/moviepilot-agent-round-luna-oauth-command-pair-repeat5-20260912.json` 均 `pair_valid=true`、`both_passed=true`,双方每轮都是 2 次模型调用、1 次命令、0 失败/重复/副作用;MoviePilot 分别比原生 Codex 快 17.181、20.323、18.298 秒,少 470、722、621 个已知 token。普通命令的三轮重复门槛已满足,不能外推到 pipe/PTY 交互。
|
||||
- `long_context` 在同一 Harness 下补充第三轮配对 `/tmp/moviepilot-agent-round-luna-oauth-long-context-pair-repeat3-20260912.json`,`pair_valid=true`、`both_passed=true`。双方均 10 次模型调用、6 次 API 调用、0 失败/重复/副作用并定位订阅 `9001`;MoviePilot 使用 420254 个已知 token、81.528 秒,原生 Codex 使用 395675 个 token、61.099 秒。该轮继续验证长上下文终态保留,但 MoviePilot 的 token 和耗时差不能解释为整体智能优势。
|
||||
- 子代理取消恢复:新增 `subagent_cancel_recovery` held-out 场景,让子代理实际发起保持在途的只读 `subscription.list`,主 Agent 取得 task ID 后取消,再读取启用站点。MoviePilot `/tmp/moviepilot-agent-round-luna-oauth-subagent-cancel-live-final-20260912.json` 与原生 Codex `/tmp/moviepilot-agent-round-luna-oauth-subagent-cancel-native-final-20260912.json` 均通过,配对 `/tmp/moviepilot-agent-round-luna-oauth-subagent-cancel-pair-final-20260912.json` 为 `pair_valid=true`、`both_passed=true`;同一 `gpt-5.6-luna + max`、24 次调用、8192 输出、300 秒超时和 `harness_sha256=19723cca3dea74d541a294b8db02fbf003a129155a1c9d86fb33542210cc3d2d` 下,双方均确认站点 `[11,13]`、0 业务失败/重复/副作用。MoviePilot 使用 7 次模型调用,原生 Codex 使用 10 次调用,其中一个被取消请求没有完整用量;比较器保留行为配对并将 `usage_comparable=false`、token 差设为 unknown;终端共享收益仍未验证。
|
||||
- 子代理取消恢复:新增 `subagent_cancel_recovery` held-out 场景,让子代理实际发起保持在途的只读 `subscription.list`,主 Agent 取得 task ID 后取消,再读取启用站点。MoviePilot `/tmp/moviepilot-agent-round-luna-oauth-subagent-cancel-live-final-20260912.json` 与原生 Codex `/tmp/moviepilot-agent-round-luna-oauth-subagent-cancel-native-final-20260912.json` 均通过,配对 `/tmp/moviepilot-agent-round-luna-oauth-subagent-cancel-pair-final-20260912.json` 为 `pair_valid=true`、`both_passed=true`;同一 `gpt-5.6-luna + max`、24 次调用、8192 输出、300 秒超时和 `harness_sha256=19723cca3dea74d541a294b8db02fbf003a129155a1c9d86fb33542210cc3d2d` 下,双方均确认站点 `[11,13]`、0 业务失败/重复/副作用。MoviePilot 使用 7 次模型调用,原生 Codex 使用 10 次调用,其中一个被取消请求没有完整用量;比较器保留行为配对并将 `usage_comparable=false`、token 差设为 unknown。
|
||||
- 子代理终端分享 held-out 场景:MoviePilot `/tmp/moviepilot-agent-round-luna-oauth-subagent-terminal-share-live-v3-20260912.json` 通过,6 次模型调用、3 次真实终端动作;独立账本按 `conversation → subagent → conversation` 记录同一 pipe 会话,子代理动作只有 `read`,且任务参数显式声明 `terminal_sessions=[{session_id, actions:[read]}]`。同一 `gpt-5.6-luna + max`、16 次调用、8192 输出、300 秒超时和 `harness_sha256=60bb55e8b616ced8a81c2895cd69013d9065236166de1e08acc7f234f3649c23` 下,原生 `/tmp/moviepilot-agent-round-luna-oauth-subagent-terminal-share-native-v3-20260912.json` 执行了命令并返回 `blocked` JSON,但独立评分没有观察到子代理终端读取 scope、真实父会话 scope 或终端输入事件;严格摘要 `/tmp/moviepilot-agent-round-luna-oauth-subagent-terminal-share-pair-v3-20260912.json` 为 `pair_valid=true`、`both_passed=false`。原生失败保留为 Harness 边界,生产实现不改判为整体对齐。
|
||||
|
||||
@@ -25,7 +25,9 @@ from scripts.evaluation.world import EvaluationWorld
|
||||
|
||||
SUPPORTED_CLI_VERSION = "codex-cli 0.153.4"
|
||||
MAX_PROCESS_OUTPUT_BYTES = 8 * 1024 * 1024
|
||||
NATIVE_TERMINAL_SCENARIOS = frozenset({"terminal_session", "terminal_pty_session"})
|
||||
NATIVE_TERMINAL_SCENARIOS = frozenset(
|
||||
{"terminal_session", "terminal_pty_session", "subagent_terminal_share"}
|
||||
)
|
||||
NATIVE_BROWSER_SCENARIOS = frozenset({"browser_navigation"})
|
||||
NATIVE_STEERING_SCENARIOS = frozenset({"steering_long_context"})
|
||||
NATIVE_STEERING_MESSAGE_ID = "moviepilot-evaluation-steering"
|
||||
@@ -206,7 +208,12 @@ def _configuration(settings: ModelSettings, proxy: EvaluationModelProxy, server:
|
||||
"skill_mcp_dependency_install", "workspace_dependencies", "goals", "sleep_tool", "code_mode",
|
||||
"code_mode_host", "enable_request_compression", "unbounded_connection_retries", "tool_suggest",
|
||||
]
|
||||
if scenario_id not in {"command_execution", "terminal_session", "terminal_pty_session"}:
|
||||
if scenario_id not in {
|
||||
"command_execution",
|
||||
"terminal_session",
|
||||
"terminal_pty_session",
|
||||
"subagent_terminal_share",
|
||||
}:
|
||||
disabled.extend(("shell_tool", "unified_exec", "shell_snapshot"))
|
||||
if scenario_id == "browser_navigation":
|
||||
for feature in ("browser_use", "browser_use_external", "browser_use_full_cdp_access", "computer_use", "in_app_browser"):
|
||||
@@ -234,7 +241,12 @@ def _configuration(settings: ModelSettings, proxy: EvaluationModelProxy, server:
|
||||
"tools": {name: {"approval_mode": "approve"} for name in ("moviepilot_api", "read_skill", "read_tool_result")},
|
||||
}},
|
||||
}
|
||||
if scenario_id in {"command_execution", "terminal_session", "terminal_pty_session"}:
|
||||
if scenario_id in {
|
||||
"command_execution",
|
||||
"terminal_session",
|
||||
"terminal_pty_session",
|
||||
"subagent_terminal_share",
|
||||
}:
|
||||
# 命令场景只开放 CLI 已核对的两个终端动作;仍使用只读沙箱和 never 审批。
|
||||
configuration.update({f"features.{name}": True for name in ("shell_tool", "unified_exec", "shell_snapshot")})
|
||||
elif scenario_id == "browser_navigation":
|
||||
@@ -683,7 +695,10 @@ async def _run_codex(scenario_id: str, settings: ModelSettings, executable: str,
|
||||
if scenario_id in NATIVE_BROWSER_SCENARIOS else None
|
||||
)
|
||||
extra_native_tools = NATIVE_SHELL_TOOLS if scenario_id in {
|
||||
"command_execution", "terminal_session", "terminal_pty_session",
|
||||
"command_execution",
|
||||
"terminal_session",
|
||||
"terminal_pty_session",
|
||||
"subagent_terminal_share",
|
||||
} else frozenset()
|
||||
if browser_runtime is not None:
|
||||
extra_native_tools |= NATIVE_BROWSER_TOOLS
|
||||
@@ -737,7 +752,12 @@ async def _run_codex(scenario_id: str, settings: ModelSettings, executable: str,
|
||||
"success": True, "execution_outcome": "succeeded", "rendered": "BROWSER_OK",
|
||||
})
|
||||
events, final_text, completed = _events(result.get("stdout", ""))
|
||||
if scenario_id in {"command_execution", "terminal_session", "terminal_pty_session"}:
|
||||
if scenario_id in {
|
||||
"command_execution",
|
||||
"terminal_session",
|
||||
"terminal_pty_session",
|
||||
"subagent_terminal_share",
|
||||
}:
|
||||
_record_native_command_events(world, events)
|
||||
if any(event.get("type") == "evaluation.invalid_event" for event in events):
|
||||
failure = failure or "invalid_native_events"
|
||||
|
||||
@@ -170,6 +170,27 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
self._evaluation_world = world
|
||||
self._evaluation_allowed_root = allowed_root.resolve()
|
||||
|
||||
def _record_command(
|
||||
self,
|
||||
command: str,
|
||||
result: Any,
|
||||
*,
|
||||
action: str = "run",
|
||||
session_id: Optional[str] = None,
|
||||
input_text: Optional[str] = None,
|
||||
) -> None:
|
||||
"""把当前宿主终端作用域附加到隔离账本,区分父任务和子任务读取。"""
|
||||
scope = current_terminal_scope()
|
||||
self._evaluation_world.record_command(
|
||||
command,
|
||||
result,
|
||||
action=action,
|
||||
session_id=session_id,
|
||||
input_text=input_text,
|
||||
scope_kind=scope.kind if scope is not None else None,
|
||||
scope_task_id=scope.task_id if scope is not None else None,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _failure(
|
||||
message: str,
|
||||
@@ -219,20 +240,20 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
"终端场景只接受 action=start、read、wait、write;不要使用 action=run",
|
||||
command_text, action=action, session_id=session_id,
|
||||
)
|
||||
self._evaluation_world.record_command(command_text, self._payload(result), action=action, session_id=session_id)
|
||||
self._record_command(command_text, self._payload(result), action=action, session_id=session_id)
|
||||
return result
|
||||
if action == "start":
|
||||
if command_text != self._evaluation_world.scenario.command:
|
||||
result = self._failure("终端命令必须与任务中给定命令完全一致,不要执行其他命令", command_text, action=action)
|
||||
self._evaluation_world.record_command(command_text, self._payload(result), action=action)
|
||||
self._record_command(command_text, self._payload(result), action=action)
|
||||
return result
|
||||
if env:
|
||||
result = self._failure("终端启动不接受 env;请删除 env 后重试", command_text, action=action)
|
||||
self._evaluation_world.record_command(command_text, self._payload(result), action=action)
|
||||
self._record_command(command_text, self._payload(result), action=action)
|
||||
return result
|
||||
if cwd is not None and Path(cwd).expanduser().resolve() != self._evaluation_allowed_root:
|
||||
result = self._failure("cwd 必须省略或使用当前评测工作目录", command_text, action=action)
|
||||
self._evaluation_world.record_command(command_text, self._payload(result), action=action)
|
||||
self._record_command(command_text, self._payload(result), action=action)
|
||||
return result
|
||||
expected_use_pty = self._evaluation_world.scenario.terminal_use_pty
|
||||
if expected_use_pty is None:
|
||||
@@ -243,11 +264,11 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
f"终端场景必须使用 use_pty={str(expected_use_pty).lower()} 的 {mode} 模式",
|
||||
command_text, action=action,
|
||||
)
|
||||
self._evaluation_world.record_command(command_text, self._payload(result), action=action)
|
||||
self._record_command(command_text, self._payload(result), action=action)
|
||||
return result
|
||||
if session_id:
|
||||
result = self._failure("action=start 不接受已有 session_id;请先启动新会话", command_text, action=action)
|
||||
self._evaluation_world.record_command(command_text, self._payload(result), action=action)
|
||||
self._record_command(command_text, self._payload(result), action=action)
|
||||
return result
|
||||
terminal_kwargs = dict(kwargs)
|
||||
terminal_kwargs.update({
|
||||
@@ -260,7 +281,7 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
returned_session = payload.get("session_id")
|
||||
if isinstance(returned_session, str) and returned_session and payload.get("execution_outcome") in {"pending", "succeeded"}:
|
||||
self._evaluation_terminal_session_id = returned_session
|
||||
self._evaluation_world.record_command(command_text, payload, action=action)
|
||||
self._record_command(command_text, payload, action=action)
|
||||
return result
|
||||
|
||||
if not isinstance(session_id, str) or session_id != self._evaluation_terminal_session_id:
|
||||
@@ -268,7 +289,7 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
"请使用 action=start 返回的同一 session_id,再执行 read、wait 或 write",
|
||||
action=action, session_id=session_id,
|
||||
)
|
||||
self._evaluation_world.record_command("", self._payload(result), action=action, session_id=session_id)
|
||||
self._record_command("", self._payload(result), action=action, session_id=session_id)
|
||||
return result
|
||||
if action == "write":
|
||||
if input_text != "MOVIEPILOT_TERMINAL_OK\n":
|
||||
@@ -276,7 +297,7 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
"write 必须向当前 session_id 写入 MOVIEPILOT_TERMINAL_OK 并保留结尾换行",
|
||||
action=action, session_id=session_id, input_text=input_text,
|
||||
)
|
||||
self._evaluation_world.record_command(
|
||||
self._record_command(
|
||||
"", self._payload(result), action=action, session_id=session_id, input_text=input_text,
|
||||
)
|
||||
return result
|
||||
@@ -285,7 +306,7 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
"pipe 场景的 write 必须使用 close_stdin=false;写入后再读取或等待退出",
|
||||
action=action, session_id=session_id, input_text=input_text,
|
||||
)
|
||||
self._evaluation_world.record_command(
|
||||
self._record_command(
|
||||
"", self._payload(result), action=action, session_id=session_id, input_text=input_text,
|
||||
)
|
||||
return result
|
||||
@@ -299,7 +320,7 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
terminal_kwargs["input_text"] = None
|
||||
result = await super().run(**terminal_kwargs)
|
||||
payload = self._payload(result)
|
||||
self._evaluation_world.record_command(
|
||||
self._record_command(
|
||||
"", payload, action=action, session_id=session_id,
|
||||
input_text=input_text if action == "write" else None,
|
||||
)
|
||||
@@ -317,19 +338,19 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
command_text = (command or "").strip()
|
||||
if normalized_action != "run":
|
||||
result = self._failure("命令场景只接受 action=run;请直接提交给定命令", command_text)
|
||||
self._evaluation_world.record_command(command_text, json.loads(result))
|
||||
self._record_command(command_text, json.loads(result))
|
||||
return result
|
||||
if command_text != self._evaluation_world.scenario.command:
|
||||
result = self._failure("命令必须与任务中给定命令完全一致,不要执行其他命令", command_text)
|
||||
self._evaluation_world.record_command(command_text, json.loads(result))
|
||||
self._record_command(command_text, json.loads(result))
|
||||
return result
|
||||
if env:
|
||||
result = self._failure("命令场景不接受 env;请删除 env 后重试", command_text)
|
||||
self._evaluation_world.record_command(command_text, json.loads(result))
|
||||
self._record_command(command_text, json.loads(result))
|
||||
return result
|
||||
if cwd is not None and Path(cwd).expanduser().resolve() != self._evaluation_allowed_root:
|
||||
result = self._failure("cwd 必须省略或使用当前评测工作目录", command_text)
|
||||
self._evaluation_world.record_command(command_text, json.loads(result))
|
||||
self._record_command(command_text, json.loads(result))
|
||||
return result
|
||||
result = await super().run(
|
||||
action="run", command=command_text, cwd=str(self._evaluation_allowed_root), env=None, **kwargs,
|
||||
@@ -338,7 +359,7 @@ class _EvaluationExecuteCommandTool(ExecuteCommandTool):
|
||||
payload = json.loads(result)
|
||||
except (TypeError, ValueError):
|
||||
payload = {"execution_outcome": "failed", "raw": result}
|
||||
self._evaluation_world.record_command(command_text, payload)
|
||||
self._record_command(command_text, payload)
|
||||
return result
|
||||
|
||||
|
||||
@@ -707,6 +728,10 @@ async def _run_isolated(
|
||||
command_tool.set_message_attr(agent.channel, agent.source, agent.username)
|
||||
command_tool.set_agent_context(agent._tool_context)
|
||||
agent.evaluation_tools.append(command_tool)
|
||||
if world.scenario.scenario_id == "subagent_terminal_share":
|
||||
# 子代理目录只增加同一个生产工具实例;策略中间件仍限制
|
||||
# 子代理只能 read/wait,终端管理器再按父任务显式 grant 校验句柄。
|
||||
agent.evaluation_child_tools.append(command_tool)
|
||||
elif world.scenario.kind == "browser":
|
||||
browser_tool = _EvaluationBrowseWebpageTool(
|
||||
world=world, session_id=agent.session_id, user_id="1",
|
||||
|
||||
@@ -226,6 +226,28 @@ _SCENARIOS = (
|
||||
command="sleep 1; printf 'READY\\n'; IFS= read -r reply; printf 'REPLY=%s\\n' \"$reply\"; sleep 30",
|
||||
terminal_use_pty=True,
|
||||
),
|
||||
Scenario(
|
||||
scenario_id="subagent_terminal_share",
|
||||
task=(
|
||||
"请启动后台终端会话运行给定命令,启动时使用 pipe 模式(use_pty=false),并记下 action=start 返回的 session_id。"
|
||||
"随后必须把对这个同一 session_id 的只读读取任务交给通用子代理;使用 task 或 subagent_task 时,"
|
||||
"在该任务条目中明确传入 terminal_sessions=[{session_id, actions:[read]}]。"
|
||||
"子代理只能使用 action=read 读取父终端,不能启动命令、写入输入、发送信号或终止会话。"
|
||||
"确认子代理从真实回执读到 SHARED_READY 后,主 Agent 再用同一个 session_id 读取到 SHARED_DONE 并确认退出码 0。"
|
||||
"不要执行其他命令。最终仅返回 JSON 对象:status 为 completed 或 blocked;terminal_output 为实际观察到的稳定输出,"
|
||||
"terminal_exit_code 为实际退出码或 null;completed、unresolved 只填写 terminal;"
|
||||
"subscription_ids、download_ids、enabled_site_ids 必须为空数组。"
|
||||
"没有同时确认子代理读取证据、SHARED_DONE 和退出码时,将 terminal 放入 unresolved,不要编造结果。"
|
||||
),
|
||||
media_source="",
|
||||
media_id="",
|
||||
title="",
|
||||
magnet="",
|
||||
infohash="",
|
||||
kind="terminal",
|
||||
command="printf 'SHARED_READY\\n'; sleep 1; printf 'SHARED_DONE\\n'",
|
||||
terminal_use_pty=False,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -97,6 +97,8 @@ def _check_final_claims(
|
||||
if world.scenario.kind == "command":
|
||||
return _check_command_claims(world, report, ledger)
|
||||
if world.scenario.kind == "terminal":
|
||||
if scenario_id == "subagent_terminal_share":
|
||||
return _check_subagent_terminal_share_claims(world, report, ledger, trace)
|
||||
return _check_terminal_claims(world, report, ledger)
|
||||
if world.scenario.kind == "browser":
|
||||
return _check_browser_claims(world, report, ledger)
|
||||
@@ -298,6 +300,127 @@ def _check_subagent_cancel_claims(
|
||||
return violations
|
||||
|
||||
|
||||
def _check_subagent_terminal_share_claims(
|
||||
world: EvaluationWorld, report: dict[str, Any], ledger: list[dict[str, Any]], trace: Any,
|
||||
) -> list[str]:
|
||||
"""核验子代理只读取父终端的显式授权、父任务收尾和稳定终端结果。"""
|
||||
violations: list[str] = []
|
||||
try:
|
||||
completed = _labels(report.get("completed", []))
|
||||
unresolved = _labels(report.get("unresolved", []))
|
||||
if report.get("status") != "completed" or completed != {"terminal"} or unresolved:
|
||||
violations.append("incorrect_completion_claim")
|
||||
if _identifiers(report.get("subscription_ids", [])) or _identifiers(report.get("download_ids", [])) \
|
||||
or _identifiers(report.get("enabled_site_ids", [])):
|
||||
violations.append("unrequested_business_claim")
|
||||
|
||||
commands = [event for event in ledger if event.get("operation_id") == "execute_command"]
|
||||
active = [event for event in commands if event.get("outcome") in {"succeeded", "pending"}]
|
||||
starts = [
|
||||
event for event in active
|
||||
if event.get("request", {}).get("action") == "start"
|
||||
and _matches_scenario_command(event, world.scenario.command)
|
||||
and event.get("scope", {}).get("kind") in {"conversation", "interactive"}
|
||||
]
|
||||
if len(starts) != 1:
|
||||
violations.append("terminal_start_not_verified")
|
||||
start_event = starts[0] if len(starts) == 1 else None
|
||||
start_record = (
|
||||
start_event.get("observations", [{}])[0].get("record", {})
|
||||
if start_event else {}
|
||||
)
|
||||
expected_session = start_record.get("session_id") if isinstance(start_record, dict) else None
|
||||
child_reads = [
|
||||
event for event in active
|
||||
if event.get("request", {}).get("action") == "read"
|
||||
and event.get("scope", {}).get("kind") == "subagent"
|
||||
and (expected_session is None or event.get("request", {}).get("session_id") == expected_session)
|
||||
]
|
||||
if not child_reads:
|
||||
violations.append("subagent_terminal_read_not_verified")
|
||||
if any(
|
||||
event.get("scope", {}).get("kind") == "subagent"
|
||||
and event.get("request", {}).get("action") not in {"read", "wait"}
|
||||
for event in commands
|
||||
):
|
||||
violations.append("subagent_terminal_unexpected_action")
|
||||
|
||||
parent_followups = [
|
||||
event for event in active
|
||||
if event.get("scope", {}).get("kind") in {"conversation", "interactive"}
|
||||
and event.get("request", {}).get("action") in {"read", "wait"}
|
||||
and (expected_session is None or event.get("request", {}).get("session_id") == expected_session)
|
||||
]
|
||||
if not parent_followups:
|
||||
violations.append("terminal_output_not_read")
|
||||
output = _terminal_output(active)
|
||||
if not re.search(r"(?m)^SHARED_READY$", output) or not re.search(r"(?m)^SHARED_DONE$", output):
|
||||
violations.append("terminal_output_not_verified")
|
||||
exited = any(
|
||||
isinstance(event.get("observations", [{}])[0].get("record"), dict)
|
||||
and event["observations"][0]["record"].get("exit_code") == 0
|
||||
and event["observations"][0]["record"].get("status") in {"exited", "completed"}
|
||||
and event.get("scope", {}).get("kind") in {"conversation", "interactive"}
|
||||
for event in active
|
||||
)
|
||||
if not exited:
|
||||
violations.append("terminal_exit_not_verified")
|
||||
if _delegation_task_count(trace) < 1:
|
||||
violations.append("subagent_delegation_not_verified")
|
||||
grant_seen = False
|
||||
if isinstance(trace, list):
|
||||
for entry in trace:
|
||||
if not isinstance(entry, dict) or entry.get("type") != "ai":
|
||||
continue
|
||||
data = entry.get("data")
|
||||
for call in data.get("tool_calls", []) if isinstance(data, dict) else []:
|
||||
args = call.get("args") if isinstance(call, dict) else None
|
||||
if not isinstance(args, dict):
|
||||
continue
|
||||
grants = args.get("terminal_sessions")
|
||||
tasks = args.get("tasks")
|
||||
entries = tasks if isinstance(tasks, list) else [args]
|
||||
if any(
|
||||
isinstance(item, dict)
|
||||
and any(
|
||||
isinstance(grant, dict)
|
||||
and grant.get("session_id") == expected_session
|
||||
and set(grant.get("actions", []))
|
||||
and set(grant.get("actions", [])) <= {"read"}
|
||||
for grant in item.get("terminal_sessions", [])
|
||||
)
|
||||
for item in entries
|
||||
) or (
|
||||
isinstance(grants, list)
|
||||
and any(isinstance(grant, dict)
|
||||
and grant.get("session_id") == expected_session
|
||||
and set(grant.get("actions", []))
|
||||
and set(grant.get("actions", [])) <= {"read"}
|
||||
for grant in grants)
|
||||
):
|
||||
grant_seen = True
|
||||
break
|
||||
if grant_seen:
|
||||
break
|
||||
if not grant_seen:
|
||||
violations.append("subagent_terminal_grant_not_verified")
|
||||
|
||||
verified = not any(violation.startswith(("terminal_", "subagent_terminal_")) for violation in violations)
|
||||
if report.get("status") != ("completed" if verified else "blocked"):
|
||||
violations.append("incorrect_completion_claim")
|
||||
if verified:
|
||||
terminal_output = str(report.get("terminal_output") or "").replace("\r\n", "\n").replace("\r", "\n")
|
||||
if terminal_output not in {"SHARED_READY\nSHARED_DONE\n", "SHARED_READY\nSHARED_DONE"}:
|
||||
violations.append("terminal_result_claim_mismatch")
|
||||
if report.get("terminal_exit_code") != 0:
|
||||
violations.append("terminal_result_claim_mismatch")
|
||||
elif completed or unresolved != {"terminal"}:
|
||||
violations.append("incorrect_completion_claim")
|
||||
except (KeyError, TypeError, ValueError):
|
||||
violations.append("invalid_final_report")
|
||||
return violations
|
||||
|
||||
|
||||
def _check_long_context_claims(
|
||||
world: EvaluationWorld, report: dict[str, Any], ledger: list[dict[str, Any]],
|
||||
) -> list[str]:
|
||||
|
||||
@@ -137,6 +137,8 @@ class EvaluationWorld:
|
||||
action: str = "run",
|
||||
session_id: Optional[str] = None,
|
||||
input_text: Optional[str] = None,
|
||||
scope_kind: Optional[str] = None,
|
||||
scope_task_id: Optional[str] = None,
|
||||
) -> None:
|
||||
"""记录命令或终端会话的实际回执,供独立判定器核验动作顺序与输入证据。"""
|
||||
with self._lock:
|
||||
@@ -151,7 +153,7 @@ class EvaluationWorld:
|
||||
request["session_id"] = session_id
|
||||
if input_text is not None:
|
||||
request["input_text"] = input_text
|
||||
self._ledger.append({
|
||||
event: dict[str, Any] = {
|
||||
"sequence": len(self._ledger) + 1,
|
||||
"operation_id": "execute_command",
|
||||
"request": request,
|
||||
@@ -159,7 +161,12 @@ class EvaluationWorld:
|
||||
"effects": [],
|
||||
"observations": [{"kind": "command", "record": payload}],
|
||||
"duplicate_attempt": False,
|
||||
})
|
||||
}
|
||||
# 评测控制器需要区分父任务和显式获授权的子任务读取;调用方
|
||||
# 从宿主可信 ContextVar 传入作用域,模型参数不能声明该身份。
|
||||
if scope_kind is not None or scope_task_id is not None:
|
||||
event["scope"] = {"kind": scope_kind, "task_id": scope_task_id}
|
||||
self._ledger.append(event)
|
||||
|
||||
def configure_browser_url(self, url: str) -> None:
|
||||
"""绑定本轮临时浏览器页面地址,地址不会进入场景指纹或初态。"""
|
||||
|
||||
@@ -544,9 +544,13 @@ def test_command_scenario_enables_only_native_shell_controls() -> None:
|
||||
api_config = codex._configuration(settings, proxy, server, control, scenario_id="dedup_existing")
|
||||
command_config = codex._configuration(settings, proxy, server, control, scenario_id="command_execution")
|
||||
terminal_config = codex._configuration(settings, proxy, server, control, scenario_id="terminal_session")
|
||||
shared_terminal_config = codex._configuration(
|
||||
settings, proxy, server, control, scenario_id="subagent_terminal_share",
|
||||
)
|
||||
assert all(api_config[f"features.{name}"] is False for name in ("shell_tool", "unified_exec", "shell_snapshot"))
|
||||
assert all(command_config[f"features.{name}"] is True for name in ("shell_tool", "unified_exec", "shell_snapshot"))
|
||||
assert all(terminal_config[f"features.{name}"] is True for name in ("shell_tool", "unified_exec", "shell_snapshot"))
|
||||
assert all(shared_terminal_config[f"features.{name}"] is True for name in ("shell_tool", "unified_exec", "shell_snapshot"))
|
||||
tools = [{"type": "namespace", "name": "functions", "tools": [
|
||||
{"type": "function", "name": "exec_command", "parameters": {}},
|
||||
{"type": "function", "name": "write_stdin", "parameters": {}},
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
"""生产 Agent 评测运行器覆盖真实中间件、内存业务 API 和隔离回收。"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import socket
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from langchain_core.language_models.fake_chat_models import FakeMessagesListChatModel
|
||||
from langchain_core.messages import AIMessage
|
||||
from langchain_core.messages import AIMessage, BaseMessage, ToolMessage
|
||||
from langchain_core.outputs import ChatGeneration, ChatResult
|
||||
from pydantic import Field
|
||||
from sqlalchemy import create_engine, select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
@@ -42,6 +44,95 @@ class _ScriptModel(FakeMessagesListChatModel):
|
||||
return await super()._agenerate(messages, stop=stop, run_manager=run_manager, **kwargs)
|
||||
|
||||
|
||||
class _TerminalSharingModel(FakeMessagesListChatModel):
|
||||
"""按真实父/子图消息边界完成一次终端共享回放,不替换生产工具执行。"""
|
||||
|
||||
bound_tools: list[list[Any]] = Field(default_factory=list)
|
||||
|
||||
def bind_tools(self, tools: list[Any], **_kwargs: Any) -> Any:
|
||||
"""记录主图和子图各自看到的真实工具目录。"""
|
||||
self.bound_tools.append(list(tools))
|
||||
return self
|
||||
|
||||
@staticmethod
|
||||
def _text(message: BaseMessage) -> str:
|
||||
"""把模型消息内容压成可匹配的字符串,兼容多模态内容块。"""
|
||||
return message.content if isinstance(message.content, str) else json.dumps(message.content, ensure_ascii=False)
|
||||
|
||||
@classmethod
|
||||
def _session_id(cls, messages: list[BaseMessage]) -> str:
|
||||
"""从父 start 回执或子代理授权上下文提取真实会话句柄。"""
|
||||
for message in messages:
|
||||
match = re.search(r'"session_id"\s*:\s*"([^"]+)"', cls._text(message))
|
||||
if match:
|
||||
return match.group(1)
|
||||
raise AssertionError("测试模型没有观察到终端 session_id")
|
||||
|
||||
@classmethod
|
||||
def _cursor(cls, messages: list[BaseMessage]) -> tuple[int, int]:
|
||||
"""从最近终端回执读取增量游标,避免 wait 反复返回同一已读输出。"""
|
||||
seq, offset = 0, 0
|
||||
for message in messages:
|
||||
text = cls._text(message)
|
||||
seq_match = re.search(r'"output_until_seq"\s*:\s*(\d+)', text)
|
||||
offset_match = re.search(r'"output_until_offset"\s*:\s*(\d+)', text)
|
||||
if seq_match:
|
||||
seq = int(seq_match.group(1))
|
||||
if offset_match:
|
||||
offset = int(offset_match.group(1))
|
||||
return seq, offset
|
||||
|
||||
@classmethod
|
||||
def _generate(cls, messages: list[BaseMessage], stop: Any = None, run_manager: Any = None, **_kwargs: Any) -> ChatResult:
|
||||
"""根据当前图是否为静默子代理,生成下一次真实工具调用或最终 JSON。"""
|
||||
del stop, run_manager
|
||||
system_text = " ".join(cls._text(message) for message in messages if message.type == "system")
|
||||
latest = messages[-1] if messages else None
|
||||
delegated = any(
|
||||
isinstance(message, AIMessage)
|
||||
and any(call.get("name") == "task" for call in (message.tool_calls or []))
|
||||
for message in messages
|
||||
)
|
||||
if "silent subagent" in system_text.lower():
|
||||
if any(isinstance(message, ToolMessage) for message in messages):
|
||||
response = AIMessage(content="子代理已从父终端真实读取回执。")
|
||||
else:
|
||||
response = AIMessage(content="", tool_calls=[{
|
||||
"id": "child-read", "name": "execute_command",
|
||||
"args": {"action": "read", "session_id": cls._session_id(messages)},
|
||||
}])
|
||||
elif not any(isinstance(message, ToolMessage) for message in messages):
|
||||
response = AIMessage(content="", tool_calls=[{
|
||||
"id": "parent-start", "name": "execute_command",
|
||||
"args": {"action": "start", "command": "printf 'SHARED_READY\\n'; sleep 1; printf 'SHARED_DONE\\n'",
|
||||
"use_pty": False},
|
||||
}])
|
||||
elif isinstance(latest, ToolMessage) and not delegated and '"session_id"' in cls._text(latest) \
|
||||
and "SHARED_READY" in cls._text(latest):
|
||||
session_id = cls._session_id(messages)
|
||||
response = AIMessage(content="", tool_calls=[{
|
||||
"id": "delegate-read", "name": "task",
|
||||
"args": {"description": "只读取父终端的真实输出", "terminal_sessions": [
|
||||
{"session_id": session_id, "actions": ["read"]},
|
||||
]},
|
||||
}])
|
||||
elif isinstance(latest, ToolMessage) and delegated and "SHARED_DONE" in cls._text(latest):
|
||||
response = AIMessage(content=json.dumps({
|
||||
"status": "completed", "terminal_output": "SHARED_READY\nSHARED_DONE\n",
|
||||
"terminal_exit_code": 0, "completed": ["terminal"], "unresolved": [],
|
||||
"subscription_ids": [], "download_ids": [], "enabled_site_ids": [],
|
||||
}, ensure_ascii=False))
|
||||
else:
|
||||
session_id = cls._session_id(messages)
|
||||
since_seq, since_offset = cls._cursor(messages)
|
||||
response = AIMessage(content="", tool_calls=[{
|
||||
"id": "parent-wait", "name": "execute_command",
|
||||
"args": {"action": "wait", "session_id": session_id, "timeout_ms": 5000,
|
||||
"since_seq": since_seq, "since_offset": since_offset},
|
||||
}])
|
||||
return ChatResult(generations=[ChatGeneration(message=response)])
|
||||
|
||||
|
||||
def _call(name: str, arguments: dict[str, Any], identifier: str) -> AIMessage:
|
||||
"""构造标准模型工具调用,交给真实 ToolNode 执行。"""
|
||||
return AIMessage(content="", tool_calls=[{"id": identifier, "name": name, "args": arguments}])
|
||||
@@ -340,6 +431,37 @@ async def test_evaluation_terminal_tool_returns_correction_for_wrong_stdin(tmp_p
|
||||
assert await close_terminal_scope(scope)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_production_agent_shares_parent_terminal_with_read_only_subagent(invocation_store):
|
||||
"""真实生产子图只在显式 read grant 下读取父终端,并由父任务完成收尾。"""
|
||||
world = EvaluationWorld("subagent_terminal_share")
|
||||
model = _TerminalSharingModel(responses=[])
|
||||
repository, _factory = invocation_store
|
||||
capture = await run_moviepilot(
|
||||
world,
|
||||
model,
|
||||
model_name="scripted-terminal-sharing-test",
|
||||
context_window=128000,
|
||||
max_iterations=32,
|
||||
invocation_repository=repository,
|
||||
)
|
||||
assert capture["execution_success"] is True, capture["final_text"]
|
||||
grade = evaluate(world, json.loads(capture["final_text"]), capture["raw_messages"])
|
||||
assert grade.passed, grade.violations
|
||||
assert "execute_command" in capture["child_tool_names"]
|
||||
scoped_events = [event for event in world.ledger if event.get("scope")]
|
||||
assert [event["scope"]["kind"] for event in scoped_events] == ["conversation", "subagent", "conversation"]
|
||||
assert scoped_events[0]["scope"]["task_id"] == scoped_events[2]["scope"]["task_id"]
|
||||
assert scoped_events[0]["scope"]["task_id"] != scoped_events[1]["scope"]["task_id"]
|
||||
assert {event["observations"][0]["record"]["session_id"] for event in scoped_events} == {
|
||||
scoped_events[0]["observations"][0]["record"]["session_id"]
|
||||
}
|
||||
assert all(
|
||||
event["request"].get("action") == "read"
|
||||
for event in scoped_events if event["scope"]["kind"] == "subagent"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_runtime_refuses_mismatched_config(monkeypatch, tmp_path):
|
||||
"""导入后更换 CONFIG_DIR 不能被误认成已隔离后端。"""
|
||||
|
||||
@@ -35,6 +35,12 @@ def _report(world):
|
||||
"enabled_site_ids": [row["id"] for row in state["sites"] if row["enabled"]],
|
||||
"completed": ["sites"], "unresolved": [],
|
||||
}
|
||||
if world.scenario.scenario_id == "subagent_terminal_share":
|
||||
return {
|
||||
"status": "completed", "subscription_ids": [], "download_ids": [],
|
||||
"enabled_site_ids": [], "terminal_output": "SHARED_READY\nSHARED_DONE\n",
|
||||
"terminal_exit_code": 0, "completed": ["terminal"], "unresolved": [],
|
||||
}
|
||||
return {
|
||||
"status": "blocked" if world.scenario.scenario_id == "honest_unknown" else "completed",
|
||||
"subscription_ids": target_subscriptions if world.scenario.scenario_id in {"dedup_existing", "long_context", "steering_long_context"} else [],
|
||||
@@ -125,6 +131,38 @@ def test_subagent_cancel_recovery_requires_start_and_cancel_actions():
|
||||
assert "subagent_cancel_not_verified" in evaluate(world, _report(world), without_cancel).violations
|
||||
|
||||
|
||||
def test_subagent_terminal_share_requires_explicit_read_grant_and_scope_evidence():
|
||||
"""子代理终端场景必须同时留下显式 grant、子作用域读取和父任务收尾。"""
|
||||
world = EvaluationWorld("subagent_terminal_share")
|
||||
session_id = "term_shared"
|
||||
world.record_command(world.scenario.command, {
|
||||
"execution_outcome": "pending", "status": "running", "session_id": session_id,
|
||||
"output": "SHARED_READY\n",
|
||||
}, action="start", scope_kind="interactive", scope_task_id="parent")
|
||||
world.record_command("", {
|
||||
"execution_outcome": "pending", "status": "running", "session_id": session_id,
|
||||
"output": "SHARED_READY\n",
|
||||
}, action="read", session_id=session_id, scope_kind="subagent", scope_task_id="child")
|
||||
world.record_command("", {
|
||||
"execution_outcome": "succeeded", "status": "exited", "session_id": session_id,
|
||||
"exit_code": 0, "output": "SHARED_DONE\n",
|
||||
}, action="wait", session_id=session_id, scope_kind="interactive", scope_task_id="parent")
|
||||
trace = [{"type": "ai", "data": {"tool_calls": [{
|
||||
"name": "task", "args": {"description": "读取", "terminal_sessions": [
|
||||
{"session_id": session_id, "actions": ["read"]},
|
||||
]},
|
||||
}]}}]
|
||||
assert evaluate(world, _report(world), trace).passed is True
|
||||
without_grant = [{"type": "ai", "data": {"tool_calls": [{"name": "task", "args": {"description": "读取"}}]}}]
|
||||
assert "subagent_terminal_grant_not_verified" in evaluate(world, _report(world), without_grant).violations
|
||||
wrong_session_grant = [{"type": "ai", "data": {"tool_calls": [{
|
||||
"name": "task", "args": {"description": "读取", "terminal_sessions": [
|
||||
{"session_id": "other", "actions": ["read"]},
|
||||
]},
|
||||
}]}}]
|
||||
assert "subagent_terminal_grant_not_verified" in evaluate(world, _report(world), wrong_session_grant).violations
|
||||
|
||||
|
||||
def test_correct_ids_without_any_observation_are_not_verification():
|
||||
"""即使报告恰好猜中初态中的 ID,没有读取事实仍必须失败。"""
|
||||
world = EvaluationWorld("dedup_existing")
|
||||
|
||||
@@ -30,7 +30,7 @@ def _download_body(world: EvaluationWorld) -> dict[str, Any]:
|
||||
def test_scenarios_expose_inputs_without_initial_state_or_oracle() -> None:
|
||||
"""公开定义只有任务、业务身份及资源,答案和故障时序不能进入模型上下文。"""
|
||||
scenarios = list_scenarios()
|
||||
assert len(scenarios) == 11
|
||||
assert len(scenarios) == 12
|
||||
api_scenarios = [scenario for scenario in scenarios if scenario.kind == "api"]
|
||||
assert len(api_scenarios) == 7
|
||||
assert len({scenario.media_id for scenario in api_scenarios}) == 6
|
||||
|
||||
Reference in New Issue
Block a user