Fixes the Claude Code Exec backend for issue #233 - #238
Open
Nuplum (45lab95) wants to merge 1 commit into
Open
Conversation
…ace support Register claude_code_exec as a full optimizer/target backend (issue microsoft#233). --backend claude_code_exec now defaults both roles to claude_code_exec so reflection sees the agent's complete session, and the SDK message stream is parsed into structured trace steps persisted as claude_trace_steps.txt and injected into the analyst prompt. - model/claude_code_backend.py (new): chat_optimizer/chat_optimizer_messages on run_claude_code_chat, reasoning_effort threaded through, retry loop that surfaces non-JSON structured replies as RuntimeError, token tracking. - model/codex_harness.py: parse/format/persist claude trace steps (text, tool_call, tool_result; drops init/thinking_tokens; 200-char tool_result cap; total truncation) + effort override on run_claude_code_chat. - trainer.py/reflect.py: inject Claude Trace Steps gated behind REFLACT_CLAUDE_TRACE_TO_OPTIMIZER, set by the trainer only for claude_code_exec targets with model.claude_trace_to_optimizer (mirrors codex gate; default true). - config.py/default.yaml/docs: model.claude_trace_to_optimizer key + flatten mapping + config.md rows. - backend_config.py + model/__init__.py: register backend, route chat dispatch, token summary, reasoning effort, deployments. - scripts/train.py, eval_only.py: symmetric default + accurate comments. - tests: tests/test_claude_code_backend.py (10 tests: parsing, dispatch, effort, retry, trainer/reflect gating); test_role_backend_resolution.py updated to the symmetric default. Verified: 58 unit tests pass; integration smoke on searchqa improved best-on-val 0.7500 -> 0.9375 with 80 claude_trace_steps.txt written; all output files valid UTF-8 (no GBK mojibake).
Nuplum (45lab95)
changed the title
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the Claude Code Exec backend for issue #233: with
--backend claude_code_exec, both optimizer and target roles nowdefault to Claude Code (symmetric default), and the agent's SDK session trace is parsed into structured
claude_trace_steps.txtfiles that are injected into the reflectionprompt — addressing context-length truncation and trajectory loss.
Implementation
backend_config.py,model/__init__.py,scripts/train.py,scripts/eval_only.pyregisterclaude_code_exec; both rolesdefault to it (a role pinned to a non-default value like
minimax_chatstill overrides).codex_harness.pyaddsparse/format/persist_claude_trace_steps: extractstext/tool_call/tool_resultsteps from the SDK messagestream, drops
init/thinking_tokensbookkeeping, capstool_resultat 200 chars, and persists one file per prediction.reflect.pyinjects#### Claude Trace Stepsonly whenREFLACT_CLAUDE_TRACE_TO_OPTIMIZER=1, whichtrainer.pysets only forclaude_code_exectargets with
model.claude_trace_to_optimizer: true(default; mirrors the existing codex gate).REASONING_EFFORTmodule global is now consumed viarun_claude_code_chat(..., effort=...)and forwarded by thedispatcher.
tests/test_claude_code_backend.py;configs/_base_/default.yaml,docs/reference/config.md, and_FLATTEN_MAPextended.Results
tests/test_claude_code_backend.py+tests/test_role_backend_resolution.py→ 58 passed (11 new cases all green).searchqarun): exit 0,accept=1, best-on-val 0.7500 → 0.9375, 80claude_trace_steps.txtwritten.merged_patch.json/config.jsonvalid; skillv0000→v0001.How to verify
PYTHONUTF8=1 uv run --extra dev pytest tests/test_claude_code_backend.py tests/test_role_backend_resolution.py -q # → 58 passedRun a small
searchqatraining with--backend claude_code_exec --optimizer_backend claude_code_exec, then check:ValueError: Unsupported optimizer backend.claude_trace_steps.txt.#### Claude Trace Steps.PYTHONUTF8=1on Windows).