fix(voice): answer every tool call - #6785
Open
longcw wants to merge 9 commits into
Open
Conversation
A realtime model keeps a tool call open server-side. Gemini Live blocks the turn until every call it emitted is answered and offers no way to cancel one, so discarding a result the handler already returned strands the session: it stops responding and subsequent generate_reply() calls produce no generation. Mirror the pipeline path on the realtime path — commit the outputs of tools that finished despite the interruption, handoffs excluded so they stay retryable — and additionally sync them to the realtime session. Gemini answers a delivered result with speech, which is unwanted right after an interruption, so the response is sent with SILENT scheduling where the API honours it (NON_BLOCKING declarations on the Gemini API). Also stop setting tool_response_scheduling on Vertex AI, which does not support the field, and warn when it is configured there instead of dropping it silently.
create_function_response already skips scheduling for vertex, so the call site does not need to repeat the rule.
…put-on-interrupt # Conflicts: # tests/test_plugin_google_realtime.py
A realtime model keeps each tool call it emitted open until the result arrives, so a result dropped because the user interrupted leaves the session waiting: Gemini stops responding and later generate_reply() calls produce no generation. The interrupted turn now answers every call that produced a result, handoffs included. A handoff answers as an error, since an interruption leaves it unapplied and an empty success would claim a transfer that never happened. Each output carries reply_required, so a session that can record a result without speaking does, and one that cannot says so once rather than failing quietly. That also replaces the blanket warning about models which generate tool replies on their own.
A call with no output is not a neutral omission. An LLM re-issues a call it never got back and runs its side effects again, and a realtime model holds the turn open waiting for it, which is how a StopResponse could strand a Gemini session with no interruption involved. Tools that produced nothing now answer anyway: StopResponse with an empty output that wants no reply, an invalid output or more than one returned agent with an error the model can act on. FunctionCallOutput carries reply_required, so ToolExecutionOutput no longer needs its own copy, and function_call_outputs on FunctionToolsExecutedEvent is no longer optional.
The tool ran before the interruption, so dropping its call lets the next inference run it again. Both paths now record the call with an error saying the handoff did not happen, which keeps the transfer askable without hiding that it was attempted. The shared construction moves to generation.py, next to make_tool_output and ToolExecutionOutput.
Warning once per session hid later ones, and neither warning said which tool it was about. Both now warn per update with the names in `extra`. `get_tool_results_for_realtime` takes `supports_silent_scheduling`, naming the capability the caller reports rather than an instruction.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: A tool call could end with no output at all — dropped when the user interrupted, or never produced when a tool raised
StopResponseor returned an invalid value. A realtime model holds each call it emitted open until it is answered, so Gemini Live stopped responding and latergenerate_reply()calls produced no generation (#6569). A pipeline LLM re-issues the call instead and runs its side effects a second time.Fix: Every call now gets exactly one output. An interrupted turn commits the results of tools that finished and delivers them to the realtime session; a handoff answers as an error, since the interruption left it unapplied.
Behavior:
FunctionCallOutput.reply_requiredsays when no reply is wanted, and each realtime plugin honours it as its provider allows — Gemini withSILENTscheduling, Ultravox withagent_reaction="listens", Phonic by not opening a turn — or warns that it cannot. That replaces the blanket warning about models which generate tool replies on their own.Breaking:
FunctionToolsExecutedEvent.function_call_outputsislist[FunctionCallOutput], no longer optional. A tool raisingStopResponsenow leaves a call and an empty output in history where it previously left nothing.Replaces #6600. Fixes #6569