fix(voice): discard the uncommitted recognition turn on false-interruption resume - #7101
fix(voice): discard the uncommitted recognition turn on false-interruption resume#7101dorukdumlu wants to merge 4 commits into
Conversation
…ption resume A confirmed false interruption resumed agent state and audio without clearing the recognition turn the false barge-in opened. The user_turn span stayed recording, so the next real utterance found it via _ensure_user_turn_span and inherited the abandoned turn''s _user_turn_start: its committed ChatMessage carried a started_speaking_at seconds before the utterance, ordered ahead of agent speech that actually preceded it. The livekit#6093 ordering guard cannot catch this because the stale anchors stay internally consistent. _on_false_interruption now calls _clear_user_turn before restoring agent state and resuming audio, unless the user is speaking at that moment; live anchors belong to the in-flight utterance, not the discarded turn. Fixes livekit#7063
…tion turn _clear_user_turn tears down and recreates the STT pipeline, which is right for the manual clear_user_turn API but wrong on every false-interruption resume: it reconnects the provider stream each time, and it destroys audio the provider is still decoding. If the barge-in was real after all (a VAD miss), that late final is the only thing that can still interrupt the resumed speech, so it must survive the discard. The resume path now calls _clear_user_turn(reset_stt=False): span, anchors, transcripts and turn bookkeeping are dropped, the STT stream is left alone. The regression test runs the real clear and pins all of it: clear before resume, span ended, anchor gone, pipeline untouched.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
A premature STT END_OF_SPEECH sets _speaking to False while VAD is still mid-segment; the flushed VAD is expected to correct it with a new SOS. If the false-interruption timer settled in that window, the discard wiped the transcript and timing of an utterance still in progress. The guard now also requires _vad_speech_started to be False.
|
Addressed Devin's yellow: it was real. A premature STT END_OF_SPEECH sets |
| if ( | ||
| self._audio_recognition is not None | ||
| and not self._audio_recognition._speaking | ||
| and not self._audio_recognition._vad_speech_started |
There was a problem hiding this comment.
🟡 Correct STT endings preserve dead turns
When an empty interruption ends in STT before VAD, _vad_speech_started remains true after flushing and suppresses cleanup. The next utterance inherits stale transcript and timing.
Prompt for agents
The false-interruption guard in AgentActivity._start_false_interruption_timer treats AudioRecognition._vad_speech_started as proof of current speech. In STT turn-detection mode, AudioRecognition._on_stt_event flushes an active VAD stream on END_OF_SPEECH but does not clear _vad_speech_started. A flush is a hard boundary and emits no END_OF_SPEECH event. If STT was correct and no new speech follows, the flag remains true when the EOU task settles, so _clear_user_turn is skipped for a dead empty interruption. Track the post-flush VAD segment separately or wait for a fresh VAD START_OF_SPEECH before treating the flag as live; add a test that drives STT END_OF_SPEECH with an active VAD segment, no subsequent VAD SOS, then lets the false-interruption callback run and verifies cleanup.
Was this helpful? React with 👍 or 👎 to provide feedback.
The premature-STT-EOS flush ends the VAD segment without an END_OF_SPEECH (silero's flush only resets state), so _vad_speech_started stays True with nothing left to clear it. When STT was right and no speech follows, the previous guard suppressed the discard forever and the stale-anchor bug returned on that path. The flush now marks the segment abandoned (_vad_speech_flushed) and only a fresh VAD START_OF_SPEECH marks it live again; the discard guard counts a VAD segment as speech only while it is unflushed.
|
The second yellow is the mirror of the first and also real: silero's flush only resets state and emits no END_OF_SPEECH, so after a correct STT ending |
1 participant
Summary
A confirmed false interruption resumed agent state and audio without discarding the recognition turn the false barge-in opened.
_ensure_user_turn_spanreturns a recording span as-is and_user_turn_startis only written when it is None, so the next real utterance silently reused both: its committed ChatMessage carried astarted_speaking_atseconds before the utterance was spoken (12.1s in the reported production timeline), ordering the user line ahead of agent speech that actually preceded it. The #6093 guard cannot catch this because the stale anchors stay internally consistent; they are just anchored to a dead turn.What changed
_on_false_interruptioncalls_clear_user_turn(reset_stt=False)before restoring agent state,_on_start_of_agent_speech, andaudio_output.resume(): the span is ended, and the timing anchors, buffered transcript, and turn bookkeeping are dropped._interrupt_by_audio_activity.reset_stt=Falsekeeps the live STT stream. Recreating the provider stream on every false interruption is costly, and it drops audio the provider is still decoding: if the barge-in was real after all (a VAD miss), that late final is the only thing left that can interrupt the resumed speech. The manualclear_user_turn()API keeps its full-teardown behavior (defaultreset_stt=True).Behavior note: a final transcript arriving after the resume now commits as a fresh turn without
started_speaking_at, since the anchors are gone. Unknown is more honest than an anchor the system already declared false, matching what #6093 does for inconsistent anchors.Test plan
pytest tests/test_false_interruption_resume.py(11 passed)_clear_user_turn, and asserts the span is ended, the anchor is gone, and the STT pipeline is untouchedFixes #7063