Browser Playback Debug and Replay Plan

Status: implementation plan
Priority: debug and replay first; do not change property-management flow behavior until the next call can prove what browser audio played.

Problem

The caller heard the agent audio play and immediately say the same thing again. The browser-session debugger can reconstruct FSM visits, but it cannot prove a browser double-play because the browser path currently retains neither the played agent PCM nor durable playback lifecycle events. A repeated state and a repeated audible prompt are different failures.

Objective

For every browser call, the retained artifact and debugger must answer:

1. Which agent audio was requested, generated, started, stopped, and completed?
2. Which state visit owned each playback span?
3. Was a prompt played twice, and was the duplicate static, model-generated, re-entry, nudge, or client replay?
4. At what exact recorded timestamp did the first duplicate begin?
5. Can an operator pause, rewind, and fork simulation immediately before that event?

Current Reality

The repository does not yet satisfy the full objective:

Implementation must follow the causal dependency order: capture truthful playback evidence, normalize the shared clock, deliver recorded media replay, export restorable checkpoints, then enable S1 state simulation. A polished fork button before those layers exist would be a mock, not a debugger.

Playback Ledger Contract

Add canonical durable events, emitted at the browser playback boundary rather than inferred from FSM intent:

source_kind is one of static_asset, model_audio, entry_line, nudge, or unknown. content_fingerprint is a session-safe fingerprint of the resolved text/audio asset; it supports duplicate detection without retaining prohibited raw content.

The browser emits acknowledgement events after it actually starts/stops playback. The server writes the immutable ledger. Client-only console logs are not evidence.

Audio Evidence Contract

1. Attempt to capture browser-played agent PCM into audio_agent_played.wav with a declared sample rate and channel map.
2. If browser audio cannot be captured, retain the playback spans above and record AGENT_PLAYED_AUDIO_UNAVAILABLE with the technical reason.
3. Never create a 44-byte empty WAV and label it recorded. The manifest must say unavailable or empty_capture.
4. The finalizer links each playback span to WAV offsets when audio exists.

Double-Play Detector

The post-call artifact derives DUPLICATE_AGENT_PLAYBACK when two completed or overlapping playback spans have the same content fingerprint and satisfy a bounded gap policy.

Output fields:

{
  "code": "DUPLICATE_AGENT_PLAYBACK",
  "first_playback_id": "...",
  "duplicate_playback_id": "...",
  "first_visit_id": "menu#1",
  "duplicate_visit_id": "menu#1",
  "gap_ms": 82,
  "source_pair": ["static_asset", "model_audio"],
  "confidence": "recorded"
}

Do not label two deliberately different prompts as duplicates solely because they occur in the same state.

Replay UX

The debugger has one shared recorded clock.

• Play advances transcript bubbles, playback spans, waveform, slot writes, guard verdicts, and transitions together.
• Pause freezes every lane at one timestamp.
• Clicking a playback span seeks to its exact audio_started event.
• Step and Reverse move one canonical event, including playback lifecycle events.
• Rewind to duplicate seeks to the first audio_requested or audio_started in a duplicate pair and highlights the second span below it.
• The inspector shows current state visit, entry/current/exit slots, audio source, content fingerprint, client acknowledgement, and the event which requested playback.
• Recorded, unavailable, and simulated evidence use distinct labels.

Two synchronized cursors

The implementation must not confuse media time with executable state:

• The media cursor is a millisecond position on the retained call clock. It can seek to any audio sample or transcript segment.
• The event cursor is an index in the canonical debugger event stream. It advances only across recorded or simulated events.

Each audio artifact declares call_start_epoch, media_start_offset_ms, sample rate, channel map, and duration. Every transcript segment and FSM event carries clock_ms on the same call clock. The UI derives the event cursor by binary searching the last event whose clock_ms is at or before the media cursor. Seeking, pausing, or rewinding therefore moves audio, transcript, state, variables, and evidence together without inventing sub-event state.

The browser uses one playback controller for the retained audio rather than one independent player per state. Play advances the media cursor; animation-frame updates project the event cursor. Step and Reverse pause playback, select the next or previous event, and seek audio to that event's media offset. Audio is never synthesized during recorded replay.

Simulation Boundary

Playback replay is immutable. The operator may place the visual cursor at any timestamp, but deterministic execution can fork only from a restorable causal checkpoint. Version 1 checkpoints are:

1. state-visit entry, before entry actions;
2. immediately before a caller-input/DTMF selection;
3. immediately before a tool invocation whose result has a fixture;
4. immediately before an audio outcome (completed, interrupted, or failed) is consumed by the FSM.

If the operator clicks Fork halfway through audio, the UI preserves that timestamp as the issue marker but restores execution from the nearest preceding checkpoint. It must display both Issue at 42.180 s and Branch executes from menu#2 entry at 40.933 s. Claiming arbitrary instruction-level restoration would require serializing the live model connection, transport buffers, audio decoder, pending callbacks, and Python stack, which this design does not do.

State test capsule

Every fork exports an immutable riff.state_test_capsule/v1 containing:

• recorded session, engine revision, flow revision, state definition, and checkpoint identity;
• entry slots plus safe fingerprints, counters, retry state, and prior visit path required by guards;
• selected caller-input evidence and alternative Gemini/Whisper/DTMF sources;
• bounded conversation/model context needed by that state;
• recorded tool results converted to read-only fixtures;
• audio policy, prompt/template inputs, voice/TTS configuration, RNG seed, and virtual clock;
• recorded expected exit, slot delta, playback requests, and invariant findings for comparison.

The capsule is the program input. The recorded timeline remains the oracle. Candidate execution runs in an isolated subprocess with network and side-effecting tools disabled.

Supported mutations

“Change audio” is split into causal operations rather than editing waveform pixels and pretending the FSM changed:

Changing agent output audio does not retroactively change the recorded call. It creates a purple simulated playback span. Newly generated audio is labelled simulated; it may be auditioned but never used as evidence of what the caller heard.

Branch continuation levels

• Level S1 — state outcome: re-run one state from its checkpoint through its exit. This is the first implementation target and is sufficient to test alternative caller answers, slot values, guards, and playback outcomes.
• Level S2 — bounded path: continue across subsequent deterministic states using recorded tool fixtures until a side effect, missing fixture, model boundary, or explicit stop condition.
• Level S3 — candidate conversation: run with a pinned model/TTS and virtual transport. This is a new simulation, not historical reproduction, and must display model, prompt, voice, and runtime revisions.

Reverse before the fork checkpoint discards the branch and restores the byte-identical recorded projection. Editing a branch never changes Entry or Recorded Exit columns.

Work Packets

PB1 — Browser Playback Event Capture

• Locate every browser audio start/stop path.
• Assign a playback ID before dispatch and carry state/visit identity through client acknowledgement.
• Persist lifecycle events through the session ledger.
• Test static, model, interrupted, and failed playback.

PB2 — Honest Audio Finalization

• Repair agent-played WAV capture or mark it unavailable with reason.
• Add manifest availability/channel-map validation.
• Test that an empty WAV cannot be reported as recorded audio.

PB3 — Artifact and Detector

• Hydrate playback events into state visits and the global timeline.
• Derive duplicate playback findings from fingerprints and span timing.
• Test same-visit static-plus-model duplicate, adjacent-visit replay, and non-duplicate distinct prompts.

PB4 — Time-Travel Debugger UI

• Render playback spans and transcript bubbles in the chronological timeline.
• Add play/pause/scrub/step/reverse and rewind-to-duplicate.
• Test that a selected duplicate span maps to its exact state visit and command cursor hash.

PB4A — Shared Clock and Media Controller

• Normalize audio, transcript, playback, and FSM timestamps onto clock_ms.
• Add one media controller with deterministic seek-to-event and event-from-media binary search.
• Prove play/pause/seek/reverse do not drift across transcript, waveform, and state projection.

PB4B — State Capsule and S1 Fork

• Export and validate riff.state_test_capsule/v1 at supported checkpoints.
• Implement isolated slot, caller answer/source, tool fixture, and playback outcome overlays.
• Re-run one state to exit and compare exit edge, variable delta, playback requests, transcript selection, and findings to the recorded visit.

PB5 — Property-Management Validation

• Reproduce the reported browser symptom with call recording enabled.
• Confirm whether the duplicate is static+model, client replay, re-entry, or another mechanism.
• Only then change the responsible flow/client/runtime behavior and add a regression fixture.

Acceptance Gates

1. Every browser playback attempt has a durable lifecycle result or explicit capture gap.
2. Agent WAV availability is truthful; empty headers are never successful recordings.
3. A controlled duplicate produces DUPLICATE_AGENT_PLAYBACK with both source refs and visits.
4. The replay UI can pause on each duplicate start and reverse to its request event.
5. CLI, API, and UI identify the same duplicate pair and determinism hash.
6. A property-management rerun either identifies the duplicate mechanism or explicitly states what evidence remains unavailable.
7. Seeking any retained audio timestamp projects the same transcript/state event after page reload; step-forward then reverse restores the original determinism hash and audio position.
8. Forking from a mid-audio issue marker visibly restores from the named prior checkpoint; it never claims to resume a live Python/model/audio stack.
9. An S1 variable or caller-answer mutation can change the simulated exit while recorded Entry, Recorded Exit, audio, transcript, and events remain byte-identical.
10. Simulated agent audio is playable but visually and structurally impossible to confuse with recorded caller-heard evidence.