Live Receptionist Supervisor Join — Detailed Design

Status: BUILDING v1 — P0/P1, idle-only PTT, and Telnyx queue-boundary evidence implemented locally; L2/L3/OTA pending

Owner: RIFF phone service / M3 lane

Date: 2026-07-14 Parent: Live Receptionist Console MVP

0. Decision

An authenticated operator can open the live receptionist console on a computer, listen to one active phone call, activate a talk control to speak to the caller, optionally take over the call, and explicitly give control back to the AI.

The first production implementation uses RIFF's existing bidirectional Telnyx media stream as the browser audio bridge:

                                                listen: caller + agent PCM
Browser microphone ◀──▶ /ws/supervisor ◀──▶ per-call SupervisorMediaHub
                                                    │
                                                    ▼
Caller ◀──▶ Telnyx ◀──▶ /ws/media ◀──▶ TelnyxTransport ◀──▶ Gemini Live
                                      human PCM ────▶│
                                      AI input/output gates

This is a real computer join, but the browser is not initially a separate carrier participant. RIFF relays audio through the phone call's existing stream. That choice preserves the current call topology and avoids assuming that a second Telnyx WebRTC/conference leg is compatible with a call already using bidirectional media streaming. Native Telnyx supervision remains a provider option after the carrier spike in §15 proves it.

The existing companion-control tap ladder is not the audio-control protocol. It supplies useful precedents—per-call routing, one deterministic owner, dedupe, and capability checks—but audio needs an exclusive expiring lease, media epochs, backpressure, and explicit AI/FSM quiescence.

1. Product behavior

The console exposes four progressively more powerful modes:

The product language must be literal:

• Listen never activates the microphone.
• Start talking is a click toggle backed by an expiring server lease; it never depends on a pointer remaining held.
• The UI distinguishes microphone capture, local WebSocket-send acceptance, Telnyx queue-boundary acknowledgement, and handset audibility. It never claims “caller can hear you” without independent caller-side evidence.
• Take over requires a confirmation and stays visibly latched.
• Give back to AI is an explicit, acknowledged transition, not a cosmetic button state.
• End call remains separate and requires its own confirmation.

Multiple authorized operators may listen. Exactly one operator may own the talk/takeover lease for a call. A second operator sees who owns it and cannot race the first operator.

Private whispering to another human agent is not in v1: the active call has one caller, one AI, and at most one speaking browser operator.

2. Scope and explicit cuts

In scope

• Same-origin browser audio on the designated phone node.
• Per-operator listen/talk/takeover capabilities.
• Multiple listen-only sessions and one exclusive speaker lease.
• Caller and delivered-agent audio mirrored to the browser without blocking the call path.
• Browser microphone audio injected into the existing Telnyx media stream.
• Deterministic AI input, AI output, tool, and static-playback gates while a human owns speech.
• Auditable state changes and a separate operator recording track.
• Safe disconnect behavior, explicit handback, and local/stub/replay test coverage.
• A media-provider interface for a later native Telnyx WebRTC participant.

Not in the first implementation slice

• Multi-node active-call discovery or cross-node media proxying.
• More than one speaking operator.
• Warm transfer to an arbitrary external telephone number.
• Browser screen sharing, video, or team chat.
• A guarantee that a carrier-level conference command works beside the existing media stream.
• Automatic language-model summaries as the authority for handback.
• Real external OTA validation without the human's literal GO.

3. Grounded constraints

The implementation must preserve these current facts:

1. riff/phone/server.py owns /ws/media and keeps active transports in-process. Console state is process-local, so the browser and call must reach the same designated node.
2. riff/phone/telnyx_transport.py::handle_media_sync decodes caller audio to mono PCM16 at 16 kHz, then may gate it during prerecorded playback before forwarding it to Gemini.
3. All AI, static, droid, and fallback speech converges at the Telnyx outbound send boundary, but static playback threads can independently advance the FSM when playback completes.
4. ws_clear requests removal of queued Telnyx output. A correlated Telnyx mark returned after the clear provides a queue boundary; neither local send return nor a returned mark proves that a handset or human heard audio. The Gemini wrapper exposes audio/text send and close, but no dependable mid-turn cancel acknowledgement.
5. riff/console_fanout.py already proves the nonblocking bounded-fanout pattern for JSON. Audio needs a separate high-rate queue, never the transcript queue.
6. web/live/audio-recorder.js already captures mono PCM16 at 16 kHz in 20 ms frames with echo cancellation, noise suppression, and automatic gain control. audio-player.js can be reused after its sample rate is made configurable.
7. riff/operator_auth.py currently identifies an operator but has no capabilities. The shared administrator URL token is not sufficient authority to put a voice onto a live call.
8. The law DID is not guaranteed to route to the same flow on M1 and M3. Single-node pinning is a correctness requirement, not merely a deployment preference.

4. Options considered

The chosen bridge is deliberately behind SupervisorMediaProvider. Product state, auth, leases, recording, UI, and audit must not depend on whether media is RIFF-relayed or carrier-native.

5. Component architecture

receptionist.html / receptionist.js
  ├─ existing /ws/console                 transcript + state
  ├─ POST /api/console/calls/:id/supervisor-sessions
  └─ /ws/supervisor                       control JSON + binary PCM
             │
             ▼
SupervisorSessionRegistry
  ├─ authenticates session/operator/capabilities/origin
  ├─ binds conv_id and owning node
  ├─ owns listen sessions
  └─ delegates call control to SupervisorController
             │
             ├─ SupervisorMediaHub         bounded listener fanout
             ├─ SpeakerLease               one owner, generation, TTL
             └─ SupervisorMediaProvider
                    ├─ RiffMediaBridge      v1
                    └─ TelnyxSupervisor     later, after §15 spike
                                      │
                                      ▼
TelnyxTransport
  ├─ caller mirror tap                    before AI playback gate
  ├─ agent send-boundary mirror tap       after local WebSocket acceptance
  ├─ operator injection queue             bounded + epoch checked
  ├─ ModelControlGate                     input/output/event quarantine
  └─ static playback freeze/resume         no hidden FSM advance

The media hub and controller are per conv_id. The global registry only locates them and must not hold a global lock while encoding, sending, waiting, or touching the model.

The provider contract is intentionally small:

capabilities() -> listen / inject / native_role_switch
open_session(call_binding, operator_binding) -> media_connection_plan
enter_human_control(control_revision, media_generation)
send_operator_frame(validated_frame)
leave_human_control(control_revision, handback_context)
close_session(reason)

media_connection_plan tells the browser whether audio uses RIFF's supervisor WebSocket or a short-lived carrier WebRTC token. It never changes the product-level control state machine.

6. Hard invariants

These are ship gates, not implementation suggestions:

1. A slow, disconnected, or malicious browser cannot delay caller audio, AI audio, FSM work, or a Telnyx send.
2. Audio and control frames are bound to exactly one conv_id; no global “current call” exists.
3. Many listeners are allowed; only one live speaker lease is allowed.
4. The server admits no microphone frame until it has acknowledged the matching control revision, media generation, and human-owned state.
5. AI audio, AI tool calls, and autonomous FSM transitions cannot reach the call while a human owns speech.
6. A static/droid playback thread cannot advance the FSM after human control has interrupted it.
7. Stale audio is never played. Releasing/revoking a lease increments the generation and flushes every older injection frame.
8. Named operator auth is required for listen/talk. The shared admin token remains read-only.
9. Call end, operator revocation, or session expiry closes media and invalidates every lease.
10. The operator track and every ownership transition are replayable and auditable.
11. A node mismatch is an explicit error. The system never presents a remote call as locally controllable.
12. Monitoring/talk features stay disabled until the business has configured the required caller notice/consent policy for its jurisdictions.

7. Control state machine

The call has one authoritative supervisor-control record:

AI_ACTIVE
   │ request push-to-talk
   ▼
PTT_ARMING ── gate/clear/mark/quiesce failed ──▶ AI_ACTIVE (error)
   │ ready_ack
   ▼
HUMAN_PTT ── release/disconnect ──▶ HANDING_BACK ── context accepted ──▶ AI_ACTIVE
   │ latch takeover
   ▼
HUMAN_TAKEOVER ── give back ─────▶ HANDING_BACK
   │ owner disconnect
   ▼
RECOVERING ── reconnect ─────────▶ HUMAN_TAKEOVER
   └─ grace expires ─────────────▶ HANDING_BACK

Any state ── call ended ─────────▶ ENDED

Each record contains:

conv_id
state
revision                    monotonically increasing control CAS
media_generation            increments on every ownership boundary
lease_id?                   opaque, never accepted across generations
lease_owner_operator_id?
lease_expires_at?
entered_at
reason
model_turn_status           idle | active | quarantining | reconnecting
parked_static_playback?     state/playback identity, never just a boolean

Every mutating request supplies expected_revision. A mismatch returns the current snapshot and does nothing. This prevents double clicks, reconnect replays, and two tabs from issuing conflicting commands.

Session state

Each browser session independently moves through:

CONNECTING → LISTENING → LEASED → TALKING → LISTENING → CLOSED

LEASED means the server has admitted the operator as speaker; TALKING means the push-to-talk control is down and current-generation mic frames are accepted. Browser UI state is derived from server acknowledgements, never optimistic button state.

8. Entering human control

Push-to-talk/takeover admission is a server transaction:

1. Authenticate the named operator, capability, call/node binding, and expected revision.
2. Acquire the exclusive speaker lease and increment media_generation.
3. Move to PTT_ARMING; immediately close the caller-to-Gemini input gate.
4. Close the AI-to-carrier output gate at the common outbound choke point.
5. Quarantine current Gemini model events: output, tool calls, and turn completion may update provider bookkeeping, but cannot mutate the FSM or emit caller-facing audio.
6. Interrupt and freeze any RIFF-owned static/droid playback. Record it as interrupted and prevent its thread's finally block from advancing the state.
7. Flush old operator frames, call ws_clear, and reset the outbound playback cursor.
8. Send a unique supervisor_clear_barrier mark and wait on the supervisor-control thread for its correlated return. The /ws/media receive loop performs only correlation and Event.set(); the waiting control thread publishes barrier evidence before ready_ptt. Caller media continues flowing while the control thread waits.
9. If the mark is absent, rejected, or late, time out and return to AI_ACTIVE without admitting microphone audio. The initial timeout is configurable with RIFF_SUPERVISOR_CARRIER_MARK_TIMEOUT_MS and defaults to 1000 ms.
10. Only after the barrier returns, move to HUMAN_PTT or HUMAN_TAKEOVER, send ready_ack with the new revision/generation/lease, and accept current-generation mic frames through the bounded operator-injection queue.

The first push-to-talk slice is enabled only when model_turn_status=idle and no RIFF-owned static audio is active. The later “Interrupt and talk” slice enables steps 5–7 against an active turn after the quarantine/reset tests pass. This is a staged rollout, not two competing designs.

9. Human media interval and handback

While a human owns speech:

• Caller PCM continues to the recorder, Whisper tap, level meter, and authorized browser listeners.
• Caller PCM does not enter Gemini or deterministic choice dispatch.
• Operator PCM goes only to the Telnyx outbound path, operator recorder track, and other authorized listeners. It never goes through Gemini.
• The first successfully submitted operator frame is followed by a nonblocking diagnostic mark. Its return means the preceding Telnyx queue boundary completed; it is not proof of handset rendering, human audibility, or speech recognition.
• Gemini output and tool calls remain quarantined.
• The FSM state/slots remain authoritative and frozen except for call-end/transport-failure facts.
• Caller and operator transcript segments are captured as evidence, but transcript guesses cannot directly mutate slots.

Handback is also an acknowledged transaction:

1. Stop accepting mic audio and increment media_generation.
2. Flush operator injection and send a nonblocking operator_burst_complete mark after the final locally submitted operator frame. Do not delay AI restoration on this diagnostic mark.
3. Finalize the human-interval caller/operator transcript and build a structured context envelope: authoritative FSM state and slots, human interval boundaries, and transcript clearly marked as observations—not committed facts.
4. Quiesce the old model turn. If it does not reach a safe boundary within the configured timeout, close and recreate the Gemini Live connection rather than allowing an old response to leak.
5. Inject the handback envelope through the model-control adapter.
6. Reopen model event processing, then model input, then model output in that order.
7. Emit one handback_complete acknowledgement and move to AI_ACTIVE.

No browser timer may reopen the AI gates. The server alone decides that handback is complete.

ASR finalization gets a bounded 1.5-second drain at handback. If either human-interval track has no usable final transcript, the envelope marks that segment unavailable; it does not invent text, block indefinitely, or commit guessed slots. Authoritative FSM state still permits a clean neutral resume such as asking the caller what they would like to do next.

For PTT owner disconnect, operator injection stops immediately and handback begins. For latched takeover, the server allows a three-second reconnect grace in RECOVERING; after that it performs the same failback transaction. The latched takeover button remains feature-flagged off until this automatic recovery passes L2 and L3 fault tests.

10. Audio taps, format, and backpressure

Caller mirror

Mirror normalized PCM16/16 kHz immediately after Telnyx decode and recorder timestamping, before the normal playback/barge-in gate. The operator must still hear the caller while the AI is speaking or while RIFF is intentionally suppressing caller input to Gemini.

Agent mirror

Mirror the carrier-normalized played PCM only after ws_send returns successfully. Do not mirror raw Gemini output: generated audio is not proof of what crossed the phone send boundary. Include the estimated carrier-play timestamp already tracked by the outbound playback cursor.

Operator injection

Browser mic frames are mono signed PCM16 little-endian, 16 kHz, exactly 20 ms (320 samples / 640 bytes). The server converts directly from 16 kHz to the negotiated Telnyx codec/rate under the same outbound audio lock used by AI/static audio. It records a separate operator track after validation and records the carrier-normalized form after local ws_send acceptance. That artifact proves the RIFF send boundary, not Telnyx queue completion or handset playback; correlated mark events provide the separate Telnyx queue-boundary fact.

Queue limits

• Listener queue: 50 frames per track (one second), drop oldest, publish a gap counter.
• Operator injection queue: 10 frames (200 ms), reject newest on full and surface congestion.
• Frame rate limit: 60 mic frames/second with a short burst allowance.
• Speaker heartbeat: every two seconds; lease TTL six seconds.
• One-time join secret TTL: 60 seconds.
• A frame more than 250 ms behind the server's current receive window is stale and rejected.

Media publishing is put_nowait only. Encoding and WebSocket writes happen in isolated sender workers. Audio never shares the console JSON fanout queue, so a transcript snapshot cannot create an audio latency spike.

Binary frame v1

All integers use network byte order. The 22-byte header is:

magic[4] = "RSM1"
kind u8                    1 caller, 2 agent send-boundary, 3 operator send-boundary, 16 mic
flags u8                   bit 0 discontinuity, bit 1 end-of-segment
media_generation u32
sequence u32
timestamp_ms u64           monotonic offset from call start
payload                    PCM16-LE, mono, 16 kHz; 640 bytes in v1

The server ignores unknown kinds and rejects malformed lengths before allocating/copying a payload.

11. Web/API contract

Create a supervisor session

POST /api/console/calls/{conv_id}/supervisor-sessions
Content-Type: application/json

{"requested_mode":"listen"}

Response:

{
  "supervisor_session_id": "opaque",
  "join_secret": "one-time opaque secret",
  "join_secret_expires_at": 0,
  "ws_path": "/ws/supervisor",
  "operator_id": "named operator",
  "allowed_modes": ["listen", "ptt"],
  "call_revision": 7,
  "media_generation": 2,
  "node_id": "designated-node"
}

The join secret is returned in the HTTPS body, never a URL/query string, and never logged. The browser opens /ws/supervisor with its normal secure operator cookie and sends a first JSON frame:

{"type":"join","session_id":"opaque","join_secret":"opaque"}

The secret is consumed once. The socket is rejected unless the secure operator session, operator identity, session row, exact configured Origin, conv_id, and node all match.

Control frames

Client frames:

claim_ptt(expected_revision)
ptt_start(expected_revision, lease_id, media_generation)
ptt_stop(expected_revision, lease_id, media_generation)
latch_takeover(expected_revision, lease_id, media_generation)
give_back(expected_revision, lease_id, media_generation)
heartbeat(lease_id, media_generation)
leave

Server frames:

snapshot
ready_ack
state_changed
lease_denied
audio_gap
congestion
carrier_media_status
carrier_media_error
handback_complete
error
call_ended

Every server control frame carries conv_id, revision, media_generation, and ts. Error frames use stable machine codes such as WRONG_NODE, CALL_ENDED, CAPABILITY_DENIED, LEASE_HELD, STALE_REVISION, STALE_GENERATION, MODEL_NOT_QUIESCENT, and ORIGIN_DENIED. Carrier admission adds CARRIER_CLEAR_FAILED and CARRIER_BARRIER_FAILED. A carrier_media_status frame carries evidence_stage=telnyx_queue and handset_verified=false; it must not be rendered as proof that the caller heard speech.

DELETE /api/console/supervisor-sessions/{session_id} closes a listen session. It does not silently release a takeover; speaker disconnect follows the recovery state machine.

12. Authentication, authorization, and audit

Extend operators with independent default-deny capabilities:

can_observe
can_listen
can_talk
can_takeover
can_end_call

Database migration preserves current operators as observe-only unless explicitly upgraded. The shared admin_token URL remains unable to listen or speak. A named operator session, exact trusted Origin, CSRF protection on POST/DELETE, secure cookie, live revocation, and periodic revalidation are required.

Provision or rotate a local operator token out of band; the command prints the raw token once and stores only its hash:

scripts/console-operator.py provision <operator-id> --listen --talk

Use grant, show, or revoke subcommands for later capability/session administration. Talk implies listen; neither command changes the default-off service feature flags.

Audit each attempted and completed mutation with:

event_id, conv_id, operator_id, supervisor_session_id,
from_state, to_state, expected_revision, committed_revision,
media_generation, result, reason_code, server_ts, node_id

Do not put transcript text, raw audio, caller phone, join secrets, session cookies, or lease secrets in the audit row. Audio/transcript evidence follows the call artifact access and retention policy.

The operator UI always shows microphone permission/state, speaker lease owner, recording status, node, call identity, and whether caller-notice policy has enabled the feature.

13. Recording and replay

Add audio_operator.wav alongside caller, generated-agent, and played-agent evidence. Preserve:

• Raw validated operator PCM at 16 kHz.
• Carrier-normalized operator PCM after local WebSocket-send acceptance.
• Supervisor state/lease transitions as timed events.
• Sanitized Telnyx stream errors and correlated mark send/acknowledgement/timeout events, each with an explicit evidence stage.
• Per-talk browser-offer, injection-queue, local-send, byte, timing, post-codec RMS/peak, and diagnostic-mark counters.
• Human interval transcript with role=operator|caller, confidence, engine, and finality.
• AI output/tool events that were quarantined, labelled not_delivered / not_applied.
• Static playback interruption and whether it was parked, replayed, or superseded at handback.

The normal stereo replay may mix locally submitted agent and operator audio on the right channel for human listening, but the separate operator track remains canonical for diagnosis.

Phone teardown writes a bounded, session-scoped bus_events.jsonl beside the canonical audio and manifest. Supervisor transitions and carrier-media events always include session_id; secrets, raw phone numbers, transcript text, and complete carrier control identifiers are excluded from these new event types.

The diagnostic active-frame level uses a fixed -45 dBFS post-codec frame threshold and reports both all-frame and active-frame RMS/peak. It is not a speech recognizer and does not alter, normalize, or gate operator audio.

Required metrics, partitioned by node_id and provider but not by caller PII:

supervisor_sessions{mode}
supervisor_claim_latency_ms
supervisor_listener_queue_depth / listener_frames_dropped_total
supervisor_mic_frames_accepted_total / rejected_total{reason}
supervisor_injection_queue_depth
supervisor_carrier_mark_latency_ms{purpose}
supervisor_carrier_mark_timeout_total{purpose,admission_required}
supervisor_carrier_stream_error_total{code}
supervisor_human_interval_seconds{mode}
supervisor_handback_latency_ms{result}
supervisor_quarantined_model_events_total{kind}
supervisor_control_leak_total{kind=audio|tool|fsm|static_advance}
supervisor_lease_expiry_total / recovery_total{result}

supervisor_control_leak_total must remain exactly zero. Initial L3 quality targets are less than 350 ms p95 from accepted browser mic frame to carrier-send return, less than 500 ms p95 listen delay, and less than 1% listener-frame loss on a healthy connection. The first measured candidate run may tighten these thresholds; it may not silently relax the zero-leak invariant.

14. Node affinity and deployment

Phase 2 inherits the console's single-node decision:

• The law DID is pinned to one accepting node before supervisor controls are enabled.
• The console page, control API, supervisor WebSocket, and /ws/media run same-origin on that node.
• GET /api/console/active includes node_id and a local control capability flag.
• A request for a nonlocal conv_id returns 409 WRONG_NODE; it never creates a detached session.
• /healthz reports the supervisor feature flags/provider and whether the node accepts live calls.
• Candidate deployment must prove SHA, clean/dirty state, configured route, and feature flags before the first call.

A future broker may proxy control/media to the owning node using an active-call directory with leases. That is a separate design; v1 does not fake active-active behavior.

Feature flags are independently default-off:

RIFF_SUPERVISOR_LISTEN_ENABLED
RIFF_SUPERVISOR_PTT_ENABLED
RIFF_SUPERVISOR_INTERRUPT_ENABLED
RIFF_SUPERVISOR_TAKEOVER_ENABLED
RIFF_SUPERVISOR_PROVIDER=riff_bridge|telnyx_supervisor
RIFF_SUPERVISOR_CARRIER_MARK_TIMEOUT_MS=1000

15. Native Telnyx provider spike

Telnyx documents browser JWT login, call-control dialing, conference join, and supervisor roles monitor, whisper, and barge. It also documents error 90045, where some commands cannot be issued while media streaming is used. The current RIFF AI call already uses bidirectional media streaming, so native supervision is a hypothesis until tested on a disposable route.

The spike must:

1. Mint a short-lived WebRTC JWT on the backend, with one telephony credential per operator.
2. Keep credentials and API keys out of browser storage and URLs.
3. Establish the current AI streaming call on a test DID.
4. Dial or join the browser leg using supervise_call_control_id/conference supervisor role.
5. Correlate the supervisor leg before the general incoming-call webhook path so it can never be mistaken for a new caller and answered by a second RIFF agent.
6. Prove monitor audio, switch to barge, speak both directions, switch back, detach, and preserve the original AI leg.
7. Capture call-control webhooks, command IDs, errors, PCAP/browser stats where allowed, and aligned caller/operator recordings.
8. Repeat while the AI bidirectional stream is active, not on an easier non-streaming surrogate.

Adopt TelnyxSupervisorProvider only if the spike passes without disrupting AI media, node ownership, recording, or handback. Otherwise retain RiffMediaBridge; product behavior does not change.

Relevant official documentation:

• Telnyx WebRTC JavaScript quickstart
• Telnyx WebRTC production practices
• Dial a call / supervised role fields
• Switch supervisor role
• Create a conference
• Join a conference
• Update a conference participant
• Programmable Voice media streaming — clear, mark, and error frames
• Telnyx API errors

Any external carrier spike is OTA work and requires the human's literal GO.

16. UI states

The live-call header adds one compact control group:

[ Listen ]  [ Start talking ]  [ Take over ]
Audio: connected · Mic: muted · AI: active · Owner: none

Behavior:

• Listen asks for audio playback permission only and starts muted until a user gesture.
• Start talking asks for microphone permission. One click requests the lease; the next click ends it. The control remains disabled until the clear-barrier mark returns and ready_ack arrives.
• While active: high-contrast LIVE — click to stop, call owner You, and AI Paused.
• Status progresses through literal evidence: Mic live, Telnyx queue ready, first talk frame crossed Telnyx queue, and handset unverified. The console never converts queue acknowledgement into “caller heard.”
• If another operator owns the lease: disabled control plus Alex is speaking.
• Take over opens a confirmation explaining disconnect recovery and explicit handback.
• During HANDING_BACK: all talk controls disabled and Restoring AI context… shown.
• A revision/generation mismatch forces a fresh snapshot before controls re-enable.
• Headphone guidance appears before first mic activation to reduce echo.
• Browser background/suspend is treated as loss of heartbeat, never as continued ownership.

Keyboard behavior and accessible names must make push-to-talk usable without a pointer. Key-up, window blur, visibility loss, and socket close all issue/trigger stop semantics; the server lease is still the final safety boundary.

17. Failure matrix

18. Test and evidence ladder

L0 — pure deterministic tests

• Every legal and illegal control-state transition.
• CAS revision races and single lease under concurrent claims.
• Lease expiry/generation rejection and one-time join-secret consumption.
• Binary packet parser, length/rate/timestamp limits, and fuzz corpus.
• Capability matrix and observe-only migration.
• Static playback freeze token prevents late completion advance.
• Handback envelope contains authoritative slots and labels ASR as observation.
• Telnyx mark correlation, unknown/late marks, timeout replacement, and stream-error fanout.

L1 — in-process fake media/model/carrier

• Caller mirror occurs before playback gate.
• Agent/operator mirror occurs only after local WebSocket-send acceptance.
• Stalled listeners and full queues cannot increase media-loop latency.
• AI audio, tools, and FSM transitions are quarantined for the whole human interval.
• PTT idle entry, release, disconnect, and stale-frame cases.
• Failed ws_clear never produces ready_ack.
• clear → barrier mark returned → ready_ack → operator media → first/final marks preserves order; missing marks and Telnyx error frames restore AI without admitting mic audio.
• Caller media mirroring remains live while the supervisor-control thread waits for the barrier.
• Carrier event fields are sanitized and session-scoped; phone teardown writes them with supervisor transitions to bus_events.jsonl.
• Two calls and two operators prove cross-call isolation.
• Recording contains aligned caller/agent/operator tracks and control events.

L2 — local browser + phone simulator

• Real microphone permission, 20 ms worklet frames, two-track playback, headphones.
• Hold/release, key-up, blur, tab suspension, refresh, network loss, and reconnect.
• One slow listener and one active speaker under synthetic packet loss/jitter.
• Transcript continues while AI input is gated and labels operator speech correctly.
• Browser automation asserts controls reflect server ACKs, never optimistic state.
• Browser automation asserts queue acknowledgement is labelled handset unverified and a mid-talk carrier error requests handback.

L3 — candidate node, synthetic/stub carrier where possible

• Pinned route/node/SHA/feature-flag preflight.
• Long call, operator churn, auth expiry/revocation, provider reconnect, service deploy guard.
• No call-path latency regression with listen fanout enabled but no listeners.
• Rollback by disabling each feature flag independently.

OTA — explicit human GO only

• One controlled law-line call: listen only.
• One controlled PTT call with AI idle.
• One interrupt-and-talk call after PTT evidence is clean.
• One takeover/disconnect/handback call after recovery evidence is clean.
• Native Telnyx provider spike is a separate GO.

Each run retains health snapshot, node/SHA, feature flags, console control trace, aligned WAV tracks, transcript engines, playback lifecycle, queue/gap counters, and the flow-evaluator report.

19. Implementation sequence

Implementation checkpoint (2026-07-14): the default-off control core, listen bridge, click-toggle browser UI, idle-only PTT path, Telnyx mark/error dispatch, fail-closed clear barrier, operator-send counters, and session event ledger are implemented with focused L0/L1 coverage. Interrupting active AI/static audio, gain changes, latched takeover, the native carrier experiment, L2/L3, and OTA proof remain gated.

P0 — contracts and default-off control core

• Add riff/supervisor_control.py: pure state machine, revisions, leases, generations.
• Add operator capabilities and migration with current users observe-only.
• Add API/session rows, one-time join secrets, audit events, exact-origin checks.
• Define SupervisorMediaProvider and a fake provider.
• Unit/concurrency/security tests. No audio path changes and all feature flags off.

P1 — listen-only bridge

• Add riff/supervisor_media.py: packet codec, per-call bounded hub, listener lifecycle.
• Tap caller PCM before the AI playback gate.
• Tap agent PCM after local WebSocket-send acceptance.
• Parameterize/reuse browser PCM player; add console Listen state.
• Prove nonblocking behavior and call isolation. Enable only listen in local/L2.

P2 — safe PTT while AI is idle

• Add bounded operator injection and separate recorder track.
• Add common AI input/output gates and provider event quarantine.
• Admit PTT only at model/static idle; implement release and disconnect handback.
• Require a correlated Telnyx mark after ws_clear, retain first/final diagnostic marks and stream errors, and keep all UI evidence labels below handset-audibility claims.
• Ship behind RIFF_SUPERVISOR_PTT_ENABLED=0 until L0–L3 pass.

P3 — interrupt and talk

• Integrate carrier clear, generated-audio interruption evidence, static freeze, model-turn quiescence, and context handback.
• Prove no old audio, tool, or transition leaks after claim.

P4 — latched takeover and recovery

• Add confirmation, reconnect grace, automatic failback, explicit give-back, and terminal call-end handling.
• Enable only after disconnect/recovery tests pass.

P5 — carrier provider experiment

• Run §15 under a separate human GO and implement TelnyxSupervisorProvider only if proven.

P6 — multi-node ownership

• Design and build an authenticated broker/active-call directory. This is not a prerequisite for a pinned single-node pilot.

Expected code ownership:

20. Definition of done

Listen-only is done when an authorized computer hears the caller and the audio actually sent to the caller, while a blocked browser has no measurable effect on the phone path.

Push-to-talk is done when one authorized operator can acquire a lease after a correlated Telnyx clear-barrier acknowledgement, submit speech, release/disconnect, and return control exactly once—with zero AI audio, tool, or FSM leakage during the human interval. Local completion still does not establish caller audibility; that requires the separately authorized OTA evidence below.

Takeover/handback is done when latched human ownership survives a short reconnect, fails back safely after grace expiry, restores model context from authoritative call state plus labelled observations, and produces complete replay evidence.

No mode is production-done until:

• named capability auth and exact-origin tests pass;
• same-call and cross-call concurrency tests pass;
• aligned operator/caller/agent artifacts prove what was sent;
• node/SHA/feature flags are visible and rollback is tested;
• the applicable L0–L3 ladder is green;
• caller-notice policy is configured;
• the next external call has a literal human GO.

21. Design iteration record

This design was stress-tested in six passes:

1. Topology: rejected the original assumption that Telnyx conference/WebRTC could be added directly to the already-streaming AI leg; selected a provider boundary and RIFF bridge first.
2. Audio truth: moved caller mirroring before the playback gate and agent mirroring after local WebSocket-send acceptance so “listen” reflects RIFF's last proven outbound boundary, not raw provider intent or unproven handset rendering.
3. Split brain: separated many listeners from one expiring speaker lease and added revision/CAS plus media generations.
4. AI leakage: expanded “pause the model” into four deterministic gates: caller input, outbound audio, provider event/tool quarantine, and static-playback freeze.
5. Recovery: made handback a server transaction with a quiescence/recreate path; added explicit PTT and takeover disconnect behavior.
6. Operations/security: retained single-node pinning, named capabilities, exact Origin, separate operator evidence, default-off flags, staged L0–OTA gates, and literal-GO protection.
7. Carrier truth: reconstructed the 2026-07-14 failed drop-in, separated browser capture, local WebSocket send, Telnyx queue completion, and handset audibility, then added mark/error evidence and a fail-closed admission barrier without changing gain, codec, framing, or call topology.

The result is implementation-ready for P0, P1, and instrumented idle-only PTT. Interrupt, gain, takeover, native Telnyx supervision, and any handset-audibility claim are intentionally gated by the evidence named above rather than implied by a button.