Voice + Render-Mode Capability Hierarchy

Date: 2026-06-30 Status: Proposal (design before build). Related: LLM-POLICY.md (the existing cascade this mirrors), preflow-architecture.

Implementation status — RESUME HERE

Branch: voice-capability-hierarchy (off main). Pushed? local-only so far.

Done (committed):

• eb7aed7 — P0.1: style is part of the static-audio cache identity. text_hash/lookup/lookup_by_text/status_for_state/write_entry take an optional style (empty = identical legacy hash). StaticAudioSpec.style threads to the synth → synthesize_opening_with_gemini_live(system_instruction=). voice_model_for_play_audio honors a real Gemini voice name (GEMINI_VOICES). Tests in tests/static_audio/test_manifest.py + test_generator.py.
• 8f66dea — this design rendered to docs/html/voice-capability-hierarchy.html
  • linked in docs/html/index.html nav.

Next (do in order):

• P0.2 — flow-level voice+style config. Add a FlowVoiceConfig (default: str, style: str) to riff/types.py parallel to FlowLlmConfig; parse a voice: flow block in riff/loader.py (mirror _parse_flow_llm ~L527); add a resolver resolve_play_audio_voice(flow, play_audio) -> (voice, model, style) that returns the state voice (or flow default) + flow style. Thread it through both the preflow path (static_audio/generator.collect_static_specs ~L64) and the phone lookups (phone/telnyx_transport._static_audio_pcm24_for_current_state ~L1241, and the lookup_by_text call for data-resolved states) so the phone finds the styled WAV. Defer web_server/browser/MCP metadata surfaces.
• P0.3 — Space Channel → Umbriel British (RISKY, isolated). Set the flow voice to Umbriel + British style; migrate the droid_full_buffer states to static_audio; handle the _droid_voice_id blast radius (telnyx_transport.py:777,1644 — removing the droid voice un-suppresses Gemini audio_out for every non-suppressed state); update the droid-coupled tests (tests/phone/test_telnyx_transport.py ~431,656; test_space_channel_flow.py:45); re-render via the preflow; operator calls the line to confirm British Umbriel before merge. Do NOT touch flows/space_channel_audio_test.yaml (droid test fixture).

Honor the Codex review findings below (per-state live voice is impossible; ad-lib = non-play_audio state; capability lives on StateDef/GoalSegment; use a generic cascade helper, not the LLM resolver).

The idea (from the operator)

"the capabilities should be able to use the LLM at a level of overall, by flow or by group or by state — this would give us a hierarchy." "prerender the voice in the preflow … but also have an option to go to gemini-live in a state … so we could test adlib-type workings."

RIFF already has this hierarchy for the LLM (state → group → flow → global cascade; FlowLlmConfig.classes + default_class; per-state llm:; inherit walks up). This proposal makes voice and render-mode first-class citizens of the same cascade.

Two new per-state capabilities

1. Voice (name + style), resolved via the cascade

A voice is a Gemini voice name + an optional style/accent instruction:

voice:
  default: anchor
  classes:
    anchor:     { name: Umbriel, style: "Polished British RP news anchor, warm and clear" }
    gatekeeper: { name: Charon,  style: "Deep, formal station-ops voice" }
states:
  some_state:
    play_audio:
      voice: gatekeeper      # class name, explicit voice, or "inherit" → group → flow → global

Resolution = state → group → flow.voice.default → global, most-specific wins — identical shape to the LLM resolver. The style maps to the TTS system_instruction (that plumbing already landed in openings.py).

2. Render mode: prerendered vs live (ad-lib)

Scope note (2026-07-10): per the adopted boundary-architecture ADR (C7), call-time synthesized narration of validated slots/runtime results — dynamic_announce — is a distinct compiled audio-only state type with its own delivery receipts and input-hub contracts, not a play_audio.render_mode value. The render-mode dial below covers voice/synthesis identity for a state's own audio, and stays.

Per state, where does the audio come from:

• prerendered (default): the preflow rendered a WAV with the resolved voice; the phone serves it from the static-audio cache (the voice-agnostic _static_audio_pcm24_for_current_state path). Fast, deterministic.
• live: route the state to the Gemini Live session for real-time generation in the resolved voice — for ad-lib / dynamic states. Higher latency, but improvisational.

This is the dial that lets a flow mix scripted briefs (prerendered) with ad-lib turns (live), per state.

Why this is now tractable (foundation already in place)

• static_audio.manifest.voice_model_for_play_audio now honors a real Gemini voice name (e.g. Umbriel) instead of forcing Achernar — so a state can pick its voice and the manifest keys it correctly.
• openings.synthesize_opening_with_gemini_live now accepts a system_instruction — the "style/accent" channel.
• The phone transport already has a voice-agnostic prerendered path (_static_audio_pcm24_for_current_state, telnyx_transport.py:1223): it loads the cached WAV by resolved voice and emits PCM. Gemini WAVs play on the phone exactly like droid WAVs — same artifact. The only thing that diverts a state away from it today is render_mode: droid_full_buffer.

Implementation plan (opt-in; existing flows unchanged)

The default voice/mode stays exactly as today, so droid flows and all current tests are unaffected. A flow opts in by adding a voice: block.

1. Types + loader — add FlowVoiceConfig (default, classes: {name, style}) parallel to FlowLlmConfig; parse a voice: flow block and per-state play_audio.voice / render_mode: live.
2. Resolver — resolve_voice(flow, state_id) and resolve_render_mode(...) reusing the LLM cascade walk (state → group → flow → global).
3. Preflow — render each prerendered state with its resolved voice + style (replace the hardcoded space_channel synth with cascade resolution).
4. Transport — for render_mode: live, set the Live session's voice + system_instruction to the resolved voice/style for that state instead of playing a prerendered WAV.
5. Adopt in Space Channel — delete the droid voice, set voice.default: anchor (Umbriel + British). Migrate droid_full_buffer states to the prerendered static path. Update the droid-coupled tests.
6. Verify on a real call — the operator dials the line and confirms British Umbriel plays (prerendered) and any live state ad-libs.

Codex plan review (2026-06-30) → REVISE; design reshaped

A pre-build Codex review found 9 substantive issues. The feasible design after folding them in:

• Voice per state applies to PRE-RENDERED audio only. Each static WAV can be a different Gemini voice (the phone looks it up by voice_model_for_play_audio → manifest). This part stands.
• Live audio has ONE voice per call, not per state. The Gemini Live session is created once with an immutable voice + system_instruction (live/policy.py, live_client.py:227). So: a flow picks a single live voice (call-level, set at session setup); you cannot swap the live voice mid-call. Drop "per-state live voice."
• "Ad-lib" is a normal dialog state, not a render_mode on play_audio. play_audio states are classified static and the model is suppressed (live/session.py:437,747); the loader only allows static_audio|droid_sentence_stream|droid_full_buffer (loader.py:655). To test ad-lib, author a normal (non-play_audio) dialog state — it already runs on the live session. It needs a response_emitted transition or it stalls (live/session.py:1525). So "ad-lib per state" = don't mark it play_audio, not a new render mode.
• Style must be part of the static-audio cache identity. Add a style (system_instruction) field to StaticAudioSpec and include it in the manifest hash (manifest.py:67 is text|voice|model today), and pass it through generate() → synthesize_opening_with_gemini_live(system_instruction=...). Otherwise changing "British RP" never re-renders.
• Use a generic cascade helper, not the LLM resolver. resolve_model_name is model-specific (adapters/__init__.py:74). Extract a shared resolve_capability(state→group→flow→global, inherit) walk; voice/llm each supply their own class table + validation.
• Capability fields belong on StateDef/GoalSegment, with flow + global defaults and a load-time validation gate (unknown voice, bad alias, every state resolves) — mirroring the LLM policy's boot-time gate.
• Thread the resolver through every surface that calls voice_model_for_play_audio today: phone playback (telnyx_transport.py:1241), preflow (generator.py:64), web/phone server metadata (web_server.py:439, phone/server.py:735), browser Live state (live/transport.py:671), and MCP state export. Plus tests.
• Space Channel droid removal is its own risky step. The _droid_voice_id gate (telnyx_transport.py:777,1644) changes audio behavior for every non-suppressed state, not just droid_full_buffer. Do it isolated, with test updates and a real-call check.

Revised, decomposed plan

• Phase 0 (small, shippable): Space Channel → prerendered Umbriel British, flow-level voice. Add style to the static-audio spec+hash; set the flow's static voice to Umbriel + British style; migrate droid_full_buffer → static; handle the droid-removal blast radius + update tests; preflow re-renders; verify on a real call. No cascade yet — one voice for the flow.
• Phase 1: generic cascade helper + voice: block (flow/group/state, with the load-time gate), threaded through the surfaces above.
• Phase 2: ad-lib dialog states — document the "non-play_audio state + response_emitted" pattern and a single call-level live voice; no per-state live voice.

Open design question

Keep voice: and llm: as parallel flow blocks (each with its own classes/cascade), or unify into one capabilities: block with llm / voice / mode sub-keys sharing a single resolver? Parallel is less refactor; unified is cleaner long-term. Recommend parallel now, unify later if a third capability appears.

Risk / discipline

• Strictly opt-in — no behavior change unless a flow adds voice:.
• The Space Channel migration is the riskiest part (droid coupling in the phone transport + tests); do it as its own reviewed step with a real-call check.
• Per house convention, this plan should get a Codex review before the build.