← Choice Health fable5-deterministic-flow-questions.md raw .md

RIFF — Deterministic Voice Flows: Architecture Questions for Fable 5

Author: Claude (Opus 4.8) with David Mar Date: 2026-07-01 Audience: Fable 5 (design/architecture review) Scope: the last two days of work (2026-06-29 → 07-01) on the RIFF "Space Channel" phone flow — a numbered voice menu (press or say 1–8) that must route deterministically and play prerecorded audio.

0. Why this doc

We spent two days turning a Gemini-driven concierge into a deterministic IVR. It mostly works — but the same classes of bug keep recurring, and we keep re-litigating the same design questions. What we ultimately want:

  1. A better way to represent all the fixed menu choices (1–8) — and a UI a human can trace and debug during a live call. (The graph renders routing topology, not the caller-facing menu, so the 8 options are invisible where it matters.)
  2. An avenue for experimentation that doesn't destabilize the deterministic core.
  3. Cost optimization across different LLMs at minimal latency.

Our thesis for how all three fall out of one architecture — pre-flow → flow → post-flow ("dreaming"), unified by simulation — is in §1.5. We want Fable 5's help pressure-testing that thesis and the concrete UI strawman (§3.5), not just the next patch. Questions are in §4; §1–§3 give you the context.

1. The system in 60 seconds

RIFF is a phone-call AI agent. A call is driven by a finite state machine (FSM) defined in YAML (flows/*.yaml): states, transitions, guards. The language layer is Gemini Live (streaming speech-in / speech-out).

Two transports share the FSM but have separate audio handling:

Key mechanisms built over these two days:

The flow's shape today:

greeting ("How may I help you?")
  └─> triage        [locked_choice: 1..8 + 9/goodbye + "options"]   ← NO LLM
        ├─ 1 launches       → report_launches_audio        (prerendered)
        ├─ 2..7 content     → report_space_content_audio    (prerendered, per topic)
        ├─ 8 bug            → collect_bug → (loop back)
        ├─ 9 / "goodbye"    → sign_off (terminal)
        └─ unrecognized/"options" → suggestions_announce → suggestions (full menu)
  every subflow ──audio_done──> back to the MAIN MENU (re-announced)

1.5 The thesis: pre-flow → flow → post-flow (dream), unified by simulation

The real organizing idea (David's framing) is a three-phase lifecycle. Almost every fix in §2 is really "we put work in the wrong phase." Getting the phases right is the architecture.

      PRE-FLOW  ───────────▶   FLOW   ───────────▶   POST-FLOW ("dreaming")
   prepare, deterministically   execute, min latency   review · experiment · optimize
   ─ fetch all content          ─ fixed menus: NO LLM   ─ replay the trace
   ─ render every WAV           ─ prerendered audio      ─ simulate alternatives
   ─ warm caches / STT          ─ LLM only at the edges  ─ try cheaper/faster LLMs
   ─ pick per-state LLM         ─ deterministic routing  ─ measure cost/latency/quality
        │                              │                        │
        └──────────  SIMULATION is the connective tissue  ──────┘
             (same simulator validates pre-flow, debugs flow, drives the dream)

Pre-flow — everything expensive or slow happens here, OFF the call. Fetch content, render all WAVs, warm the choice-STT, and decide which LLM each state will use. The output is a fully-prepared, mostly-deterministic call plan. (The blocking-fetch reconnect-loop scar, 643369b1, is exactly a pre-flow job that leaked into flow.)

Flow — the live call, latency-minimal by construction. Fixed menu choices resolve with no LLM (deterministic, instant — that's the whole point of locked_choice). Prerendered audio plays immediately. The LLM touches the call only at declared edges (free-text collect; no-match fallback), and there it runs the cheapest model that meets quality. Latency is spent only where a model is genuinely unavoidable.

Post-flow ("dreaming") — where experimentation lives. After the call, replay the recorded trace and simulate variations: different wording, different LLMs (cost/latency/quality trade-offs), different routing thresholds, new menu options. This is the answer to the tension the user named — (a) fixed choices stay deterministic in flow, while (b) the avenue for experimentation lives in post-flow, so trying things never destabilizes the live deterministic core. Wins get promoted back into pre-flow/flow. We already have much of the substrate: quality.db, LLM judges, the all-options harness (scripts/space_channel_all_options_sim.py), per-state model cert, and an "Effective LLM Policy" inheritance cascade (state→group→flow→global) that isn't fully wired to the primary turn yet.

The three things we want this to deliver (David's words): (a) a better way to represent all the fixed menu choices; (b) an avenue for experimentation; (c) cost optimization across different LLMs at minimal latency. The claim is that all three fall out of getting pre-flow / flow / post-flow right, with simulation as the shared engine — not out of another one-off patch.

2. What we built in two days (grounded in commits)

A compressed timeline of the fixes, because the pattern of fixes is itself the subject:

Content & voice foundation (06-29 → 06-30 midday)

The "Gemini improvises the menu" bug (06-30 evening)

The deterministic menu (06-30 evening)

The blocking-fetch trap (06-30 night) — a scar we still route around

Prefetch / Design-1 (07-01 early)

Welcome direct-dial + goodbye (07-01 early)

The web transport lagging the phone (07-01 morning) — the divergence tax

Fully deterministic hub + loop-back (07-01 midday) — this session

3. The recurring tensions (the patterns that keep biting)

These are the through-lines. Each is a place where a design decision would stop a class of bugs.

T1 — Determinism vs. the microphone. The router is perfectly deterministic once it has clean input. But the input is Gemini's ASR, which misheard "network" → "work" on a live call — so the router correctly refused to guess, and the caller hit a slow no-match reprompt. DTMF is immune; speech is not.

T2 — The 8 options are invisible in the graph. Six of the eight (2–7) route to the same report_space_content_audio state, so the map collapses them into one edge. The caller-facing menu (1–8) lives inside the locked_choice node. The operator can't see "what does 4 do" at a glance.

T3 — Two transports, every fix landed twice. Phone and web share the FSM but not the audio path. The commit log literally contains "mirror phone fix" and "the REAL live bug" — the same defect rediscovered on the other transport.

T4 — Routing tied to the model's turn boundary. The FSM advanced on turn_complete; Gemini decides when that fires (VAD), so routing inherited a 13–46s tail. locked_choice early-dispatch sidesteps it, but only for states we've explicitly made deterministic.

T5 — Fresh vs. non-blocking. Prerendered audio is fast and safe but can be stale; a realtime fetch is fresh but blocked the async loop and crashed the Gemini socket. We now hard-code "launches never fetches on the loop." That's a rule, not an architecture.

T6 — Offline-green, live-broken. Our FSM sims and the all-options harness were green while live calls stalled — because the bugs live in the live timing (recv-pump order, transport push, ASR partials, turn boundaries), which the sims don't model. Every real fix this session came from reading a live call log, not a test.

T7 — Act/tool states in the conversational path are an anti-pattern. Every time the FSM lands on a required_tool state that hands the mic to Gemini, Gemini improvises. Our fix is always the same: route around it to prerecorded audio. But we keep re-introducing the shape.

T8 — One state, many contents, revisited. report_space_content_audio is a single state reused for 6 topics and re-entered on every loop-back. That reuse broke state-id-keyed dedup and slot-fill assumptions. Is "one reusable state" even the right model?

3.5 The visualization we keep circling (the real ask)

We have rebuilt the FSM view ~6 times in two days (ff1321a4 → 87b70982 → 9a539db5 → f5edf099 → e394e345 → 18dc932b) and still don't have the thing we want: a simple picture of the states that a human can trace and debug during a live Space Channel call. Below is a strawman. Please tear it apart — the goal is to stop re-litigating this.

Principle 1 — one node per thing the caller experiences, not per YAML state

The YAML has plumbing states (suggestions_announce + suggestions are an announce atom and a locked-choice atom that together are one thing: "the menu"). Draw the molecule, not the atoms. Click to expand into atoms only when debugging. Target vocabulary — five node shapes, nothing else:

shape meaning Space Channel example
▭ Say plays a prerecorded WAV, then auto-advances greeting, every briefing, goodbye
◇ Choose deterministic router: press/say → route (no LLM) triage hub, the numbered menu
▱ Collect gathers free text via the LLM bug description
⬒ Do a tool/fetch (should be out-of-band; flag if on the mic) (none, by design)
⬤ End terminal sign_off

Principle 2 — two synced views, one source of truth (the YAML)

Principle 3 — trace & debug are first-class, live

┌───────────────────────────────────────────────────────────────────────────┐
│  Space Channel · ● LIVE call  ·  turn 3  ·  ⏱ last route 0.3s               │
├──────────────────────────┬────────────────────────────────────────────────┤
│  CALLER VIEW (keypad)     │   ENGINEER VIEW (states)                        │
│                           │                                                 │
│  ┌───┐┌───┐┌───┐┌───┐     │     ▭ greeting                                  │
│  │ 1 ││ 2 ││ 3 ││ 4 │     │        │ audio_done                            │
│  │Lau││New││Wx ││UFO│🟢    │     ◇ triage ●gold  ← "missions" → mission_ops │
│  └───┘└───┘└───┘└───┘     │        ├─1→ ▭ launches   🟢                     │
│  ┌───┐┌───┐┌───┐┌───┐     │        ├─2..7→ ▭ briefing (per topic) 🟢        │
│  │ 5 ││ 6 ││ 7 ││ 8 │     │        ├─8→ ▱ bug                              │
│  │Mis││Net││Rad││Bug│     │        └─9→ ⬤ goodbye                          │
│  └───┘└───┘└───┘└───┘     │        ▭ briefing ─┈loop┈─▶ ◇ menu             │
│   press or say · 9=bye    │                                                 │
├──────────────────────────┴────────────────────────────────────────────────┤
│  TRACE (why we are here)                              [▶ step] [⏸] [⟲ sim] │
│  0.0s  ▭ greeting          played greeting.wav (5.0s)                       │
│  6.2s  ◇ triage            heard "capabilities" → matched alias → OPTIONS   │
│  8.1s  ◇ menu              heard "missions"     → matched alias=mission ✓   │
│  8.1s  ▭ briefing[mission] ▶ playing mission_ops.wav (24.5s)   🟢 pushed    │
│  ⚠ 33s ▭ briefing          WAV ended, no audio_done from browser → STALL?   │
└───────────────────────────────────────────────────────────────────────────┘

The trace panel is the debugger: every row is what the caller did → how the router decided → what played, with timing. A red row is a diagnosis ("state went active but its WAV never played — Gemini is improvising" already exists in 8bb182c8; "WAV ended, no audio_done → stall" is the bug we hit today). The same view runs on live (observer WebSocket) and offline sim (step through paths with no call), so you debug the identical picture whether or not you're on a call.

What's unresolved about the strawman

4. Questions for Fable 5

Please engage with the architecture. Where you'd change the model, say what you'd replace it with and what breaks. Concrete counter-designs welcome.

A. A better view of 1–8 (visualization + mental model)

Q1. Our graph draws routing destinations, so 2–7 collapse into one "Content Briefing" edge and the caller-facing menu (1–8) is hidden inside the locked_choice node (T2). What's the right visual model for a menu where N caller choices fan into M destinations? Options we've considered: (a) label each edge with its digit set; (b) render 8 "choice chips" as first-class nodes that alias into the destination states; (c) a dual view — a "caller view" (1–8 as a keypad) and an "engineer view" (state topology) that stay in sync. Which — and why? Is there a canonical representation for IVR/menu graphs we should adopt rather than invent?

Q2. We want an operator to answer "what happens when I press 4?" in one glance, including the audio it will play and whether that audio is prerendered/fresh/stale. What belongs on the node vs. the edge vs. a side panel? Is a graph even the right primary artifact, or is a table (option × input-method × destination × audio-state × freshness) the better operator surface, with the graph as secondary?

Q3. The same state (report_space_content_audio) appears once in the graph but means six different briefings (T8). Should a reused, content-parameterized state be drawn as one node (honest to the FSM) or six (honest to the caller)? What's the principle for when to unfold a parameterized state in a diagram?

B. Determinism vs. natural language / ASR (T1)

Q4. Deterministic routing depends on clean tokens, but ASR mishears ("network"→"work"). Where should robustness live? (a) Enrich aliases with known ASR confusions (whack-a-mole); (b) fuzzy/phonetic match inside locked_choice (risk: false matches route the caller wrong — arguably worse than a reprompt); (c) a small, dedicated, constrained speech recognizer that only has to distinguish 8 known options (grammar-constrained ASR) instead of open-vocab Gemini; (d) confirm-on-low-confidence. What's the right default, and how do you set the confidence threshold so you neither mis-route nor nag?

Q5. When nothing matches, the reprompt is slow (it still waits for the model's turn boundary — T4). Is "deterministic router + LLM only as a no-match fallback classifier" the right escalation ladder? If so, how do we keep the fallback from re-introducing the latency and improvisation we just removed — does the fallback return only a route (never audio), and never touch the mic?

Q6. Should the design bias callers toward DTMF (immune to ASR) without feeling like a 1990s phone tree — e.g. "press or say," barge-in, and voice as a fast-path over a reliable keypad substrate? Where's the line between "deterministic and reliable" and "robotic"?

C. Holding the flow together (routing + conversation)

Q7. Today routing advanced on Gemini's turn_complete (T4); we bypass it with locked_choice early-dispatch on the transcription. Is there a cleaner invariant: routing must never depend on the language model's turn boundary — the FSM owns turns, the model is a stateless "utterance → intent/text" function? What would we lose (barge-in nuance? backchannel?) by enforcing that?

Q8. T7: act/required_tool states in the conversational path invite the model to improvise; our fix is always "route around them to prerecorded audio." Should all tool calls / data fetches be out-of-band (done in the preflow or a background worker, never a conversational state the caller can land on)? If a tool genuinely must run mid-call, what's the pattern that keeps the mic away from the model?

Q9. T5: how do we get fresh-enough content without call-time blocking? Is serve-stale-while-revalidate (play the prerendered WAV instantly, kick a background refresh for next time) the right default for all dynamic content — including the volatile launch data we currently refuse to fetch on the loop? What freshness SLA is honest to state on a phone call?

D. Two transports, one contract (T3)

Q10. Phone and web share the FSM but not the audio path, so fixes land twice ("mirror phone fix"). What's the right seam? A single "audio intent" the FSM emits (play(state, resolved_wav, voice)) that each transport merely renders? A shared "static-audio playback + advance-on-done" module both transports call? How do we make it impossible to fix a playback bug on one transport and not the other?

Q11. The web path only advances a static state when the browser signals "audio done"; the phone path uses its own recv-pump. This asymmetry caused the loop-back stall. Should "the WAV finished, advance the FSM" be a transport-agnostic contract with a server-side timeout fallback so a missed client signal can't strand the caller?

E. Testing that catches live bugs (T6)

Q12. Every real fix this session came from a live call log, not a test; our FSM sims were green while calls stalled. What's the minimum-fidelity simulation that would have caught these — one that models ASR partials, turn_complete timing, transport push order, and the browser's play/ack round-trip? Is it worth building a "live-faithful" harness, or is the honest answer "instrument production heavily and read logs" (we added visible static-audio pushed/SKIP logging for exactly this)?

Q13. Meta-question: given T1–T8, if you were redesigning this from scratch as a deterministic, prerecorded, voice+DTMF IVR with an LLM strictly at the edges, what would the core abstractions be? Where exactly is the LLM allowed to touch the call, and what's the contract at that boundary?

F. Pre-flow / flow / post-flow, experimentation, and cost-at-minimal-latency (§1.5)

Q14. Is pre-flow → flow → post-flow(dream) the right top-level decomposition, with the invariant "if it's slow, expensive, or non-deterministic, it belongs in pre-flow or post-flow — never in flow"? Where does that invariant break (e.g., genuinely live data like "is this launch still go?"), and what's the escape hatch that doesn't reintroduce the mic-to-model improvisation and blocking-fetch traps?

Q15. We want (a) fixed deterministic menus and (b) an avenue for experimentation in one system. Is the clean split "flow is fixed/deterministic; experimentation lives in post-flow simulation and is promoted into flow only after it beats the baseline"? How do we represent a menu that is mostly fixed but has one experimental branch, in both the model and the UI, without the experiment leaking latency or nondeterminism into the fixed paths? Is there a notion of a "shadow" / candidate branch that runs only in simulation against recorded calls?

Q16. Cost via different LLMs at minimal latency. We have a per-state LLM policy (inheritance cascade state→group→flow→global) but eval is noise-dominated (judge σ≈1.5 on a 1–10 scale), so sub-point quality deltas are unreliable. How should per-state model selection be decided in pre-flow and validated in post-flow so we pick the cheapest/fastest model that holds quality — given the noise floor? What's the right objective function over {cost, latency, quality, humanness}, and how do you keep it honest when quality is barely measurable?

Q17. "Dreaming" as a loop. Should post-flow be automated (the system proposes variations — reword this WAV, swap this model, add this alias — simulates them against recorded/synthetic calls, and surfaces ranked wins) or operator-driven (a human explores in the UI)? If automated, what stops it from overfitting to the eval judges (we've been burned: two "signals" turned out to be measurement artifacts)? What's the human-in-the-loop checkpoint before a dream is promoted to flow?

Q18. One simulator, three phases. Simulation must validate pre-flow (does every option deliver fresh audio?), debug flow (step a path, see why it routed), and drive post-flow dreaming (what-if). But our sims were green while live calls stalled (T6). What fidelity does the shared simulator need — and can the same engine honestly serve all three phases, or do pre/flow/post need different simulators with a shared trace format?

5. What "good" looks like

We'd consider this solved when:

Fire away — counter-designs and "you're framing this wrong" answers are the most useful.


*Appendix: key files — `flows/space_channel_network_ops.yaml` (+ `flows/subflows/space_channel_network_ops/*`), `riff/live/session.py` (locked_choice early-dispatch), `riff/live/transport.py` (web audio push), `riff/phone/telnyx_transport.py` (phone audio), `riff/static_audio/` (prerender + `lookup_latest_by_state`), `web/space-channel-map.html` (flow map), `scripts/space_channel_all_options_sim.py` (all-options harness).*