The RIFF Evaluation & Goal-Hierarchy System
How a voice-AI flow engine measures itself — at four nested levels, with cheap deterministic gates per commit and an expensive judged matrix at night.
Onboarding path
Use this sequence when joining the project, reviewing a live-call issue, or handing context to another AI worker.
Product Priority WorkbenchNew2026-07-11
Rank RIFF initiatives against editable, provenance-bearing principles; re-score the full queue with one bounded model call. Model the price floor, inspect dependencies, and keep one plumbing booking pilot ahead of adjacent product ideas.
Browser phone simulatorNew2026-07-09
Run a phone-shaped call locally with an entered caller number, a routed business line, real DTMF, Replicant identity prefill, live provider tools, and the FSM debugger on one screen. Mutations are blocked unless explicitly enabled.
Live Supervisor JoinNew2026-07-14
Computer listen/talk design with click-toggle control, a fail-closed Telnyx queue barrier, truthful handset evidence, and deterministic AI handback.
Call Replay DebuggerNew2026-07-11
Replay aligned caller and outbound audio against one labelled time scale for Caller, Agent, State, and Finding evidence. Inspect the exact fired guard and reason, trace a variable across every Entry/Exit boundary, compare repeated visits, and audit pauses without model calls.
How to Debug a CallNew2026-07-10
Operator and AI runbook for choosing a call, trusting only aligned replay evidence, comparing Gemini and Whisper observations, stepping the same artifact through the browser or CLI, and separating sent audio from carrier-heard proof.
Debugger Audio CausalityNew2026-07-12
Trace one logical response through provider generation, every playback attempt, interruption or restart, state context, and the exact audio the caller received. The paired implementation plan divides capture, artifact, detector, API, CLI, and UI work into bounded packets that lower-capability agents can execute and verify.
Unified Replay Completion PlanNew2026-07-10
Evidence-backed implementation contract for one truth-oriented debugger. It compares generated, outbound, legacy composite, and aligned replay WAVs across three real calls; proves which repetitions were actually sent; and divides canonical loading, per-channel Whisper, causal timelines, durable markers, flow fixes, and isolated branch simulation into testable work packets.
1. Mental model
Understand the goal hierarchy: flow quality is a symptom, and state/group metrics are the path to root cause.
2. Build a flow
Learn the pack schema that compiles a business profile into deterministic voice behavior, companion surfaces, and tests.
3. Import facts
See how website and company data become strict packs without letting generated prose become source-of-truth facts.
4. Debug and gate a callNew
Start here when a caller hears silence, a menu sounds wrong, or the transcript says words were generated but the audio does not prove they were heard. The post-run gate turns those findings into a failing command after OTA or local runs.
5. Record the lesson
Write the fix journal entry with symptom, root cause, test gate, and exact call/log evidence so the next agent starts smarter.
6. Promote safelyNew
Use the M3 candidate lane and M1 stable lane deliberately, with health checks before live traffic changes.
RIFF is a voice-AI flow engine. A flow is a conversation script for a phone agent — “book a plumbing visit,” “take a coffee order,” “reserve a table.” Under the hood a flow is a finite-state machine (FSM): a graph of states, with edges (transitions) guarded by deterministic string expressions evaluated against the call's slot bag. The large language model (LLM) only produces language; it never decides which state comes next. That separation — declared graph in the FSM, words from the LLM — is the spine the whole evaluation system is built on.
This documentation describes everything built to answer one question reliably: “is a flow good, and if not, exactly which part is broken?” The answer is not a single number. It is a four-tier goal hierarchy, where every level declares a goal and is instrumented with a metric for that goal.
The four-tier hierarchy
Read top-down. Each tier rolls up into the one above it, and each tier has its own “what good means” and its own measurement.
collect, every confirm, every action — grouped into one bucket.collect the slots, confirm the read-back, act (a tool succeeds), reach a terminal, or handoff safely.group_cohesion = goal_yield · (0.35·efficiency + 0.65·transition_coherence) — do the group's states work together? Group cohesion »--analyze → --matrix → --segment-matrix →
--state-matrix --flow F.
Computed vs. judged — the cost split
The single most important design decision: separate what we can compute deterministically from what we must judge with an LLM.
| Computed aspects | Judged aspect | |
|---|---|---|
| which | correctness, completion, errors, latency, and all per-state / group metrics | humanness only |
| source | the live event stream (state transitions, guards, tool calls, timing) | an LLM grading a transcript against a rubric |
| noise | none — deterministic | ~1.5 points single-run (judge-dominated) |
| cost | free, fast | API calls, slow |
| cadence | every commit (the CI gate) | nightly (the judge panel) |
Because only humanness pays the judge-noise tax, most of the scorecard is trustworthy and cheap, and we spend LLM calls only on the genuinely subjective axis. See Methodology & lessons for why the noise floor forces repeat-N and why “harden the instrument” is the prime directive.
The quality database
Everything lands in one SQLite file, docs/flow-eval/quality.db, driven by
scripts/eval_db.py. Five tables hold the truth; a dozen read-time views slice it.
| Table | Grain | What it holds |
|---|---|---|
runs | one per eval run | commit, branch, dirty flag, agent model, judge flags — the feature captured at run start |
scores | flow × persona | legacy quality/robustness/success + per-cell variance (quality_runs) |
aspect_scores | conversation × aspect × judge | the tidy long-format matrix; computed rows use judge='(computed)' |
state_scores | conversation × state | per-state root-cause metrics, tagged with the state's segment |
flow_mods | flow × commit | every git commit that touched a flow's YAML — so a score change can be tied to an edit vs. noise |
Read this in order
Boundary architecture ADRNewadopted 2026-07-10
The architecture direction: adopt the FSM / deterministic-code / LLM boundary as eleven enforced contracts (C1–C11) and evolve the current engine in place; the event-bus rewrite is declined with explicit reconsideration triggers. Every contract needs a compile/load gate, runtime behavior, trace evidence, and a replay test — a declaration alone doesn't count. First applied scope: the 512-2777-LAW line (compiler copy-seam fixes FX-100, stale-flow regeneration + routing alignment FX-101, and a committed-artifact contract test for the line).
Session Debug WorkspaceNew2026-07-07
One merged debug artifact per call ("why did it ask for the address twice?" answered in three clicks) and a live state inspector inside flow-studio: required/filled/missing triad, guard evaluations, transition verdicts — derived views over events riff already emits, no stored state. Shares its data spine with the Call Replay Atlas and makes the atlas's SLOTS/VERDICTS lanes lossless for long calls. Ships with an 8-task TDD plan.
Two-node release channelsNew2026-07-08
Operational contract for running M3 as the candidate/latest-code lane and M1 as the stable fallback. Documents the implemented scripts/node-mode.sh commands, richer /healthz fields, live-traffic semantics for stable/candidate/shadow/drained, and the current gap: caller-ID canary routing and simulator --node M1|M3|both are still planned.
Node routing policy v3New2026-07-09
Design approved, not deployed. Defines the Cloudflare Worker/Durable Object ownership authority, caller and line policy, fail-closed behavior, clean deployment gates, and phased path to a real routing dashboard. Includes a dated implementation table: the property flow is live, webhook authentication is committed but awaits the Telnyx public key, and routing/fallback controls remain preview-only.
Fable 5 node-routing reviewNew2026-07-09
Historical adversarial review of v2 and its v3 disposition. Covers unauthenticated ingress, unknown forwarding outcomes, answer fencing, crash-recoverable deduplication, persistent fail-closed node mode, candidate fallback limits, soft drain, key rotation, and Worker/DO rollback.
Replicant ticket operationsNew2026-07-08
Fable5 review draft for extending the live Replicant integration beyond identify/create/follow-up: tenant-visible list/get/update/cancel operations, operation-level idempotency, tenant-writable field limits, soft-delete semantics, retry behavior, and open contract questions.
Business Pack import architectureNew2026-07-03
Design brief for shifting onboarding left: company JSON and AI-generated profiles normalize into a strict RIFF Business Pack, then compile into deterministic flows, companion surfaces, scheduler policy, audio, and tests.
Website ingest importerNew2026-07-09
Point it at a business's website, get a phone menu: two commands turn mfclaw.com's schema.org JSON-LD into a pack that compiles clean (0 errors, 0 warns) with a 150-character, 32-word greeting over 14 hidden voice-only FAQ intents. No LLM in the fact path — FAQ answers are spoken verbatim. Now with the defect ledger: seven real bugs this build found and fixed — a 60-second browser-tier hang, two business nodes silently merged wrong, an inverted JSON-LD traversal order, 11 alias collisions from one FAQ set, an 827-character menu greeting whose TTS render failed, a build gate that guessed the cause, and a silent id-collision data loss — each with its measured numbers and a named regression test.
Progressive web intakeNew2026-07-04
A generated intake webpage for Space Channel, law, plumbing, electrical, and future verticals: phone collects L0; web progressively offers contact, detail, points, security, and credit/billing modules. Space Channel option 0 is live as an experimental SMS-verified profile intake.
Field stall policyNew2026-07-05
Fable 5 review brief for shifting intake safety left: every voice-collected field declares how silence behaves, and build-time rules fail unsafe stalled paths before live calls.
Judge-the-Judge & disciplineNew2026-06-24
Never trust a verdict you can't cross-check. The judge panel (Gemini/Qwen/Codex), the deterministic guard, the scanner that over-flagged 60%, and why repeat-N beats one run.
Call Replay & Layered AtlasNew2026-07-07
One command turns any call — live, archived, or simulated — into a self-contained page with four time-aligned lanes: what the caller heard, the FSM path, the slot ledger (who set/erased what), and transition verdicts ("edge 0 matched; Webb shadowed"). Plus --atlas: the compiled flow drawn as trust layers (guest / known / verified), because the flow a caller walks mutates with what they've proven. TDD from the doc's own cases: tests/test_call_replay.py.
Session Facts: end-flow behaviorNew2026-07-07
Use-case-first contract born from two broken live calls: a passcode gate that destroyed a verified caller's menu choice, and a consent question re-asked after consent was saved. Five facts per call (identity, auth level, consent, chosen route, journey), gates that announce once instead of interrupting, and the two telemetry lanes (slot_write, transition_verdict) that make every "never" clause a one-line assertion. Every case maps to a test in tests/test_session_facts_endflow.py; spec twin docs/session-facts-proposal.md.
How-To: brand your follow-along page2026-07-06
One YAML block gives a flow (or a whole org) its own colors, badge, tagline, and caller-friendly journey step labels. Covers what lights up as the phone reads (menu glow, karaoke word-sweep, fresh-slot flash — all take your highlight color) and a gallery of all 8 running flows with their real menus and slot echoes.
Config cascade & per-org branding2026-07-06
One six-level cascade (call/state/state-group/flow/org/global), five independent domains, a provenance-carrying resolver — and every flow's follow-along link in its own colors. Codex-reviewed v3.
Business admin access: panel + loginNew2026-07-08
Every deployed business gets its own admin panel, a passwordless token → scope login, and a one-page handoff doc the deploy writes. The panel reads only that business's callers/leads (WHERE scope = ?); a second business's token shows a disjoint view. Covers the two FAQs (it's for all businesses; the token is a deploy artifact, not a build one — and redeploy rotates it), 5 use cases mapped to tests, an architecture diagram, and a code map. Built on per-business identity scoping; token stored hash-only, handoff 0o600+gitignored (FX-028).
Effective LLM Policy2026-06-23
Route the right model per state (inheritance + override). The four cases, the proof harness, the drift risk. Note: the first routing assignment was a misdiagnosis — see the validation update.
Space Channel subflowsNew2026-06-29
A skills-style parent/subflow design for Space Channel: lazy manifest discovery, route_candidate dispatch, public Space Channel data cards, and simulator gates before OTA tests.
Calendar flows & tutorialsNew2026-07-02
The whole calendar story for a newcomer: the two booking engines (and the dead-air tradeoff), the full use-case catalog, a tour of every calendar flow, and a state-by-state walkthrough of the booking tutorial with its five safety invariants.
Goal-Hierarchy design
The flagship spec: flow → group → state, each with a goal and a metric. Start here.
The evaluation matrix
Tier-3 flow scorecard. Computed aspects + judged humanness, crossed by agent.
Per-state metrics
Tier-1 root cause. How the event stream becomes progress / stall / dwell.
Group cohesion
Tier-2. Do a group's states collaborate, or thrash? The bounded metric.
Archetypes
Tier-4. The same subpart across all flows — systemic-weakness finder.
GoalSegment schema
The typed group contract (new this session) — the data future phases consume.
Judges
The roster: remote, local, and CLI judges. Spread = humanness uncertainty.
Backend mocks
Letting backend-gated flows reach their payoff in eval — without faking guards.
Holdout validation
Dev vs. held-out personas — the overfitting guard. Never debug the held-out set.
CI quality gate
The per-commit deterministic gate + the pre-push hook.
Session replay
Re-drive recorded sessions after a change — the regression check.
Stall recovery (FSM)
The stalled guard + sets: edges — deterministic recovery from a withholding caller.
Call quality & reviews2026-06-28
The two-layer post-call review system: live audio recording privacy cascades, Whisper-vs-Gemini timeline debugging, VAD tuning, and nightly batch evaluations.
Post-call dream loopNew2026-06-29
The closed-loop architecture for turning a real phone failure into targeted simulations, judged A/B/C variants, and evidence-backed promotion.
Call timeline mockNew2026-06-29
A deterministic phone-call timeline view: logged FSM facts, measured caller/agent audio, expected processing tones, silence gaps, and embedded JSON for machine review.
Voice Quality PlaybookNew2026-06-28
The playbook for voice reliability: detecting timing races, reproducing them with scripts/voice_case.py, testing with FakeLiveClient in CI, and fixing systemic bugs (B-318, B-319, B-320).
Space Channel phone audio learningNew2026-06-29
The Telnyx outbound pacing lesson: bad phone audio with clean browser audio points to chunking/pacing before TTS. Covers 20 ms chunks, absolute deadlines, pre-encoded media, and when to pre-generate Space Channel prompts.
Space Channel content refreshNew2026-06-29
The refreshable Space Channel phone-content design: four-hour cache, dashboard topic summaries, buffered droid playback, and commands for recurring updates.
Methodology & lessons
The noise floor, harden-the-instrument, per-persona decomposition, holdout discipline.