Overview › Operations › Post-call dream loop

Post-Call Dream Loop

How RIFF should turn one real phone failure into targeted simulations, judged A/B/C variants, and a promotion decision that can be replayed.

The rule One phone run should teach the simulator. The simulator should test fixes before another real caller pays the cost. Real OTA calls still require explicit human GO.

Review the run

post_call_analyze.py writes deterministic facts, judge output, and candidate fixes.

Compile scenarios

A future compiler turns the failure evidence into targeted synthetic callers.

Test variants

Baseline and A/B/C run the same replay, flow-eval, and observable A/B tests.

Promote with proof

The winner replaces the original only after evidence, tests, review, and human approval.

What exists today

Tool	Role in the loop
`scripts/post_call_analyze.py`	Builds the per-session review bundle under `logs/post_call/<session_id>/`. It now writes `review_facts.json`, `judge.json`, and `dream_fixes.json`.
`riff/flow_eval`	Runs persona-driven simulations, optional LLM-as-judge scoring, repeat-N sweeps, and ReportCard JSON/Markdown.
`scripts/replay_session.py`	Replays a live call from `logs/turns.jsonl` to find state, slot, tool-call, or fabrication drift.
`scripts/replay_eval.py`	Replays recorded eval transcripts against current code to catch regressions after a change.
`riff/abtest.py`	Scores scripted A/B runs from observable facts: completion, turns, slot fill rate, escalation, and unanswered questions.
`scripts/nightly_review_cycle.sh`	Runs judge, audio review, recording coverage, DB ingest, and failure-mode summaries for recent calls.

The artifact handoff

The first concrete link is now the post-call bundle. The dream loop should consume these files, not scrape prose:

logs/post_call/<session_id>/
  review_facts.json     # deterministic facts: latency, repeats, empty agent turns, timeline waits
  judge.json            # LLM-as-judge scores and proposed fixes when --judge runs
  dream_fixes.json      # normalized candidate fixes with evidence and validation text
  index.json            # UI pointers to the above

Baseline vs A/B/C

The variant runner does not exist yet. Its safest shape is separate git worktrees, one per universe:

runs/dream_loop/<run_id>/
  scenario_pack.json
  hypotheses.json
  baseline/report-card.json
  variant-a/report-card.json
  variant-b/report-card.json
  variant-c/report-card.json
  comparison.json
  decision.md

Each arm must run the same flows, personas, repeat count, judge settings, replay corpus, and scenario pack. A candidate cannot win by changing the measurement instrument.

Promotion discipline

Hard fail deterministic replay regression, lower completion, missing event evidence, fabrication, or service artifact loss.
Hard fail any real OTA action without explicit human GO.
Candidate win target failure disappears in the synthetic scenario pack and judged/deterministic scores do not regress elsewhere.
No auto-merge the loop recommends a winner; normal review and commit discipline still lands the change.

Current gaps

The missing pieces are explicit: a scenario compiler for dream_fixes.json, an isolated worktree runner for baseline vs A/B/C, a comparison/adjudication report, and a promotion command that stays blocked until review evidence exists.

Switchboard note On 2026-06-29 the local Switchboard service at 127.0.0.1:9102 was unreachable, so this pass used direct Claude Code and Antigravity CLI calls instead of broker-assigned tasks.

Source note: docs/codex/2026-06-29-phone-simulation-dream-loop.md, scripts/post_call_analyze.py, riff/flow_eval/, scripts/replay_session.py, scripts/replay_eval.py, and riff/abtest.py.