Calendar/Scheduler Simulator — Report & Findings

Date: 2026-07-02 Audience: Fable 5 (morning review) What: a deterministic simulator + ~120 permutation tests over the RIFF calendar/scheduler offer→book loop (scheduling_v1, as defined in flows/austin_plumbing.yaml), and the real bug the simulator caught.

1. Why this exists

The existing deterministic sim harness (tests/test_sim_harness.py) deliberately excludes the scheduling flows — "they use tool calls." So the calendar path — the most valuable, highest-risk booking logic — had no deterministic permutation coverage; it was only exercised by the flaky, LLM-driven eval. This work closes that gap.

2. The simulator (tests/scheduler_sim.py)

• FakeCalendar — a scriptable fake schedule_event backend. It mirrors the exact session slots the real cal_provider adapter mirrors, because the FSM guards read those slots (codex-verified contract): _booked_event_id (booking_confirmed), _last_booking_error_kind=="slot_conflict" (slot_busy) / other (booking_failed), empty offered → _scheduling_offered_count==0 (no_openings), and the engine-written selected-slot trio (selected_slot_ready). It generates offers on whatever date the engine probes, so it is immune to which clock the runtime uses.
• SchedulerSim — a no-LLM driver: loads the real production flow, enters scheduling_triage, and drives each turn through run_turn → maybe_handle_locked_choice_turn → the real offer engine (maybe_prefetch_offer_slots / maybe_book_selected_slot). Every turn asserts adapter.calls == [] — proving the deterministic path was exercised and the LLM was never touched. Fresh session per case; the fake is the only calendar.

Driver + contract were codex-reviewed and APPROVED before implementation.

3. Coverage — 133 permutation cases (130 pass, 0 xfail, 3 skip)

All deterministic (no LLM), fast (~6s). Determinism note (found + fixed): an early re-run exposed wall-clock flakiness — a same-day 08:00 offer falls inside the runtime's min-advance cutoff and is filtered when the suite runs at 06:xx, shifting the ordinal→slot mapping. The initial "clock-immune" design covered the probe date but not min-advance filtering of today's early slots. Fixed at the root: FakeCalendar returns no slots for the earliest probed date, so every offer lands on a future day past min-advance — verified stable across repeated runs at different times. (This is why re-running tests at different times of day matters.)

4. Finding B-CAL-1 — caught by the simulator, then FIXED and validated

The bug. On the run_turn/cascade path, a same-slot race at the booking write left the dead selection set. The required-tool dispatch books, then its post-book walk chains book_selected_slot → slot_busy → offer_slots, where scheduling_selected_slot_ready (the selection is still set at that instant) immediately re-fires and routes back to scheduling_confirm_slot_hold on the same conflicting slot (~3× then escalate) instead of offering fresh times. The live voice path was unaffected (it cleans up separately in session.py:2124); the bug was bounded — it never fabricated a booking.

Root-causing it took two reverted attempts. Cleanups placed in locked_choice.py::maybe_handle_locked_choice_turn (end-of-function, then after dispatch_required_tools) both still landed at confirm_slot_hold — proving the retry loop happens inside dispatch_required_tools' own post-book walk, so any downstream cleanup is too late. That localized the fix to the exact spot.

The fix (riff/live_turn_controller.py, in dispatch_required_tools):

1. Before the post-book walk — if _last_booking_error_kind == "slot_conflict" and nothing is booked, run prepare_slot_conflict_retry to clear the dead selection, so slot_busy lands the caller on offer_slots for a fresh pick.
2. After the walk — once routed to an offer state with no selection, clear the lingering slot_conflict so it can't clobber the caller's next selection on the rebook. Scoped to an unbooked slot_conflict (non-booking required tools unaffected); a no-op on the live path, which had already cleared the selection.

Validated (fix + prove no regression):

• Calendar sim 125 green: conflict → offer_slots → re-pick → booked; persistent conflict re-offers and never fabricates; happy path intact. The two former xfail(strict) guards now pass; markers removed.
• Targeted scheduling+live suites: identical pre-existing failures with and without the fix (zero new) — proven by a clean-HEAD diff.
• Full suite: 109 failed (same count as baseline), +15 passing. The only delta failure was a flaky telnyx audio-synth test unrelated to scheduling (passes on re-run). codex's stale-_scheduling_offered_count concern was checked and is unreachable in the conflict path (a conflict implies offers existed, so count > 0 and no_openings_for_day cannot fire).

This is the arc that matters for the review: the deterministic simulator caught a real latent bug the LLM eval would have masked, and the same harness then validated its fix — with a full-suite regression diff, not vibes.

4a. B-CAL-2 — a second bug, found by edge-hunting off the fix

Probing conflict/failure edge combinations after the B-CAL-1 fix surfaced a follow-on: two consecutive same-slot races left the caller unable to book. After the first conflict, the _required_tool_fired latch routes the second conflict's booking through maybe_book_selected_slot (not dispatch_required_tools, which the B-CAL-1 fix cleared). That path re-offered fresh slots but left the slot_conflict error lingering, and it then clobbered the caller's next selection on the eventual rebook (booking never completed). Fix: the same cleanup one level over — clear the stale error after the conflict has re-offered (codex-reviewed SAFE; re-checks the current error value per codex's hardening nit). Validated the same way: sim 126 green (test_repeated_conflicts_then_success_books), targeted scheduling+live zero-new, full suite 108 failed / 7622 passed (zero new vs the B-CAL-1 baseline). Two real conflict-recovery bugs, both caught and fixed by the same harness.

4b. Aggregate integration check (whole-repo)

Full-suite run on main with both B-CAL fixes + all new tests integrated: 7,622 passed / 108 failed / 183 xfailed / 30 skipped / 5 errors (8:00). Versus the pre-work baseline (42dd6850: 7,467 passed / 110 failed / 183 xfailed), the net effect of everything landed this session is +155 passing, −2 failing, and crucially zero new failures introduced — every remaining failure/error is pre-existing (unrelated modules: editor_preview, live_models, complex_flows, sim_harness, dashscope, ops_shell, and pre-existing scheduling flow-schema drift), confirmed by clean-HEAD regression diffs on each source change. The two B-CAL fixes were each validated by a targeted scheduling+live diff and a full-suite diff before landing; the shadow slice by a per-slice isolation diff. Method note: run with -p no:faulthandler — a rare native-lib (torch/sentencepiece) transient can otherwise dump on an unrelated test; re-runs are clean.

4c. Scope boundary (what the sim does NOT cover, and why)

The simulator covers the deterministic locked-choice offer→book loop end to end. It deliberately does not cover three scheduling states, and this is a property of the flow, not a missing test:

• scheduling_day_picker / scheduling_time_picker — collect states that need LLM slot extraction to resolve a spoken day/time.
• scheduling_read_back — a confirmation gate that is not a locked_choice, so its yes/no is resolved by the model, not the deterministic resolver.
• scheduling_collect_contact — free-text contact collection (LLM).

Driving these via run_turn with an empty FakeAdapter exhausts the adapter (the LLM would be called) — which is exactly why test_sim_harness.py also excludes scheduling collect states. Verified empirically. These paths belong to the LLM-in-the-loop eval, not a deterministic simulator; forcing them here would produce fragile tests. The offer engine, guards, transitions, and booking handshake — the high-risk logic — are fully covered deterministically.

The other booking module (book_appointment_v1, used by serene_mobile_massage / skyline_apartments / tutorial_book_appointment) was investigated and is out of deterministic scope by design. Its offer_slots is a collect state (LLM slot-extraction), not a locked_choice, so run_turn can't drive the pick — and, structurally, the B-CAL bug class does not apply to it: B-CAL-1/2 were specific to the locked_choice auto-advance re-fire (a stale scheduling_selected_slot_ready bouncing the caller back to confirm). book_appointment_v1's collect-based offer_slots has no such auto-advance guard, so a same-slot race there simply routes slot_busy → fetch_offer and waits for a fresh (LLM-driven) pick. Its booking-race behavior is therefore eval territory, not a deterministic-sim gap.

5. One-line status

The calendar path now has deterministic permutation coverage it never had; 116 cases pass, and the simulator earned its keep on day one by catching a real conflict-retry gap the LLM eval would have masked.