← RIFF docs

Session Debug Workspace

One merged debug artifact per call, and a state inspector inside flow-studio — the observability payoff of the "global workspace" pattern, built as derived views over data riff already emits.

design v0 · 2026-07-07 · owner dmar · codex consult 019f3f35 · md source docs/session-debug-workspace-design-v0.md · implementation plan docs/superpowers/plans/2026-07-07-session-debug-workspace.md

1 · Origin and framing

This started from the Transformer Circuits "global workspace" discussion — a small shared space every module can read. Auditing riff against that proposal showed riff already implements the workspace pattern in the superior, derived form: facts in session.slots decoupled from FSM position (types.py:590), missing fields derived per turn (never stored), policy vetoing LLM tool calls (the I-2 confirm-state ban), and the two main slot-write lanes — tool-handler diffs and transition sets/clears — emitted with attribution as slot_write {slot, old, new, kind, writer}. (Not every write: direct session.slots.update/.pop paths still bypass that choke point, which is exactly what UC5 makes measurable.)

What riff lacks is the workspace's observability payoff: one artifact answering "what did the system know, when, and why did it act?" — and a place in flow-studio where an operator sees it. The evidence exists but is scattered across four artifacts, two of which evaporate.

2 · What exists today (verified)

3 · The gaps

Gap 1 — scattered and lossy evidence. "Why did it ask for the address twice?" means hand-joining turns.jsonl + state_trace.jsonl + bus_events.jsonl. And the two most valuable event types — slot_write and transition_verdict — are not durable: they live in the 500-event RAM ring until teardown dumps what remains (phone calls only; web/console/sim sessions get no dump at all). A 40-turn call loses its early slot history precisely when it's the interesting call.

Gap 2 — flow-studio is blind. The editor where authors work shows a state glowing and a transcript — not the live triad (required / filled / missing at that turn), not guard evaluations, not transition verdicts. All of it already exists (get_state_snapshot returns missing_slots_in_state; the state-trace panel renders guards on voice.html) — it just isn't surfaced where authoring happens.

Gap 3 — eval/live schism. Evals compute the richest per-turn record in the codebase, then throw it away. Eval failures can't be scrubbed interactively next to the flow being edited, and there's no single canonical record format to diff an eval run against a live call.

4 · Design principles (constraints, not aspirations)

5 · The design — two phases

Phase A — merged debug view over existing logs (prove the shape)

A1 · Durable slot_write + transition_verdict. Two strings added to _RELEVANT_EVENT_TYPES (state_trace_writer.py:46). Written as events happen — lossless where the teardown ring-dump truncates, and covers non-phone sessions. Values are already redacted at the emit sites. The durable file alone doesn't change atlas output (call_replay.py reads only bus_events.jsonl) — plan Task 1b extends the teardown dump to merge it, deduped by the (seq, ts, type) key.

A2 · /api/debug/session_facts?session_id=X. One endpoint merging, per turn: the turns.jsonl row + required_slots_in_state and missing_required_slots (computed server-side from post_turn_snapshot against the flow's required slots — the same derivation is_state_complete uses; the triad needs the required list, not just the missing list) + slot_diff + guards_evaluated + transition_verdict, from all three event sources deduplicated by the (seq, ts, type) key the state-trace endpoint already uses. Per-state visit aggregation stays on the existing /api/debug/state_trace. The live voice path emits no turn_complete boundary today — plan Task 2b adds that emit; older live sessions group unboundaried events to the first turn (stated fallback). Pure merge function in riff/session_debug.py; thin handler.

{
  "session_id": "voice-1777009875064",
  "turns": [{
    "turn_index": 2,
    "user_utterance": "uh, two months",
    "from_state": "collect_duration", "to_state": "collect_duration",
    "required_slots_in_state": ["service_duration_minutes"],
    "missing_required_slots": [],
    "slot_diff": [{"slot": "service_duration_minutes", "old": null,
                   "new": "<str:8>", "kind": "tool", "writer": "update_slots"}],
    "guards_evaluated": [{"guard": "all_state_slots_filled_and_valid",
                          "result": false, "slots_read": ["service_duration_minutes"]}],
    "transition_verdict": {"winner": null,
      "candidates": [{"to": "confirm", "when": "…", "result": "no"}]}
  }]
}

A3 · flow-studio state inspector. Extend flow-studio's own side panel (decision: do not transplant state-trace-panel.js — it's visit-oriented and styled for voice.html; whole-panel reuse would rot). With a replay/live session attached: the scrubber's turn selects to_state; #spSlots switches from static chips to the triad (filled green, missing amber); new sections render guards (name, ✓/✗, slots_read) and the transition-verdict candidate table. One session_facts fetch on the existing 3-second polling cadence.

Phase B — canonical TurnRecord v2 + eval parity

B1 · TurnRecord v2 builder. log_turn's entry construction becomes build_turn_record(result, ctx, *, state_snapshot, turn_events, slots_before, kind, run_id). New optional fields: schema_version: 2, kind, run_id, missing_required_slots (from the post-transition snapshot), slot_diff, transition_verdict, guards_evaluated (capped at 24).

B2 · Eval sink + replay. flow_eval/driver.py writes each turn through the same builder to logs/evals/<run_id>/turns.jsonl. The turns API gains a tolerant normalized reader (malformed lines skipped, v1 defaults stamped) and an opt-in way to load an eval run into the same flow-studio scrubber.

6 · Use cases — step by step

7 · Why this is useful (grounded)

• The pain is named in the code. Comments cite specific calls (voice-1777009875064), specific bugs (B-181 "booked while nothing was written" is why tool_calls carry results), and buffer-aging failures (B-148: the editor lost live sessions because the ring aged out state_entered). Every one of those investigations becomes a UC1/UC2 replay.
• Eval gates get replayable evidence — every judged failure becomes a scrubbable artifact instead of CI text.
• Write discipline becomes observable (UC5) — unattributed_diff measures the choke-point leak instead of letting it silently rot future records.
• Zero drift risk, zero latency cost — every field derives from data already in memory or on disk.

8 · Risks and deterministic defenses

9 · What we are explicitly NOT building

• A stored workspace object (missing_fields/next_best_action as state) — stored derivations go stale the moment the caller says "actually, it's the toilet."
• A universal slot schema in the runtime — vertical knowledge lives in tree data (Tree DSL contract).
• Numeric LLM confidences — categorical outcomes only (matched / no / shadowed, guard true/false).
• Per-turn LLM reflection calls — the deterministic guards already do this in microseconds, without hallucinated "all clear"s.
• J-lens / activation-level interpretability — needs model internals hosted APIs don't expose; not a runtime debugging tool.

10 · Open questions

1. Should /api/sessions/active drop the voice- prefix heuristic in Phase B for a kind-field filter? (Leaning yes; not blocking.)
2. Eval replay loading UX: query param vs picker in flow-studio — decide when B2 lands.
3. Rotation for state_trace.jsonl once slot_write volume lands (v1 punted rotation; A1 increases volume).