RIFF Two-Node Release Channels

Date: 2026-07-07
Status: Design proposal with Phase 1 implemented
Scope: M3 development/candidate node, M1 stable/fallback node, phone traffic routing, drain/undrain, promotion, rollback, and node comparison.

Core Idea

RIFF should treat M1 and M3 as two release channels, not just two copies of the same process.

• M3 is candidate: the fast development machine where new code lands first, gets restarted often, and is used for test calls, simulator runs, and canary traffic.
• M1 is stable: the fallback node on the last known-good build. It keeps answering production traffic while M3 is being changed, tested, or rolled back.
• Promotion is explicit: after M3 proves healthy, M1 is updated to the same commit and both nodes can rejoin as stable.
• Comparison is intentional: calls and simulations should be routed to a specific node or replayed on both nodes. Random active-active routing across different builds is a debugging trap.

The deployment question should never be "is M1 drained?" alone. The better question is "which node is stable, which node is candidate, and which traffic is allowed to reach each one?"

Amazon-Style Business Tenets

These tenets are written in the style of strong operating principles: customer backwards, mechanism-driven, and explicit about tradeoffs.

1. Never miss a real call for the sake of a deploy. A caller should always have at least one healthy stable node available before a candidate node is restarted or tested.

2. Make the blast radius visible and small. A candidate build should receive only canary traffic until it passes defined checks. The default path for unknown callers stays stable.

3. Use mechanisms, not heroics. Drain state, release channel, git SHA, active-call count, static-audio freshness, and routing decisions should be machine-readable. Operators should not infer them from logs.

4. Single source of truth for promotion. A node is promoted by commit SHA and mode, not by "whatever is currently running." Promotion requires both code identity and health evidence.

5. Two-way doors stay reversible. Candidate deploys must be easy to roll back: drain M3 and leave M1 serving stable traffic. Promotion to both nodes is a higher bar.

6. Compare before debating. When M3 feels better or worse than M1, run the same scenario against both nodes and compare traces, latency, audio assets, route decisions, and final slots.

7. Operational truth beats intent. Health pages, call exports, and node status must show the node, mode, SHA, routing table version, and drain state. If the UI says both nodes are online, both must be able to accept the traffic they claim to accept.

8. Automate the boring checks. .env syntax, routing JSON, active calls, static-audio audit, and expected phone-number mappings are deployment gates, not manual memory.

Why This Is Good

This gives RIFF a safer, faster release motion:

• Safer live changes: M1 keeps production stable while M3 absorbs risky changes like LAW auth prompts, barge-in changes, or new flow compiler output.
• Faster debugging: a test call can be pinned to M3. If it fails, we know which code answered.
• Real A/B evidence: M1 and M3 can run the same simulator or replay payloads so latency, route quality, and audio behavior can be compared.
• Cleaner rollback: if M3 regresses, drain M3 and leave M1 alone.
• Better confidence before promotion: M1 updates only after M3 has proven the commit under test.

Use Cases

1. LAW Auth Prompt Change

Problem: a LAW prompt says "six-digit" while the SMS sends a four-digit code.

Desired workflow:

1. M1 remains stable.
2. M3 pulls the fix and becomes candidate.
3. Caller ID +15124317196 or the LAW DID test window routes to M3.
4. Operator calls 512-277-7LAW and verifies the spoken prompt.
5. Static-audio audit confirms the LAW WAV matches the current text.
6. Promote the commit to M1.

Why this is good: the bug can be tested live without exposing all callers to a candidate build.

2. Barge-In Latency Comparison

Problem: M3 feels faster than M1, or one node misses "0" while the other routes correctly.

Desired workflow:

1. Run the same LAW or Space Channel scenario against M1 and M3.
2. Compare time-to-first-audio, user-speech-to-route, static audio duration, DTMF/locked-choice decisions, and final state.
3. Mark the finding as code, data, audio, or infrastructure.

Why this is good: the team stops arguing from memory and gets node-by-node evidence.

3. Space Channel Content Refresh

Problem: Space Channel audio is regenerated frequently and can dirty local working trees.

Desired workflow:

1. M3 refreshes content and validates generated audio.
2. M1 is not updated until the content build is committed or deliberately excluded.
3. Promotion script stashes only approved local generated artifacts on M1 before pulling.

Why this is good: local generated files do not silently block deployment or get mixed into unrelated releases.

4. Emergency Rollback

Problem: a candidate build starts dropping calls or looping in the menu.

Desired workflow:

1. Set M3 drained.
2. Confirm M1 is stable and accepting calls.
3. Preserve M3 logs and call exports.
4. Replay the failing call locally before another deploy attempt.

Why this is good: rollback is one command and the failed build stays available for diagnosis.

Design

Release Channels

Each node has a release channel:

The release channel is separate from process health. A node can be healthy but drained, or unhealthy while marked stable. Routing must check both.

Node Modes

Add a small persistent mode file on each node:

/tmp/riff-phone-node/MODE

Allowed values:

stable
candidate
shadow
drained

DRAINED can remain as a compatibility marker, but MODE becomes the richer source of truth. For compatibility:

• MODE=drained creates or honors /tmp/riff-phone-node/DRAINED.
• Removing DRAINED should not automatically mean stable; the node should explicitly transition to stable or candidate.

Health Contract

/healthz should include:

{
  "ok": true,
  "node": "M3",
  "role": "M3",
  "mode": "candidate",
  "release_channel": "candidate",
  "git_sha": "ec34a636",
  "branch": "main",
  "dirty": false,
  "active_calls": 0,
  "drained": false,
  "accepting_live_calls": true,
  "routing_table_hash": "sha256:...",
  "started_at": "2026-07-07T12:00:00Z"
}

The follow-along page should keep the current left/right node indicator, but it should show mode as well:

• left half: M3
• right half: M1
• green: stable and accepting traffic
• yellow: candidate
• gray: drained
• red: unhealthy

Routing Contract

Do not rely on Cloudflare random active-active behavior when nodes run different commits. Use one of these routing levels.

Level 1: Manual Drain

Current behavior:

• drain M1, all new calls go to M3;
• undrain M1, both can answer.

This is safe for simple testing but poor for comparison.

Level 2: Canary Caller IDs

Route known operator caller IDs to M3 candidate while everyone else remains on M1 stable.

Example policy:

{
  "default_channel": "stable",
  "canary_callers": {
    "+15124317196": "candidate"
  }
}

This needs a deterministic ingress/router layer or per-node webhook selection.

Level 3: Canary Dialed Numbers

Route a specific DID to candidate:

{
  "+15122777529": {
    "flow": "legal_intake",
    "channel": "candidate"
  }
}

Useful when testing a single vertical such as LAW.

Level 4: Explicit Test Token

Allow a temporary test flag in the incoming webhook or an admin-set call plan:

{
  "next_call_from": "+15124317196",
  "route_to_node": "M3",
  "expires_at": "2026-07-07T12:15:00Z"
}

This is the cleanest operator experience: "my next call should go to M3."

Promotion State Machine

M1 stable @ old SHA
M3 candidate @ new SHA
        |
        | deploy candidate to M3
        v
M1 stable @ old SHA
M3 candidate @ new SHA, healthy
        |
        | canary checks pass
        v
M1 drained -> pull new SHA -> restart -> health check
        |
        | M1 healthy
        v
M1 stable @ new SHA
M3 stable @ new SHA

Rollback:

M3 candidate fails
        |
        v
M3 drained
M1 stable continues serving

Commands

Add a single node-control command:

scripts/node-mode.sh M1 status
scripts/node-mode.sh M1 stable
scripts/node-mode.sh M1 candidate
scripts/node-mode.sh M1 shadow
scripts/node-mode.sh M1 drain
scripts/node-mode.sh M1 undrain

Add release workflow commands:

scripts/deploy-candidate.sh --node M3
scripts/compare-nodes.py --flow legal_intake --scenario law_auth_4digit
scripts/promote-stable.sh
scripts/rollback-candidate.sh --node M3

Keep scripts/deploy_rolling.sh, but change it to call these lower-level commands instead of carrying all deployment logic inline.

Preflight Gates

Every deploy, drain, undrain, and promotion should run:

bash -n .env
python - <<'PY'
import json, os
json.loads(os.environ["RIFF_DIALED_ROUTING_TABLE_JSON"])
PY
git rev-parse HEAD
git status --short
curl /healthz

Additional routed-flow gates:

• static-audio audit for every routed flow;
• phone-number routing table check;
• active calls must be zero before forced restart;
• node mode must match the intended operation;
• M1 pull must stash or preserve local generated audio artifacts before git pull --ff-only.

Compare-Nodes Output

A node comparison should produce a small report:

scenario: legal_intake/law_auth_4digit
candidate_sha: ec34a636
stable_sha: a5e63178

M1 stable:
  time_to_first_audio_ms: 640
  auth_prompt_text: four-digit
  final_state: menu
  result: pass

M3 candidate:
  time_to_first_audio_ms: 590
  auth_prompt_text: four-digit
  final_state: menu
  result: pass

delta:
  M3 faster by 50 ms
  no route divergence
  no prompt divergence

For a live call, comparison can be done by replaying the captured caller turns through the other node in shadow mode.

Implemented First Cut

Commit 4c64d3b5 implemented the node-mode foundation. This is the current operator contract that Claude, Codex, and human operators should use.

Built now:

• scripts/node-mode.sh manages the persistent mode file /tmp/riff-phone-node/MODE.
• scripts/phone-node-start.sh honors MODE and the legacy /tmp/riff-phone-node/DRAINED marker.
• /healthz reports mode, release_channel, git_sha, branch, dirty, active_calls, drained, accepting_live_calls, routing_table_hash, and started_at.
• /api/phone/nodes and the voice-page node chip show stable, candidate, shadow, drained, and offline states.
• .env shell syntax and RIFF_DIALED_ROUTING_TABLE_JSON JSON syntax are validated before phone-node restart.

Not built yet:

• scripts/deploy-candidate.sh --node M3
• scripts/promote-stable.sh
• scripts/rollback-candidate.sh
• deterministic caller-ID canary routing
• simulator --node M1|M3|both routing

The current system makes node state explicit and machine-readable. It does not yet make live Telnyx/Cloudflare traffic route by caller ID.

Agent Quickstart

Check local node status:

scripts/node-mode.sh local status

Check a named node, delegating through SSH when RIFF_M1_SSH or RIFF_M3_SSH is configured:

scripts/node-mode.sh M3 status
scripts/node-mode.sh M1 status

Mark M3 as the candidate node:

scripts/node-mode.sh M3 candidate

Mark M1 as stable:

scripts/node-mode.sh M1 stable

Take a node out of live tunnel service but leave the server available for local debugging:

scripts/node-mode.sh M3 shadow

Drain a node completely from new tunnel traffic:

scripts/node-mode.sh M3 drain

Return a drained node to stable mode:

scripts/node-mode.sh M3 undrain

Compatibility:

• scripts/phone-node-start.sh --drain still creates DRAINED and now writes MODE=drained.
• scripts/phone-node-start.sh --undrain clears DRAINED and writes MODE=stable.
• DRAINED overrides MODE in /healthz so old operational habits remain fail-safe.

Live-Traffic Semantics

stable and candidate are the only modes where phone-node-start.sh --ensure will start the Cloudflare tunnel connector. shadow and drained keep the phone server reachable locally but do not rejoin the live tunnel.

Simulator Routing Guidance

The simulator should default to M3/candidate once node targeting is added, because M3 is the latest-code validation lane. The planned option shape is:

python -m riff.flow_eval --flow legal_intake --node M3
python -m riff.flow_eval --flow legal_intake --node M1
python -m riff.flow_eval --flow legal_intake --node both

Recommended semantics for the future implementation:

• default --node candidate, resolving to M3 when M3 reports mode=candidate;
• --node M3 hits the M3 phone/service base URL directly;
• --node M1 hits the M1 phone/service base URL directly;
• --node both runs the same scenario against both nodes and stores a comparison artifact;
• refuse ambiguous node selection when both nodes report the same mode but different SHAs unless --node is explicit.

Simulator routing should not depend on the public phone ingress. It should hit the selected node's base URL directly so validation does not disturb live callers.

Current Canary-Routing Gap

The system does not yet support "my caller ID goes to M3; everyone else goes to M1." Until a deterministic ingress/canary layer exists, live testing should use manual drain/shadow operations and direct simulator/replay targeting.

Implementation Plan

Phase 1: Safer Drain/Undrain

1. Done: add scripts/node-mode.sh.
2. Done: store node mode in /tmp/riff-phone-node/MODE.
3. Done: update scripts/phone-node-start.sh to honor MODE.
4. Done: add bash -n .env and routing JSON validation before restart.
5. Done: add mode, git_sha, dirty, and accepting_live_calls to /healthz.

Exit criteria:

• node-mode.sh M1 status reports node, mode, SHA, dirty state, active calls, drained state, and health.
• A malformed .env blocks restart before launch.

Phase 2: Candidate Deploy

1. Add scripts/deploy-candidate.sh --node M3.
2. Restart only M3.
3. Keep M1 stable.
4. Add canary route declaration, even if manual at first.
5. Produce a deploy report with SHA, tests, static-audio audit, and health.

Exit criteria:

• M3 can run a candidate SHA while M1 remains stable.
• Operator can send their next test call to M3 intentionally.

Phase 3: Promotion

1. Add scripts/promote-stable.sh.
2. Pull exact candidate SHA on M1.
3. Restart M1 while drained.
4. Verify M1 health and route table.
5. Set both nodes to stable.

Exit criteria:

• M1 and M3 both report the same SHA and mode=stable.
• Promotion refuses if SHAs differ unexpectedly.

Phase 4: Node Comparison

1. Add scripts/compare-nodes.py.
2. Support simulator scenarios first.
3. Add call replay against the non-answering node in shadow mode.
4. Store comparison artifacts in docs/flow-eval/node-compare/<date>/.

Exit criteria:

• LAW, Space Channel, and property-management scenarios can compare M1 vs M3.
• Reports include latency, route decisions, prompt text, final state, and slot deltas.

Open Questions

1. Should canary routing live inside RIFF phone ingress or outside it at the tunnel/webhook layer?
2. Should M1 ever run candidate, or is candidate always M3 by convention?
3. Should shadow be a full running phone node or just a replay endpoint?
4. How do we expose "my next call goes to M3" in the operator UI?
5. Should generated Space Channel content be promoted independently from code, or always committed as part of a release?

Recommended First Cut

Build the smallest mechanism that changes daily behavior:

1. Done: add node-mode.sh.
2. Done: add richer /healthz.
3. Done: add .env and routing JSON validation before restart.
4. Next: add deploy-candidate.sh --node M3.
5. Next: add promote-stable.sh.

Do not build a full ingress router first. Start with deterministic operator workflow and status visibility. Once the team trusts the mode model, add canary caller-ID routing and node comparison.