RIFF Phone-Line Failover: Active-Active Phone Nodes (M3 + M1)

LIVE as of 2026-07-04

The phone line (riff.chattychapters.com / phone.chattychapters.com) is served by two machines at once: the M3 Pro (primary dev box) and the M1 MacBook Pro. Both are self-healing: each recovers from a reboot (LaunchAgent RunAtLoad) and from a crash (a 120-second "ensure" loop restarts whatever is down). No human action needed.

TL;DR for operators

QuestionAnswer
What failed over?The live phone path only (Telnyx → riff.chattychapters.com → riff.phone.server).
How?Both machines run a cloudflared connector for the same customer-bot tunnel. Cloudflare's edge splits new calls between them and instantly stops using a dead connector.
What if a machine reboots?Its node restarts at login via LaunchAgent com.riff.phone-node; the other machine carries calls meanwhile. A call in progress on the dying machine drops — there is no mid-call handoff.
What if the server crashes?The ensure loop notices within 120s and restarts it.
Where is the M1?david@100.86.206.128 (Tailscale) — note the username is david, not davidmar.
Is it proven?Yes. With the M3's connector killed, public probes returned 200 and the companion WebSockets reconnected through the M1.
Does SMS/bot traffic break?Not while the M3 is up. The M1's connector proxies sms. and bot. hostnames back to the M3 over Tailscale. Those services are not redundant — M3 down = SMS down.

The phone-node role (repo-tracked)

Everything lives in the repo — both machines run the same code, updated by git pull:

Connector config per machine: the script uses ~/.cloudflared/customer-bot.yml if present (M1 — keeps its separate recorder-riff tunnel in config.yml untouched; also proxies sms./bot. back to the M3), otherwise the default ~/.cloudflared/config.yml (M3).

RIFF_PHONE_NODE_WEB=1 in a node's .env (set on the M3 only) makes the same script/watchdog also manage riff.web_server on :8765 — the browser voice-testing surface and the sms.chattychapters.com webhook target. Watchdog restarts run with RIFF_CAL_SMOKE_TEST=0 so unattended restarts don't write calendar marker events.

Logs on both machines: /tmp/riff-phone-node/ (phone-server.log, cloudflared.log, startup.log, launchagent.log).

Health checks (run from the M3)

# Two connectors should be listed (one per machine)
cloudflared tunnel info customer-bot

# Local + peer node health (ok, flow, active calls)
curl -s http://127.0.0.1:8767/healthz
ssh david@100.86.206.128 'curl -s http://127.0.0.1:8767/healthz'

# Public URL answers
curl -s -o /dev/null -w "%{http_code}\n" https://riff.chattychapters.com/healthz

Restart a node by hand

scripts/phone-node-start.sh            # refuses if calls are active
scripts/phone-node-start.sh --force    # drops active calls on THIS node only
# or via launchd:
launchctl kickstart -k gui/501/com.riff.phone-node

Pin test calls to one machine (drain the other)

New code does not reach both machines at once: the machine you restart runs your build immediately; the other stays on last night's main until the 4:15am sync. Since Cloudflare may route a call to either node, drain the node you are NOT testing on:

# from the M3, take the M1 out of the tunnel:
ssh david@100.86.206.128 'cd ~/src/riff && scripts/phone-node-start.sh --drain'
# ...test on the M3 (all live calls now land here)...
ssh david@100.86.206.128 'cd ~/src/riff && scripts/phone-node-start.sh --undrain'

Drain kills only the connector — the drained node's server stays up for local testing. The ensure loop and full restarts respect the drain flag (/tmp/riff-phone-node/DRAINED), so it stays drained until you undrain it.

The panel node light

voice.html shows a node chip above the version chip (bottom-right): green 2/2 nodes when both machines serve the tunnel; amber on any of 1/2 (a node down), · 1 drained, or ⚠ builds differ between nodes. Hover lists each node's build, drain state, and active calls. Backed by /api/phone/nodes (admin-gated), which probes each peer in RIFF_PHONE_PEERS via its public /healthz (now reporting version and drained). Poll interval 30s.

Hard-won gotchas (read before "improving" this setup)

  1. Never keep credentials in ~/Downloads (or Desktop/Documents). macOS TCC blocks LaunchAgent file access to those folders by hanging the open() syscall forever — no error, no prompt. This bit both machines on the same day. The Google service-account JSON lives at ~/.config/riff/ on each machine. Symptom if it regresses: server log stops right after "registered cal_provider" and the process sits in __open (check with sample <pid> 2).
  2. SSH and tmux sessions hide problem #1 (they inherit disk access). Test boot-path changes with launchctl kickstart, not just a shell.
  3. The plist needs AbandonProcessGroup=true or launchd kills the nohup'd server/connector when the start script exits.
  4. Cloudflare pins clients to a connector. Repeated curls from one place all hit one machine — that's normal. Any given call may land on either machine; the split is uneven, not 50/50. Each call is one WebSocket and lives entirely on whichever machine caught it.
  5. The two .env files are a merge, not a copy. The M1 has keys the M3 doesn't (TELNYX_APP_ID, launch tokens, OAuth client pair) — preserved at the bottom of the M1's .env. Backup: ~/src/riff/.env.bak-20260704 (M1).

Known gaps / follow-ups