phone-agent-scheduler

What it is

An open-source Python application that acts as an automated receptionist for service businesses (plumbers, electricians, HVAC). It integrates with Telnyx for real-time voice communication, uses local STT/TTS for low-latency responses, and employs a hybrid architecture combining a deterministic Finite State Machine (FSM) for booking logic with LLMs (Sonnet/Gemini) for natural language understanding. It features a 'Flowstate' shim with 8-layer safety nets to prevent hallucinations and ensure brand compliance during customer interactions.

Features

• Real-time voice pipeline via Telnyx WebSocket with sub-3-second response times
• Two-phase commit booking requiring verbal confirmation before calendar writes
• Deterministic 13-state FSM for structured appointment scheduling and business hour enforcement
• Flowstate shim with 19 declarative YAML assistants and 8-layer LLM safety net
• Web UI for chat testing, model switching, and latency monitoring
• After-hours callback queue with JSONL persistence and automated follow-up scheduling

Quickstart

python3 scripts/tour.py

Architecture

flowchart TD
    Caller((Caller)) -->|Voice Stream| Telnyx[Telnyx Voice API]
    Telnyx -->|WebSocket| Server[Phone Agent Server]
    subgraph Server_Process
        direction TB
        Auth[Auth Middleware] --> Router[Request Router]
        Router -->|Chat/UI| WebUI[Flask Web UI]
        Router -->|Voice Session| FSM[FSM Engine]
        FSM -->|Intent Detection| LLM_Intent[LLM Intent Detector]
        FSM -->|Tool Execution| Tools[Tool Registry]
        Tools -->|Read/Write| GCal[Google Calendar API]
        FSM -->|Safety Net| Shim[Flowstate Shim]
        Shim -->|Delegate| LLM_Core[Core LLM Sonnet/Gemini]
    end
    Server -->|Callbacks| Queue[Callback Queue JSONL]
    Server -->|Logs| CallLog[Call Log]

How it's built

Built with Python and Flask, leveraging the `voiceos` kernel for session management and event handling. It uses a two-agent architecture: one for intent detection and conversation flow, and another for tool execution (Google Calendar API via OAuth). The system enforces business rules through a 13-state FSM for scheduling and a declarative YAML-based 'Flowstate' engine for other assistant types. Security is handled via Bearer token authentication and webhook signature verification.

How it runs

sequenceDiagram
    participant C as Caller
    participant T as Telnyx
    participant S as Server (FSM)
    participant L as LLM (Sonnet)
    participant G as Google Calendar
    
    C->>T: Speaks request
    T->>S: WebSocket Audio Frame
    S->>S: STT Processing
    S->>L: Detect Intent & Extract Entities
    L-->>S: Intent: Book Appointment
    S->>S: Check Business Hours & Buffer
    S->>G: Find Available Slots
    G-->>S: Return Slots
    S->>L: Generate Proposal Response
    L-->>S: Natural Language Proposal
    S->>T: TTS Audio Response
    T->>C: Plays Response
    C->>T: Confirms Booking
    T->>S: WebSocket Audio Frame
    S->>S: Two-Phase Commit Verification
    S->>G: Create Event
    G-->>S: Event ID
    S->>S: Log Call & Send SMS Confirmation

How to apply & reuse

Deploy on a server with persistent storage for call logs and callback queues. Configure environment variables for Telnyx API keys, Google Calendar OAuth credentials, and Anthropic/Gemini API keys. Use the provided `scripts/tour.py` for initial setup and validation. Integrate with existing business phone numbers via Telnyx DID configuration pointing to the server's WebSocket endpoint.

At a glance