Training Data Methodology

Overview

The goal: compress Claude-quality phone conversation ability into models that run locally for free.

Data Format

Conversation Level

Each conversation is stored as a JSON object:

{
  "id": "train_001",
  "scenario": {
    "problem": "clogged_drain",
    "personality": "default",
    "path": "happy_path"
  },
  "turns": [
    {"role": "agent", "text": "Hi, Smith Plumbing...", "phase": "greeting"},
    {"role": "caller", "text": "Yeah, my sink is backed up.", "phase": "problem_determination"}
  ],
  "booking_details": { "service": "...", "date": "...", "time": "...", "caller_name": "...", "phone": "..." },
  "metadata": { "turn_count": 14, "has_booking": true, "phases_covered": ["greeting", ...] }
}

Turn Level (What the model actually trains on)

Each conversation is decomposed into multiple turn-level prompt/completion pairs. For each agent reply in a conversation, one training example is created:

• Prompt: System prompt + all prior turns (user and assistant)
• Completion: The single agent reply

This means a 14-turn conversation with 7 agent replies produces 7 training examples. The 170 conversations yield 922 training examples in the train split alone.

The Greeting Turn Problem

In phone calls, the agent speaks first. But training templates expect user→assistant ordering. We solve this with a synthetic [phone rings] user turn before the greeting, so every training example follows the user→assistant pattern.

Scenario Matrix (3 Axes)

Each conversation is defined by a combination of three independent axes:

Problem Types (8)

Caller Personalities (7)

Conversation Paths (8)

Splitting Strategy

Why Not 50/50

A 50/50 train/test split wastes training signal. LoRA fine-tuning updates only ~1.5% of model parameters and works well with 500–1000 examples. The test set needs to be large enough to measure quality reliably across all phases, but no larger.

Stratified Split (75 / 12.5 / 12.5)

Stratification: The split is not random. Conversations are grouped by path (the most behaviorally distinct axis), and each path is proportionally distributed across all three sets. This guarantees:

• Every conversation path appears in train, val, AND test
• All 8 problem types are covered in each set
• All 7 personalities are covered in each set

What Each Set Proves

The val set is checked automatically during training. If val loss diverges from train loss after epoch 2, the model is overfitting — reduce LoRA rank from 8 to 4, or increase dropout.

The test set is used ONLY during tournament evaluation. It's the "exam" — touching it during training would invalidate the results.

Phase Balancing

The dataset is intentionally weighted to cover all 11 canonical phases, with targeted generation for underrepresented phases:

Bold rows are the phases that were specifically targeted for additional generation.

Why these targets? The model needs enough examples of each phase to learn the distinct behavior pattern. Greeting and goodbye are structurally simple — the model picks them up fast. Match & propose, emergency escalation, and cancellation require nuanced multi-turn behavior that needs more examples.

Validation Pipeline

Every conversation passes through two validation stages:

Structural Validation (hard gates — must pass)

• All required fields present (id, scenario, turns, metadata)
• First turn is agent (agent answers the phone)
• Minimum turn count: 6 for bookings, 4 for non-bookings
• No forbidden AI patterns ("As an AI", "as a language model")
• All phase labels are canonical (after normalization)

Business Validation (soft warnings — flagged for review)

• Reply length: Agent turns must be ≤50 words (it's a phone call)
• Emergency triage: Emergency calls must reach escalation within 6 turns
• Confirmation completeness: Booking confirmations must use the caller's name
• Booking details: All booking fields (service, date, time, name, phone) must be present
• DIY safety: DIY recommendations for dangerous problems (sewer, water heater, emergency) must also recommend a professional
• Pricing policy: Agent-quoted prices must be within reasonable range
• No premature promises: Agent shouldn't promise specific availability before checking calendar

37 business warnings are mostly incomplete booking_details in edge-case conversations.

File Structure

data/
├── scenario_matrix.json              # The 8x7x8 scenario definitions
├── conversations/
│   ├── batch_01.json                 # Original generation (20 conversations)
│   ├── batch_02.json                 # ...
│   ├── batch_03.json
│   ├── batch_04.json
│   ├── batch_05.json
│   ├── batch_06_hard.json            # Hard cases: hangups, cancellations, angry callers
│   ├── batch_07_hard.json            # Contrastive examples: corrections, interruptions
│   ├── batch_08_match_propose.json   # Phase-targeted: slot negotiation
│   ├── batch_09_hangup.json          # Phase-targeted: caller exits
│   ├── batch_10_cancellation.json    # Phase-targeted: retention and cancel
│   ├── batch_11_summarize.json       # Phase-targeted: booking read-back
│   └── batch_12_emergency.json       # Phase-targeted: safety triage
├── all_conversations.json            # Merged raw (before normalization)
├── all_conversations_normalized.json # Merged with canonical phase labels
└── splits/
    ├── train.json / train.jsonl      # 128 conversations
    ├── val.json / val.jsonl          # 21 conversations
    ├── test.json / test.jsonl        # 21 conversations
    ├── train_mlx.jsonl               # 922 turn-level examples (MLX format)
    ├── val_mlx.jsonl                 # 159 turn-level examples
    └── test_mlx.jsonl                # 145 turn-level examples

Scripts

Regeneration

To regenerate or extend the dataset:

1. Add new conversation files to data/conversations/batch_NN_description.json
2. Run python3 scripts/normalize_and_split.py — normalizes phases, validates, creates stratified splits
3. Run python3 scripts/convert_to_chat_templates.py -i data/splits/train.json -f mlx -o data/splits/train_mlx.jsonl (repeat for val/test)
4. Check phase distribution — if any phase has <30 training examples, generate more targeted conversations

The pipeline is designed to be additive. New conversations are added as new batch files; the normalizer and splitter always re-process everything from scratch.