Website Ingest Importer — schema.org → `riff_business_pack_v0`

Status: shipped, live-verified on mfclaw.com. First target: https://mfclaw.com/, a Rhode Island personal-injury firm. Design spec: docs/superpowers/specs/2026-07-08-website-ingest-importer-design.md.

1. What it does, and the two commands to run it

Point this at a small business's public website and get back a working phone menu — without hand-typing the business's name, phone, hours, FAQ answers, or menu into YAML. The importer reads the same structured data the business already publishes for Google (schema.org JSON-LD), never the page's prose, and turns it into a caller-shaped business pack that compiles the normal way. If the site doesn't publish enough structured data, the importer says exactly what's missing and stops — it never ships a menu built on a guess.

Step 1 — snapshot the site:

python scripts/site_snapshot.py https://mfclaw.com/ --out data/site_ingest/mfclaw --fetch auto

Step 2 — extract the profile and build the pack:

python -m riff.business_pack.importers.website data/site_ingest/mfclaw --out packs/generated

The --fetch flag on site_snapshot.py accepts auto (try http, escalate to browser only when should_escalate fires — the default), http (force the cheap tier), or browser (force Playwright, useful when you already know a site blocks plain requests). On the current mfclaw pack this pipeline is fully closed: parse_pack returns 0 errors, 0 warns and compile_pack succeeds — verified directly against packs/generated/mfclaw.pack.yaml as this page was written. (legal_intake, the hand-authored comparison pack, ships with 5 warns.)

2. Why it works: two closed vocabularies

The obvious objection is that every website is different — different layout, different copy, different navigation. That's true, and it's also irrelevant, because the importer never reads any of that. It reads exactly one thing: a block of JSON called schema.org JSON-LD, embedded in the page's <head>.

Supply side. Search engines reward businesses that publish this structured data with "rich results" (the star ratings, hours, and phone numbers you see directly in Google search results). So the SEO plugins almost every small-business site already runs — Yoast, RankMath, and similar — emit it automatically. The business owner never wrote a line of it; their web developer's plugin did, for Google's benefit. mfclaw.com ships this in its <head>:

LegalService.name         "The Law Offices of Michael F. Campopiano"
LegalService.telephone    "+14012883888"
LegalService.email        "mfc@mfclaw.com"
LegalService.address      21 Douglas Ave, Providence, RI 02908
LegalService.areaServed   [State: Rhode Island, State: Massachusetts]
LegalService.knowsLanguage ["en", "es", "pt"]

…plus fourteen Question/Answer pairs: a complete FAQ, written and published by the firm itself.

Demand side. riff/business_pack/schema.py already defines a closed target vocabulary on the other end: eight CAPABILITY_KINDS a pack's menu entries can be (intake_form, message, handoff, content, companion_link, calendar_booking, replicant_ticket_ops, menu_link).

The importer is therefore a mapping between two closed vocabularies — what SEO plugins emit, and what a business pack accepts — not a general web-scraper. It never parses the middle of a page, which is the one part that varies without bound from site to site.

3. Snapshot-first architecture

The pipeline has a strict one-way shape:

fetch (network, impure)  →  snapshot/  →  extract_profile (pure)  →  profile.json
                                                                          │
                                            overlay.yaml (human) ─────────┤
                                                                          ▼
                                                          website_to_pack (pure)
                                                                          │
                                                                          ▼
                                              parse_pack → compile_pack → flow

Once the raw bytes are written to disk, nothing downstream ever touches the network again. Every step after the fetch is a pure function of files already on disk, and every one of those steps is unit-tested offline — a fact structurally enforced, not just promised: test_website_importer_never_imports_the_network is an AST-level check that riff/business_pack/importers/website.py never imports a networking module, the same pattern tests/providers/test_slot_mapper.py::test_slot_mapper_does_not_import_cal_provider uses to keep a different adapter honest. scripts/site_snapshot.py owns every socket.

4. The fetch tiers, and why the browser tier exists

It is tempting to assume a plain HTTP request is enough. It isn't. A plain-user-agent curl against mfclaw.com returns 226 bytes — a ModSecurity Not Acceptable! denial, not the page. The identical request with real browser headers (the ones an actual Chrome tab sends) returns 693,467 bytes, containing both JSON-LD blocks. Real small-business hosting stacks block requests that don't look like a browser; that's not a hypothetical, it's what happened on the very first fixture site.

So the fetcher has two tiers, in scripts/site_snapshot.py:

http — httpx with full browser headers. The default; cheap and fast.
browser — a real Playwright Chromium session. Escalation only, because it's slow and heavier, and only fires when the HTTP tier is provably thin.

The decision to escalate is a pure, unit-tested function in riff/business_pack/importers/website.py — only the I/O around it touches the network:

def should_escalate(status: int, body: bytes) -> str | None:
    """Return the escalation reason, or None to keep the HTTP snapshot."""

It fires on any of three triggers: status outside the 2xx range; len(body) < 20_000; or zero application/ld+json blocks found. The 20KB floor sits well under mfclaw.com's 693KB homepage and well over the 226-byte ModSecurity denial — more than an order of magnitude of headroom on both sides, so one number cleanly separates the two cases actually observed. Escalation is never silent: every trigger is logged and recorded in index.json as that page's tier, because a bare fallback that just tries again without saying why is exactly the shape that hid a real bug for ten sprints in this repo before.

Building the browser tier surfaced its own bug — see defect (a) below: the obvious way to wait for a page to finish loading (networkidle) hangs for a full minute on any real marketing site. That bug was invisible for a while, precisely because the browser tier only runs after the HTTP tier already failed.

5. What the importer refuses to do

The obvious website-to-pack mapping walks straight into features that exist in the pack schema but do nothing at runtime in v0. The importer emits none of them, on purpose.

No LLM in the fact path. The fourteen FAQ answers on mfclaw.com are words the firm itself wrote, reviewed, and published under its own name — including a line like "the statute of limitations in Rhode Island is three years." The importer speaks these answers on the phone verbatim. The rule that makes this safe is absolute:

a model may compress and label; it may never originate a fact.

Nothing is inferred, so nothing can be hallucinated, and the firm has nothing new to indemnify beyond what it already published. Truncating a long answer to a speakable length (320 characters) is done deterministically, on a sentence boundary — never by asking a model to shorten it. For a law firm this is a bar-advertising constraint before it is a quality one: the phone is legally safer when it says exactly what the website already says, and nothing else.

No submit: block. riff/business_pack/schema.py:855 accepts a submit: block but ignores it: no generic submit tool is registered in the runtime (record_issue is a state name used by other flows, not a callable tool), so a pack declaring submit: gets a loud submit-ignored warn finding at build time. The importer omits it entirely rather than implying a claim goes somewhere it does not. Confirmed facts persist via slots and the RunRecord.

No screening_gate. riff/business_pack/schema.py:1157 accepts a screening_gate's tool and blocked_line fields "for forward compat but unused," and grep proves neither is read anywhere else in the codebase. riff/business_pack/compile.py:3688 compiles the gate into a state with speech_act: inform and exactly one transition, when: always — it announces a line, then advances unconditionally. It is not a gate; it's a checkpoint that never checks anything. areaServed: [Rhode Island, Massachusetts] reads like a screening gate — the obvious mapping is "block callers outside RI/MA" — and the resulting pack would cheerfully take a claim from Ohio anyway, because the gate never actually blocks anyone. So the importer maps service_area[] to a hidden, spoken content capability instead: a line the caller can ask to hear ("We serve Rhode Island and Massachusetts."), never a gate that pretends to enforce anything. It carries no DTMF digit and announce: false (see §6), so it never interrupts the menu greeting either. The build now also closes this trap for every other pack: a new screening-gate-field-ignored warn finding fires on any pack — ingested or hand-authored — that declares tool or blocked_line, symmetric with submit-ignored. It fires on packs/legal_intake.pack.yaml today (that pack's warn count is 5, up from 3 before this lint existed).

Those fourteen FAQs exist because Google rewards FAQPage markup with rich results. That's SEO-shaped information architecture, tuned for a search results page. The person calling a personal-injury firm was, quite plausibly, in a car accident this morning. They don't want "press 7 to hear about pure comparative negligence." A phone menu holds about five options before a caller gives up; a faithful transcription of the site's FAQ index would build a hostile phone tree instead of a helpful one. So the menu is hand-ranked in overlay.yaml, not derived from the site's own navigation.

MFC Law's menu, once ranked — verified by compiling packs/generated/mfclaw.pack.yaml directly:

DTMF	Capability	`kind`	Source
`1`	Start a new claim	`intake_form`	contact-form fields
`2`	Existing case or status	`message`	—
`3`	Fees and costs	`content`	FAQ “How much does it cost to hire a personal injury lawyer?”, verbatim
`4`	Something else — leave a message	`message`	safe-decline terminal

Plus fourteen hidden, voice-only intents (13 FAQ answers + service_area) — reachable by saying an alias, never recited in the greeting. The compiled greeting is exactly:

"Here's the menu. Press 1 — Start a new claim. Press 2 — Existing case or status. Press 3 — Fees and costs. Press 4 — Something else — leave a message."

Digits must start at 1 (an optional leading "0" is allowed before it). riff/business_pack/schema.py:1021-1024 computes the expected sequence as [str(n) for n in range(1, len(actual) + 1)] and rejects anything else at build time with a dtmf-not-sequential error — a menu can't ship with digits 2, 3, 4 and no 1.

Note there is no handoff capability on this menu. handoff means a live warm transfer, and MFC Law has no forwarding number to transfer to. Dropping it isn't a free choice, though — it forces capability 4 to be a message. The schema defines SAFE_DECLINE_KINDS = frozenset({"message", "handoff"}): whatever a caller lands on after declining the consent gate must be a capability that can't loop back into the gated path. With handoff off the table, message is the only option left, so the menu is required to have one.

Hidden intents: `announce: false`

See defect (e) for the incident that produced this field. In short: a Capability now has announce: bool = True (riff/business_pack/schema.py). Setting it False makes a hidden intent — reachable by voice, never recited in the menu greeting. Routing is untouched, because the announce state carries only a spoken line while the menu state's locked_choice carries the aliases — two different states, both built from the same capability list. The website importer marks all 13 voice-only FAQ capabilities, and the service_area capability, as hidden.

The eight practice areas are not capabilities

Car accidents, slip & fall, dog bite, motorcycle, premises liability, product liability, pedestrian, and Uber/Lyft are not eight menu entries, and they are not slot options either — schema.py has no options/enum mechanism on slots; matter_area is a plain {type: string}, and legal_intake models its own matter stage as free text too. So each service does two things instead: its label becomes an example in the intake form's prompt ("What happened? For example: car accident, slip and fall, dog bite…"), and its aliases merge into the intake capability's own alias list, so "rear ended", "bitten by a dog", and "uber" all route straight to digit 1. A caller doesn't want to hunt through a phone tree for their practice area; they want to say what happened and be routed.

This is also the one place the site's structured data runs out. The observed JSON-LD @types on mfclaw.com are LegalService, PostalAddress, State, Person, WebSite, SearchAction, ImageObject, Question, and Answer — there is no Service or OfferCatalog anywhere in it. So services[] (the list of practice areas) can't come from the profile at all; it comes from overlay.yaml, hand-authored. The overlay isn't a workaround here — it's the place where facts the site never typed get stated by a human, on the record.

7. Going live

Ingest produces a pack. It does not produce a phone line. Those are two different kinds of artifact, made by two different steps:

A build artifact is anything the importer writes: profile.json, packs/generated/mfclaw.pack.yaml, the compiled flow. Reproducible from a snapshot; safe to regenerate any time. A cold rebuild of the mfclaw pack is byte-identical to the checked-in one except for the generated_at header line — enforced by test_import_is_deterministic.
A deploy artifact is a decision a human makes once, outside the importer: which real phone number this business answers on.

The generated pack therefore omits business.phone entirely. The number is bound later, by scripts/deploy_business.py:

# deploy_business.py:225-229
declared_phone = str((raw.get("business") or {}).get("phone") or "")
if declared_phone and declared_phone != did:
    _step("pack phone matches --did", False,
          f"pack says {declared_phone}, deploying to {did}")
    return 1

An empty phone deploys cleanly to any --did you give it. A non-empty phone pins the pack to exactly that one number — deploying it anywhere else fails this gate on purpose, so a redeploy can't accidentally land a pack on the wrong line. Three of the repo's thirteen packs declare business.phone today (armadillo_ink, legal_intake, property_management), and each is pinned to the number it's actually deployed on. The other ten leave it empty — including two more that are also live right now (space_channel_widgets, electrical_intake), because declaring the phone is optional even after go-live; it only matters once you want the pin. A placeholder number would be worse than an absent one: it doesn't skip the gate, it fails it, the moment it doesn't match --did exactly.

Order of operations, once the pack compiles clean:

Register the number. Buy or port a DID in Telnyx, attach it to the connection and messaging profile. 10DLC registration belongs to the number, not the business.
Deploy. .venv/bin/python scripts/deploy_business.py packs/generated/mfclaw.pack.yaml --did +1… — this runs the gates, builds static audio, walks both welcome branches, writes the DID into RIFF_DIALED_ROUTING_TABLE_JSON in .env, restarts the phone node, and preflights DID→org routing across all lines. On a keyed build, the gate for this pack passes clean: 0 FAIL, 0 [ERROR], and menu-sweep verify: all flows sweep-ready.
Record the binding — recommended, not required. Once deployed, you may write that DID into the pack's business.phone. From then on, the gate refuses to deploy this pack anywhere else — that's the point, if you want the pin. Two live packs (space_channel_widgets on +13215596277, electrical_intake on +15122777311) skip this step and work fine; the cost of skipping is only that nothing stops a later redeploy to the wrong number.
Point the website at it, last. Update the firm's published number only after a real call has verified the line works.

Step 4 is reversible; step 3 is the tripwire. Do them in that order — not first, in case the line isn't actually working yet. Note that profile.json's own business.telephone (+14012883888, the firm's real published line) is never the pack's phone. It's provenance only: proof of where the facts came from, not where the flow answers calls.

8. Seven defects this build found, and what each taught

Every defect below survived until the real thing was run — a live fetch, a real compile, a real TTS render, or reading the spoken output aloud. None of them was hypothetical by the time it was found; each has a measured number attached.

(a) The browser tier hung

wait_until="networkidle" never fires on a site with trackers. Measured on mfclaw.com: networkidle → 60-second timeout; domcontentloaded → 1.5s, HTTP 200, both JSON-LD blocks present; load → 2.3s. The browser tier only runs after the HTTP tier already fails, so this hang was invisible until someone actually forced the escalation path. The fix, in scripts/site_snapshot.py's _fetch_browser: wait for domcontentloaded, then wait up to 5 seconds (advisory, not fatal) for a script[type="application/ld+json"] element to attach — covering the client-rendered case that's the browser tier's whole reason for existing, and logging a warning rather than failing if none shows up.

(b) One entity, two nodes

mfclaw.com emits two business nodes sharing one @id (https://mfclaw.com/#organization): a Rank Math stub (name without "The", logo, openingHours) and a hand-authored LegalService (name, telephone, email, address, areaServed, knowsLanguage). A shared @id means a shared entity in JSON-LD, so _merge_business_entities in riff/business_pack/importers/website.py merges them in document order, later non-empty value winning — never picks one. Picking "the complete node," even a defensible-sounding heuristic, would have silently dropped openingHours, because it lives only on the stub.

(c) Traversal order inverted across nesting keys

_ld_nodes used to push a node's @graph, mainEntity, and itemListElement children onto one LIFO stack in three separate stack.extend() calls — so the last key processed popped first, inverting document order across keys even though each key's own children were internally correct. Because the merge rule in defect (b) is "later non-empty value wins," two same-@id nodes reached through different nesting keys would have merged backwards, silently choosing the wrong field value on any conflict. The single-key ordering test passed the whole time, which is exactly why this survived: the bug only shows up when nodes arrive via different keys. The fix gathers all three keys' children into one combined list, in key order, before pushing it (reversed) once.

(d) Aliases are a pack-global problem

riff/business_pack/schema.py unions every match surface (aliases + DTMF digits) per menu_group and raises the error finding menu-token-collision when a token is claimed by two capabilities — parse_pack rejects the whole pack. A naive per-question alias generator produced 11 collisions across MFC Law's 14 FAQs: injury was claimed by six capabilities, personal by five, because every question is about personal injury. A function that only sees one question at a time cannot know that. So alias assignment moved into website_to_pack's _assign_faq_aliases, which sees every capability at once, reserves every overlay-declared alias and DTMF digit first (the human's taste always wins), and drops greedy words — generic English an ordinary caller says about anything, the same posture as schema.py's _GREEDY_URGENCY_ALIASES. Without that filter, words like after, still, which, types, and valid become routing tokens, and "after" would send a caller straight to the car-accident FAQ.

(e) A 56-second menu

riff/business_pack/compile.py's _menu_prompt_and_controls (around line 2330) appends "Or say <alias>." for every capability without a DTMF digit. legal_intake has exactly one such capability, so nobody noticed. The website importer generates thirteen. The mfclaw greeting compiled to 827 characters, 140 words — about 56 seconds of speech reciting every FAQ title before the caller could say anything — and its TTS render failed outright. The fix: Capability gains announce: bool = True; setting it False makes a hidden intent (see §6). Routing is unaffected, because the announce state carries only the spoken line while the menu state's locked_choice carries the aliases — two different states built from the same capability list. Measured, on the actual compiled pack: 827 → 150 characters, 140 → 32 words, routing table unchanged. Default is True, so every pre-existing pack compiles byte-identically.

(f) The build gate guessed

riff/business_pack/__main__.py's build command used to capture the audio renderer's stderr, discard it, print only the last line of stdout, and print "FAIL (audio render — is GEMINI_API_KEY set?)" on any nonzero exit. The key was live the whole time (HTTP 200, 50 models available). The real summary was rendered=1 skipped=14 errors=1, and the single error was menu_full_announce — defect (e), the 56-second greeting. It now prints every [ERROR] cue name, the last five lines of the renderer's actual stderr, and the process exit code, instead of one generic guess.

(g) Silent data loss on id collision

FAQ capability ids were the question's slug truncated to 60 characters, deduped on that id. Two distinct questions sharing a 60-character prefix collapsed into one, and the second vanished with no warning. This wasn't hypothetical: mfclaw.com already publishes a question whose id lands exactly on that boundary (how_can_i_contact_the_law_offices_of_michael_f_campopiano_if); a sibling question differing only after character 60 would have silently erased it. The fix dedupes on the full question text instead (an identical question repeated really is a duplicate, and is still dropped) and disambiguates colliding-but-distinct ids with a deterministic _2, _3 suffix in document order.

Every one of these seven lived in a path that looked covered — tests were green in every case. Each was found only by running the real thing: a live fetch against mfclaw.com, a real compile of the generated pack, a real TTS render with a live API key, and by reading the spoken greeting aloud instead of just checking that it compiled.

Two more, found the same way

A stray space before punctuation. Stripping inline <strong> tags out of the FAQ HTML left a space before the following punctuation mark, so digit 3 spoke "you pay nothing upfront . We work on a contingency fee basis , meaning…". Three of the fourteen answers had this, including the one promoted to digit 3. The fix lives in _clean (riff/business_pack/importers/website.py) and is whitespace-only — no word is added, removed, or reordered, so the "spoken text is a verbatim prefix of the published answer" invariant still holds.

9. Never clauses

Never let a model originate a fact. Facts come from JSON-LD or the overlay.
Never emit spoken text that is not traceable to published site copy or the overlay.
Never fall back silently. Log the trigger; record the tier; or re-raise.
Never give legal advice or predict outcomes (inherited from the pack system_prompt).
Never emit a menu without a message safe-decline terminal.
Never write a pack that does not parse_pack and compile_pack clean.
Never check raw scraped HTML into git. The sha256 in index.json is the provenance.
Never write business.phone into a generated pack. deploy_business.py --did binds it. The firm's published telephone is provenance, never the line the flow answers on.
Never emit a submit: block or a screening_gate. Both are inert in v0. A gate that announces and always advances is not a gate; areaServed is spoken content.

10. Code map and test matrix

scripts/site_snapshot.py                       # two-tier fetch → index.json; the only module
                                                 # that touches the network
riff/business_pack/importers/website.py         # extract_profile + website_to_pack (pure)
data/site_ingest/mfclaw/                        # index.json, profile.json, overlay.yaml;
                                                 # raw pages/ gitignored
packs/generated/mfclaw.pack.yaml                # generated pack: 0 errors, 0 warns
tests/business_pack/test_website_importer.py    # 48 tests, offline, no network in any test
scripts/deploy_business.py                      # existing; binds business.phone at deploy time

The importer suite is 48 tests covering U1–U23 from the design spec, plus the seven defects above and the stray-space cleanup — each defect has a named regression test (for example test_same_id_nodes_merge_in_document_order_across_nesting_keys for defect (c), test_hidden_intents_stay_routable_and_shorten_the_greeting for defect (e), test_distinct_questions_with_colliding_slugs_are_both_kept for defect (g)). Adding this suite took the full business-pack test count from 373 to 423 passing, with zero new failures elsewhere.

Full design rationale: docs/superpowers/specs/2026-07-08-website-ingest-importer-design.md. Agent-facing quick reference: .claude/skills/ingest-website/SKILL.md.

Website Ingest Importer — schema.org → riff_business_pack_v0