Voice AI Systems Built Beyond the Platform Ceiling

Voice AI platforms are everywhere in 2026 — Retell, Vapi, Bland, and a dozen others let you spin up a phone agent in an afternoon. They work for appointment booking, simple FAQ routing, and outbound confirmation calls. Then the requirements get harder, and the platform that got you to the demo breaks under the weight of the real use case. Our approach is to build custom voice pipelines from best-of-breed components for exactly those cases.

The Platform Ceiling Is Real

Platforms handle the easy demo, then hit their ceiling. The insurance intake call needs to track fifteen data fields across a branching conversation. The medical triage bot needs to recognize drug names that sound like common English words. The payments flow needs PCI-DSS isolation with no card data touching the LLM. These are the use cases where off-the-shelf systems break.

The failure record is well documented. McDonald's spent two years and 100+ restaurants with IBM before ending its drive-through voice AI in June 2024 — the system could not handle accents, background noise, or order corrections. By contrast, Wendy's FreshAI with Google Cloud delivered a 22-second speed improvement by investing in natural language handling for pauses, corrections, and complex customizations. The difference was engineering depth applied to real-world audio.

We architect each pipeline for the latency budget, vocabulary, and regulatory constraints of its specific engagement, an approach detailed in our research on why voice AI has to move beyond API wrappers.

Latency Is the Conversation Killer

The standard pipeline runs speech-to-text, then an LLM for reasoning, then text-to-speech. Each step adds 100–300ms plus network overhead, pushing total round-trip latency to 1–2 seconds — well past the 800ms threshold where trust dies. We engineer latency out at every layer:

• ASR: Deepgram nova-3 delivers sub-300ms streaming transcription with a 5–7% median word error rate in production, versus Whisper's batch-oriented architecture that processes audio in 30-second chunks.
• TTS: Cartesia Sonic produces first audio in approximately 40ms; ElevenLabs Flash v2.5 in approximately 75ms.
• LLM inference — where most latency hides: speculative response generation begins formulating likely responses while the caller is still speaking, streaming the first audio tokens before the full response is generated.

Telephony adds its own hidden latency. Twilio's Media Streams WebSocket connections introduce jitter that never appears in lab testing; their newer ConversationRelay product reduces this overhead, and Genesys AudioHook has similar characteristics. SIP trunk selection, codec transcoding, and jitter buffer tuning each contribute 20–50ms that compound silently. We profile the full audio path from microphone to speaker — not just the AI inference step — because that is where real-world latency lives.

Domain Vocabulary Breaks General-Purpose ASR

General-purpose speech recognition optimizes for conversational English. It handles "I'd like to schedule an appointment" well. It does not handle "atorvastatin 40 milligrams QD," "force majeure clause in section 12.3(b)," or "part number XKCD-4419-REV-C" without help. Custom vocabulary features like phrase boosting exist in most ASR providers, but they are fragile when boosted terms are phonetically similar to common words.

For regulated industries where recognition errors have downstream consequences, our approach is to fine-tune ASR models on domain-specific corpora:

• Medical voice recognition requires training on clinical notes and physician dictations.
• Legal dictation requires exposure to archaic phrasing and citation formats.
• Financial services calls reference ticker symbols and proprietary product names that no general model has seen.

Google Research demonstrated in 2023 that augmenting training data with synthetic accented speech improved recognition accuracy for underrepresented accents without degrading performance on well-represented ones. We apply the same principle to domain vocabulary: augment the training distribution with the exact terms the system will encounter, validated against real audio from the deployment environment.

Conversation State Is an Engineering Problem, Not a Prompt Problem

Simple voice agents feed the full conversation history into an LLM context window and rely on the model to track what has been discussed. This works for short, linear interactions. It fails on complex multi-turn conversations where the caller provides information out of order, corrects previous statements, or the conversation branches based on collected data.

We design structured dialog management that separates conversation state from the language model. Required fields, validation rules, branching logic, and escalation triggers are defined in a state machine the LLM cannot override; the model handles natural language understanding and generation within the boundaries the state machine sets. This means the system can track that the caller already provided their date of birth in turn three even if they mention a different date in turn seven, that an insurance claim requires all fifteen data fields before routing to adjudication, and that the system cannot commit to a price, a deadline, or a coverage determination without explicit authorization.

For healthcare voice AI, this architecture is not optional. HIPAA requires that the system never disclose protected health information to unauthorized parties, which means the dialog manager must enforce access controls at the conversation level — not rely on prompt instructions that the LLM might ignore under adversarial pressure or context window drift, a reliability stance we detail in our research on architecture and reliability in deep AI systems.

The Nuance Migration Nobody Is Talking About

Nuance's on-premise speech stack reaches end of sustaining support around June 2026, and roughly 30% of Nuance's customer base is still running on-premise. Microsoft is pushing migration to Dynamics 365 Contact Center and Azure AI services, and HCLTech launched a "Nuance Migration Factory" for at-scale transitions. But the real challenge is not swapping ASR engines — it is preserving the dialog logic embedded in VXML grammars refined over years of production use.

Enterprise IVR trees encode business rules, exception handling, and edge cases documented nowhere except the VXML itself. A rip-and-replace migration that starts fresh with an LLM-based system will spend months rediscovering edge cases the old system already handled. We approach Nuance migrations as dialog logic extraction followed by architecture modernization: parse existing VXML flows, extract the state machine and business rules into a portable representation, then implement on modern infrastructure. A realistic timeline is 18–24 months for complex contact centers — not the 90-day plans vendor marketing suggests.

Regulatory Compliance Is Not an Add-On

The FCC confirmed in 2024 that TCPA restrictions on artificial or prerecorded voices apply to AI-generated speech. Any voice AI making outbound calls requires prior express consent for informational calls and prior express written consent for marketing calls. Violations carry $500–$1,500 per call in statutory damages with strict liability — no intent required. The one-to-one consent requirement, postponed to April 2026, requires individual consent per seller, closing the lead-generation loophole many voice AI vendors have been operating through.

Beyond TCPA, voice AI deployments also face:

• PCI-DSS when handling payment data — card numbers must never reach the LLM or appear in logs.
• HIPAA for healthcare interactions — BAAs, minimum necessary access, and audit trails.
• State-level rules — Colorado's AI Act (effective June 2026) and Illinois' BIPA.

Our approach is to architect compliance into the pipeline from the beginning: consent capture and recording mechanisms, PCI-scoped audio routing that isolates card data from the AI processing path, call recording consent handling that adapts to the caller's jurisdiction, and audit trails that document exactly what the system said and why.

Build, Buy, or Compose

The build-vs-buy framing is outdated for voice AI in 2026. The real decision is what to compose. Open-source frameworks like Pipecat (from Daily) and LiveKit Agents provide vendor-neutral pipeline orchestration; ASR, LLM, and TTS components can be swapped independently, and telephony connects through standard SIP trunks or WebRTC. The question is where your use case needs custom engineering versus where an off-the-shelf component is genuinely sufficient — and we answer it honestly.

If the use case is appointment scheduling with standard English speakers and no regulatory constraints, a platform at $0.07–0.09 per minute is the right answer, and we will tell you so. Every engagement starts with actual requirements — latency budget, vocabulary complexity, regulatory exposure, call volume projections, and existing telephony infrastructure. An engagement is scoped to design the architecture, select components, build the customizations, and hand over a system the team owns without per-minute vendor dependency.