I built production voice AI infrastructure using LiveKit for real-time audio streaming, Twilio for PSTN/SIP telephony, Deepgram for speech-to-text, and ElevenLabs for text-to-speech - powering 50+ deployed AI agents that handle 5,000+ monthly phone calls with under 200ms voice response latency and 99.7% uptime.
Production voice AI is not a demo problem - it is a systems problem. When callers dial a real phone number and expect a human-quality conversation, every millisecond of latency and every dropped tool call erodes trust. At UseCortana, I architected the voice infrastructure that now powers 50+ deployed AI employees handling 5,000+ monthly phone calls with under 200ms voice response latency and 99.7% uptime.
This post explains the architecture, vendor choices, and lessons that transfer to any team building voice agents at scale. If you are comparing approaches, my earlier VoiceCake platform case study covers multi-channel conversational AI from a product perspective - this post goes deeper on the telephony and real-time audio layer.
The production voice stack in 2026
The industry has converged on a cascaded pipeline for tool-heavy agents: streaming STT → LLM with function calling → streaming TTS, orchestrated over a real-time media layer. LiveKit Agents is the open-source framework I used for WebRTC transport, turn detection, and provider plug-ins.
| Layer | Role | My stack |
|---|---|---|
| Telephony | PSTN/SIP bridging | Twilio |
| Real-time media | WebRTC rooms, audio routing | LiveKit |
| Speech-to-text | Streaming transcription | Deepgram |
| Reasoning | Tool use, dialog policy | OpenAI, Anthropic, Google via Vercel AI SDK |
| Text-to-speech | Low-latency voice output | ElevenLabs |
| Orchestration | Durable multi-step workflows | Temporal |
Speech-to-speech models like OpenAI Realtime are compelling for small talk, but production agents that charge cards, book appointments, or query CRMs still default to cascaded pipelines for observability and tool reliability - a pattern echoed in every major 2026 voice AI architecture guide.
How a call flows end-to-end
- Caller dials a Twilio-provisioned number. Twilio bridges the PSTN leg into a LiveKit room.
- Agent worker picks up the job, joins the room, and subscribes to the caller's audio track.
- Deepgram streams partial transcripts as the caller speaks.
- Turn detection - LiveKit's semantic model decides when the caller finished their thought (not just when volume dropped).
- LLM reasons with tool access - calendar APIs, CRM lookups, RAG retrieval over product docs.
- ElevenLabs streams synthesized speech back into the room while the LLM still generates tokens.
- Barge-in - if the caller interrupts, TTS flushes within ~150ms and the new utterance routes back to STT.
The loop repeats until the call ends or transfers to a human agent via automated live call transfer.
Architectural decisions that mattered
Semantic turn detection over naive VAD
Volume-based voice activity detection breaks on filler words, background noise, and elderly callers who pause mid-sentence. LiveKit ships a semantic turn-detection model that reads prosody and lexical cues - combined with Krisp noise cancellation, it handles open-office background chatter that kills naive VAD.
Tune silence thresholds per vertical: sales calls want ~400ms, IVR replacement ~250ms, healthcare intake ~600ms.
Encrypted credential management
Agents access Twilio credentials, CRM OAuth tokens, and customer PII in the same session. Credentials are encrypted at rest, scoped per tenant, and never logged in plain text. This is table stakes for any SaaS deploying agents on behalf of clients.
Regional co-location
Hosting agent workers in the same region as your LLM and TTS endpoints cut median latency more than any model swap. Cross-region hops add 80–150ms per turn - unacceptable when your target is sub-500ms perceived latency.
Live call transfers
Not every call should stay with AI. When sentiment drops, the caller asks for a human, or a tool fails twice, the system warm-transfers to a live agent with context summary - a pattern I also refined on the HVAC voice agent project.
Observability and eval harness
Every turn emits an OpenTelemetry span with ASR, LLM TTFT, TTS first-byte, and tool-call durations. Without turn-level traces, you are flying blind when a model version silently degrades.
I also recommend a nightly eval job that replays 100–500 canned scenarios and grades responses against a rubric - the same discipline I applied during RLHF work on Gemini at Turing.
Results at scale
- 50+ AI employees deployed across client accounts.
- 5,000+ monthly AI-driven phone calls handled in production.
- <200ms voice response latency on the critical path.
- 99.7% uptime across the voice infrastructure.
- Agent setup time reduced from hours to under 15 minutes via the Next.js SaaS dashboard.
When to build vs buy
Managed platforms like Retell, Vapi, or VoiceCake accelerate time-to-market. Build on LiveKit when you need multi-tenant SaaS, self-hosting for compliance, custom turn-taking, or voice cloning at scale.
Related work
- VoiceCake AI platform case study - multi-channel voice and chat agents
- RAG vector memory for AI agents - grounding voice agents with retrieval
- Temporal durable workflows - reliable multi-step agent orchestration
- View all projects · Work together
Common questions
What is the best architecture for production voice AI agents in 2026?
Most production voice agents use a cascaded pipeline - streaming speech-to-text, an LLM with tool calling, and streaming text-to-speech - orchestrated over WebRTC via LiveKit or a comparable real-time media layer. Speech-to-speech models are used selectively for rapport-heavy flows, not as the default.
Why use LiveKit for AI voice agents instead of a managed platform?
LiveKit provides WebRTC media transport, semantic turn detection, barge-in handling, and plug-in support for major STT/LLM/TTS providers without vendor lock-in. It is ideal when you need self-hosting, multi-tenant SaaS, or deep control over the audio pipeline.
How do you achieve sub-200ms voice response latency?
Stream partial transcripts and LLM tokens at every stage, run retrieval in parallel with LLM time-to-first-token, host agents in the same cloud region as your models, and use a fast TTS provider with low first-byte latency like ElevenLabs Flash or Cartesia.
How does Twilio integrate with LiveKit voice agents?
Twilio PSTN/SIP trunks bridge phone calls into LiveKit rooms. The agent worker joins the room, subscribes to the caller audio track, runs the STT-LLM-TTS loop, and publishes synthesized speech back - enabling AI agents to answer real phone numbers.
What observability do production voice agents need?
Turn-level traces with ASR, LLM, TTS, and tool-call timings via OpenTelemetry, plus a nightly eval harness that replays canned scenarios to catch model regressions. Without both, silent quality degradation is inevitable.