Open-Source LiveKit Chained Voice Pipeline: Clone, Add Keys, Run a Booking Agent

Teams learning voice AI often ask for a minimal but real template: not a slide deck, not a hosted SaaS — something they can clone, plug keys into, and hear a tool-calling agent on a LiveKit room.

I published two public repos that do exactly that:

• VoiceAI-Backend — FastAPI control plane + LiveKit agent worker
• VoiceAI-Frontend — Next.js browser UI that joins the room and plays remote audio

Together they implement a chained (cascaded) voice pipeline for a restaurant booking persona — Luma Bistro / host Maya Rivera.

What “chained modeling” means here

Browser (Next.js)
  → POST /api/v1/livekit/sessions  →  FastAPI :8080
  → LiveKit Cloud room
       ↑ worker: Deepgram STT → OpenAI LLM + tools → ElevenLabs TTS
       tools → HTTP → same backend (mock /availability + /reservations)

Each hop is explicit. That is why assessments and teaching favor cascades: you can measure STT latency, tool success, and TTS time separately. It is also why cascades feel slower than speech-to-speech (Hume, Cartesia, etc.) in production sales calls — I discuss that trade-off in LiveKit vs Retell and the Use Cortana voice architecture post.

Clone and run

1. Backend (VoiceAI-Backend)

git clone https://github.com/HarrixK/VoiceAI-Backend.git
cd VoiceAI-Backend
python3 -m venv venv
source venv/bin/activate   # Windows: .\venv\Scripts\Activate.ps1
pip install -r requirements.txt
cp .env.example .env

Set at least:

• LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET
• OPENAI_API_KEY
• DEEPGRAM_API_KEY
• ELEVENLABS_API_KEY

Then:

python main.py
# API:   http://0.0.0.0:8080
# Docs:  http://localhost:8080/docs
# Health: GET /api/v1/health

With START_LIVEKIT_WORKER=1 (default in the example), the API spawns the LiveKit worker for you.

2. Frontend (VoiceAI-Frontend)

git clone https://github.com/HarrixK/VoiceAI-Frontend.git
cd VoiceAI-Frontend
npm install
cp .env.local.example .env.local
# NEXT_PUBLIC_LUMA_API_BASE=http://localhost:8080
npm run dev

Open http://localhost:5173 → Start call → allow mic.

Expect status roughly: creating session → connected → agent joined → agent speaking. A 2:00 call timer starts when remote agent audio plays (matches backend MAX_CALL_SECONDS).

Reset seed data between scenarios:

curl -X POST http://127.0.0.1:8080/admin/reset

Full smoke steps and layout notes live in each repo README — frontend · backend.

What the agent demonstrates

This is a demo / assessment stack (in-memory mock reservations, not a production deploy), but the voice patterns are real:

• Confirm-before-write — tools require confirmed=true before mutating bookings
• Idempotency-Key + duplicate fingerprints — safe retries on create
• 503 once then recover — temporary availability failure with client retry
• Idle nudges — if the user is quiet ~4 seconds, the agent prompts again
• 2-minute hard cap — keeps demo costs and sessions bounded
• Mock handoff — spoken transfer line via POST /handoff

Automated T1–T7 scenarios (create, unavailable time, barge-in party size, modify, cancel, API failure, duplicates) are documented in the backend README. Offline pytest hits the mock API in-process; the live tool harness needs the server running.

When to use this vs speech-to-speech

I chose the cascade because the assignment required chained modeling. In client work I often migrate cascades to Hume speech-to-speech once latency shows up in conversion — while keeping the same studio and Twilio wiring. See production voice at Use Cortana.

Repo map

Assessment docs (architecture answers, evaluation results, AI tools disclosure, handover) ship with the frontend repo.

Related work

• Hire a LiveKit voice AI developer
• LiveKit vs Retell vs custom stacks
• Calling agent MVP checklist
• Use Cortana: LiveKit cascade → Hume
• Hire Voice AI · Contact