Voice AI Platform
A multi-tenant platform for AI-powered phone agents
This is a company project. The write-up is intentionally generalized: no client names, call transcripts, or source code appear here. The performance figures below are my own measurements from the work. What follows is my own account of the problem, the architecture, and the decisions I made as the sole engineer.
This is a multi-tenant platform for building and running AI-powered phone agents. Operators configure an agent once (its persona, script, knowledge, and behaviour), and the platform then places and answers real phone calls on its behalf, holding a natural spoken conversation with the person on the other end and recording everything for later review.
Under each call is a real-time voice pipeline: speech is transcribed to text, the text is handled by an LLM that reasons over the agent's configuration and knowledge base, and the reply is spoken back with a text-to-speech voice, all within the tight latency budget a human conversation demands. Both inbound and outbound calls run over SIP telephony, and every call is stored with its transcript, outcome, and latency metrics for post-call analysis.
I built the whole system as the sole engineer over six months: the API, the voice-agent worker, the retrieval layer, the admin dashboard, and the infrastructure that ties them together. That meant owning every decision from telephony and real-time media all the way up to the operator-facing UI.
The platform is split into focused services rather than one monolith. A FastAPI core API owns the domain (tenants, agents, campaigns, calls, and auth) and is the single source of truth the dashboard talks to. A separate voice-agent worker, built on LiveKit's agents framework, is where each live call actually runs: it bridges the SIP audio, drives the STT → LLM → TTS loop, and streams speech in both directions.
Knowledge lives behind its own retrieval service. Documents are chunked, embedded, and stored in a Qdrant vector database, and the agent queries it mid-conversation to ground its answers. Keeping retrieval as a distinct service meant the ingestion side could evolve independently of the latency-sensitive call path.
Background and cross-service work (ingestion jobs, campaign dialling, and other async tasks) runs through a Redis-backed task queue, so the real-time call path never blocks on slow work. PostgreSQL is the durable store for tenants, agents, and call records; the dashboard is a Next.js app for the internal operators.
The operator experience is built into a Next.js dashboard (the console is 31 routes and 107 components) featuring a visual conversation-flow builder (“journeys”) powered by React Flow, letting operators map out complex call journeys. It also handles SIP trunk administration, inbound number routing, campaign management, call analytics charts built with Recharts, and full agent version history, so operators can safely roll back changes.
A dedicated Python test harness sits alongside the core services to validate changes before they hit production, and the backend carries 2,585 tests. It runs automated load tests, SIP trunk ramp tests, dry-run campaigns that exercise routing without placing real calls, and an automated conversation evaluation and grading harness that compares speech-to-text accuracy.
The recurring design tension across all of it was latency. A phone conversation is unforgiving: too much delay between someone finishing a sentence and the agent replying and it stops feeling human. Much of the engineering went into streaming each stage of the pipeline and measuring per-stage latency so regressions were visible instead of anecdotal. The measured results are in the latency section below.
- ConnectAn inbound SIP INVITE reaches LiveKit, which opens a room and dispatches a job to the voice-agent worker. Outbound calls leave a queue, pass campaign and dial-lock checks, and dial out over the SIP trunk.
- LoadThe agent joins the room, loads the tenant’s configuration, prompts and conversation flow, and waits until the SIP leg is actually answered.
- GreetThe greeting plays first, before any caller speech, so the line is never silent on pickup.
- Turn loopVoice-activity detection and streaming STT decide when the caller has finished. Intent guards run, the knowledge base is queried, the LLM replies and calls tools, and TTS streams the answer back. The caller can interrupt at any point.
- EndThe call ends on an end-call tool, a terminal node in the flow, a hang-up, a maximum duration, or a transfer.
- ReviewThe worker saves the transcript and per-stage latency metrics before it leaves the room. The API then grades the call with an LLM to set its disposition and trigger follow-up actions.
One conversational turn, split by stage. These are medians over 1,057 production turns, taken from the per-stage metrics every call stores. The stages add up to 1,318 ms against a measured 1,303 ms round trip, so no time is unaccounted for.
Two other fixes were measured the same way. The first was geography: speech-to-text was resolving to a US endpoint 235–255 ms from the servers in Mumbai. The provider’s EU endpoint serves the same models at 118 ms, so moving there took about 120 ms off every turn.
The second fix was audio quality. On a bad call the telephony mixer that feeds the caller was starving for audio, and it dropped 4.37% of the agent’s own speech. I fixed the mixer and added a short pause at pickup, then measured four live calls:
| Broken call | “Acceptable” call | Mixer fix | + Pickup pause | |
|---|---|---|---|---|
| Buffer starvations / min | 71.0 | 52.9 | 23.5 | 20.8 |
| Stalls over 100 ms | 23 | 3 | 0 | 0 |
| Agent speech discarded | 4.37% | 1.22% | 0.08% | 0.00% |