Voice AI Platform
A multi-tenant platform for AI-powered phone agents
This is a company project. The write-up is intentionally generalized: no client names, call transcripts, or source code appear here. The performance figures below are my own measurements from the work. What follows is my own account of the problem, the architecture, and the decisions I made as the sole engineer.
This is a multi-tenant platform for building and running AI-powered phone agents. Operators configure an agent once (its persona, script, knowledge, and behaviour), and the platform then places and answers real phone calls on its behalf, holding a natural spoken conversation with the person on the other end and recording everything for later review.
Under each call is a real-time voice pipeline: speech is transcribed to text, the text is handled by an LLM that reasons over the agent's configuration and knowledge base, and the reply is spoken back with a text-to-speech voice, all within the tight latency budget a human conversation demands. Both inbound and outbound calls run over SIP telephony, and every call is stored with its transcript, outcome, and latency metrics for post-call analysis.
I built the whole system as the sole engineer over six months: the API, the voice-agent worker, the retrieval layer, the admin dashboard, and the infrastructure that ties them together. That meant owning every decision from telephony and real-time media all the way up to the operator-facing UI.
The platform is split into focused services rather than one monolith. A FastAPI core API owns the domain (tenants, agents, campaigns, calls, and auth) and is the single source of truth the dashboard talks to. A separate voice-agent worker, built on LiveKit's agents framework, is where each live call actually runs: it bridges the SIP audio, drives the STT → LLM → TTS loop, and streams speech in both directions.
Knowledge lives behind its own retrieval service. Documents are chunked, embedded, and stored in a Qdrant vector database, and the agent queries it mid-conversation to ground its answers. Keeping retrieval as a distinct service meant the ingestion side could evolve independently of the latency-sensitive call path.
Background and cross-service work (ingestion jobs, campaign dialling, and other async tasks) runs through a Redis-backed task queue, so the real-time call path never blocks on slow work. PostgreSQL is the durable store for tenants, agents, and call records; the dashboard is a Next.js app for the internal operators.
The operator experience is built into a Next.js dashboard (the console is 31 routes and 107 components) featuring a visual conversation-flow builder (“journeys”) powered by React Flow, letting operators map out complex call journeys. It also handles SIP trunk administration, inbound number routing, campaign management, call analytics charts built with Recharts, and full agent version history, so operators can safely roll back changes.
A dedicated Python test harness sits alongside the core services to validate changes before they hit production, and the backend carries 2,585 tests. It runs automated load tests, SIP trunk ramp tests, dry-run campaigns that exercise routing without placing real calls, and an automated conversation evaluation and grading harness that compares speech-to-text accuracy.
The recurring design tension across all of it was latency. A phone conversation is unforgiving: too much delay between someone finishing a sentence and the agent replying and it stops feeling human. Much of the engineering went into streaming each stage of the pipeline and measuring per-stage latency so regressions were visible instead of anecdotal. The pipeline is tuned to a 1.3s p50 round-trip. Discarded speech went from 4.37% to 0% through voice-activity-detection and endpointing tuning, and routing speech-to-text regionally cut 235ms to 118ms.