Voice AI Platform
A multi-tenant platform for AI-powered phone agents
This is a company project. The write-up is intentionally generalized: no client names, transcripts, internal metrics, or source code are shown. What follows is my own account of the problem, the architecture, and the decisions I made as the sole engineer.
This is a multi-tenant platform for building and running AI-powered phone agents. Operators configure an agent once (its persona, script, knowledge, and behaviour), and the platform then places and answers real phone calls on its behalf, holding a natural spoken conversation with the person on the other end and recording everything for later review.
Under each call is a real-time voice pipeline: speech is transcribed to text, the text is handled by an LLM that reasons over the agent's configuration and knowledge base, and the reply is spoken back with a text-to-speech voice, all within the tight latency budget a human conversation demands. Both inbound and outbound calls run over SIP telephony, and every call is stored with its transcript, outcome, and latency metrics for post-call analysis.
I built the whole system as the sole engineer: the API, the voice-agent worker, the retrieval layer, the admin dashboard, and the infrastructure that ties them together. That meant owning every decision from telephony and real-time media all the way up to the operator-facing UI.
The platform is split into focused services rather than one monolith. A FastAPI core API owns the domain (tenants, agents, campaigns, calls, and auth) and is the single source of truth the dashboard talks to. A separate voice-agent worker, built on LiveKit's agents framework, is where each live call actually runs: it bridges the SIP audio, drives the STT → LLM → TTS loop, and streams speech in both directions.
Knowledge lives behind its own retrieval service. Documents are chunked, embedded, and stored in a vector database, and the agent queries it mid-conversation to ground its answers. Keeping retrieval as a distinct service meant the ingestion side could evolve independently of the latency-sensitive call path.
Background and cross-service work (ingestion jobs, campaign dialling, and other async tasks) runs through a Redis-backed task queue, so the real-time call path never blocks on slow work. PostgreSQL is the durable store for tenants, agents, and call records; the dashboard is a Next.js app for the internal operators.
The recurring design tension across all of it was latency. A phone conversation is unforgiving: too much delay between someone finishing a sentence and the agent replying and it stops feeling human. Much of the engineering went into streaming each stage of the pipeline and measuring per-stage latency so regressions were visible instead of anecdotal.