Making a Multi-Agent LLM API with Python & FastAPI

Running several language models is normal now: a fast small model for quick tasks, a big one for deep reasoning, maybe a vision model for screenshots. Instead of wiring each one into your app separately, put a single FastAPI service in front of them. It picks the model, streams the reply, and gives you one place to handle retries, logging and costs.

By the end of this guide you'll have a working multi-agent router with streaming responses, a task-classifier, and a clean JSON contract your front end can consume.

The plan

1. A classifier decides what kind of task a prompt is (code, chat, vision, long-form).
2. A router maps each task type to a model.
3. The API streams the answer back with SSE so the user sees words appear.

Scaffold the project

mkdir llm-router && cd llm-router
python -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn httpx python-dotenv

Create main.py. We'll keep it small but complete — a real router with streaming.

Classify the task

Before we route, we need to know what we're dealing with. In production this is often a tiny model call; here a deterministic keyword check keeps the guide runnable without a classifier model.

from typing import Literal

Task = Literal["code", "chat", "vision", "longform"]

def classify(prompt: str) -> Task:
    p = prompt.lower()
    if any(w in p for w in ("image", "screenshot", "photo", "picture", "diagram")):
        return "vision"
    if any(w in p for w in ("code", "function", "bug", "refactor", "python", "javascript", "sql")):
        return "code"
    if len(prompt.split()) > 120:
        return "longform"
    return "chat"

ROUTES = {
    "code": "gpt-4o",           # strong at code
    "chat": "gpt-4o-mini",      # cheap and fast
    "longform": "claude-sonnet-5",  # long context
    "vision": "gpt-4o",         # multimodal
}

Pick real model IDs from your own provider. The point is the pattern — one classifier, one routing table — not these exact strings.

Stream the answer

Streaming is what makes the API feel alive. We use server-sent events so tokens arrive as they're generated.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import httpx

app = FastAPI(title="LLM Router")

async def stream_model(model: str, prompt: str):
    async with httpx.AsyncClient(timeout=60) as client:
        # Swap for your provider's streaming endpoint.
        async with client.stream("POST", "https://api.example.com/v1/chat",
                                 json={"model": model, "messages": [{"role": "user", "content": prompt}]}) as r:
            async for line in r.aiter_lines():
                if line:
                    yield f"data: {line}\n\n"
    yield "data: [DONE]\n\n"

@app.post("/v1/ask")
async def ask(prompt: str):
    task = classify(prompt)
    model = ROUTES[task]
    return StreamingResponse(
        stream_model(model, prompt),
        media_type="text/event-stream",
        headers={"X-Model": model, "X-Task": task},
    )

Run it

uvicorn main:app --reload
# in another shell:
curl -N -X POST "http://localhost:8000/v1/ask?prompt=write%20a%20python%20function%20to%20reverse%20a%20string"

The X-Model and X-Task headers let you see exactly what the router chose for each request — invaluable for debugging and for cost attribution.

Where to take it next

• Add a retry layer with exponential backoff when a model 429s.
• Persist every prompt and response to a ledger for cost tracking.
• Use a real classifier model for the routing decision.
• Add authentication and per-client rate limits.

You now have the skeleton of the same architecture that powers real AI products. From here it's a short hop to agents that call tools, and to fine-tuned models that know your domain.