OpenAI Realtime Voice Agent (Browser WebRTC)

Build a speech-to-speech voice agent in the browser with OpenAI's Realtime API, ephemeral client secrets, and the Agents SDK RealtimeSession.

A browser voice agent that talks over WebRTC: your server mints a short-lived ephemeral key, the page connects a RealtimeSession, and the model handles audio turns, barge-in, and tools in one realtime session.

Steps

Step 1 — Install

In a browser app (Vite or Next.js), install the Agents SDK and Zod v4:

npm install @openai/agents zod

Step 2 — Mint an ephemeral client secret on your server

Never put a project sk- key in the browser. Your backend calls POST https://api.openai.com/v1/realtime/client_secrets with a server API key:

curl -X POST https://api.openai.com/v1/realtime/client_secrets \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"session":{"type":"realtime","model":"gpt-realtime-2.1"}}'

Use the response top-level value field (prefix ek_) as the client secret. Mint a fresh one when needed; tokens are short-lived.

Step 3 — Create the agent and session in the browser

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions: "You are a helpful voice assistant.",
});

const session = new RealtimeSession(agent, {
  model: "gpt-realtime-2.1",
});

await session.connect({
  apiKey: "ek_...(ephemeral key from your server)",
});

Step 4 — Run over HTTPS or localhost

Grant microphone permission, and talk. In the browser the SDK uses WebRTC and configures mic capture and playback for you. On Node it falls back to WebSocket.

Step 5 — Optional tools and safety

Attach tools, handoffs, and guardrails to RealtimeAgent the same way as a text agent. If you identify end users, set OpenAI-Safety-Identifier on the server request that creates the client secret (hashed internal user ID).

Notes

• Prefer the GA flow: no OpenAI-Beta: realtime=v1 header; use session.type and /v1/realtime/client_secrets.
• Standalone package alternative: @openai/agents-realtime with the same RealtimeAgent / RealtimeSession imports from that package.
• Check current model IDs and prices on OpenAI's model and pricing pages before shipping; audio tokens dominate long calls.

Sources (official docs, checked 2026-10-15)

• Getting started with the Realtime API
• Voice Agents Quickstart (Agents SDK)
• GPT-Realtime-2.1 model
• OpenAI pricing
• npm @openai/agents

Failure modes

• Putting a long-lived sk- project key in the browser exposes it; mint ek_ ephemeral secrets on your server via POST /v1/realtime/client_secrets.
• Microphone permission denied or missing HTTPS/localhost blocks WebRTC capture.
• Stale Realtime beta clients still send OpenAI-Beta: realtime=v1 or old event shapes; use the GA client_secrets and session.type flow.
• Audio token cost rises quickly on long sessions; end idle sessions and check current Realtime pricing before going live.

Markdown version: /blueprints/voice-agent.md

Related blueprints