Cosmo voice calls — research + early plan

Findings, not yet a build brief. Sun 3 May 2026.

One sentence: a /call command mints a tokenised URL → you open it in a browser → it feels like a call → Cosmo on the other end is the same Cosmo you already talk to in Telegram, but voice-tuned.

The interesting word here is token. The token doesn't just authenticate the call — it resolves to a session server-side. The session knows which Telegram chat (or which Claude Code cwd + branch) the call was initiated from, the recent turns, the project context, the memory scope. So when you `/call` from your H2OS chat mid-conversation about a dispenser, Cosmo on the call already knows.

This means a separate decision is forced into the open: what is a "session" in Cosmo, exactly? The codebase already overloads the word three ways. We'll get to it (§4).

OpenClaw the project doesn't ship a voice mode in core. The community add-on is Purple-Horizons/openclaw-voice. Worth knowing the shape:

What's worth borrowing from any of these:

• Silero VAD in the browser via @ricky0123/vad-web. Runs as ONNX in an AudioWorklet, ~1ms per frame. onSpeechStart is the barge-in signal; onSpeechEnd finalises the user turn.
• WebSocket TTS over HTTP TTS when the provider supports it — text-stream in, audio-stream out, no sentence buffering, prosody preserved across chunks.
• Sentence-boundary streaming from LLM tokens to TTS, so the user starts hearing Cosmo before Cosmo is done thinking.
• Filler-while-tool-runs — when a tool call begins mid-turn, emit a short "looking at your calendar..." either via the prompt or via pre-baked audio.

What we explicitly are not borrowing:

• OpenClaw's brain. Cosmo's brain stays.
• OpenClaw's local Whisper. We already pay for OpenAI; we're not running a Whisper server. (Streaming STT provider is a separate decision — see §6.)
• LiveKit / Pipecat as a runtime dependency. We're a single-user agent; the framework overhead doesn't earn its keep at our scale. We copy the patterns, we don't adopt the framework.

This is important to document because it changes what's "new build" vs "wire it up differently."

The existing voice path

• /voicemode (bot.js:1763-1773) — toggles a per-user in-memory flag. Not persisted. Only changes the output mode.
• Inbound voice notes (bot.js:1300-1342) — Telegram .ogg → OpenAI Whisper API → text → existing request queue.
• Outbound voice replies (bot.js:1058-1087) — full text → OpenAI tts-1 with voice nova → .opus → Telegram sendVoice.

The existing path is file-based and half-duplex. Whole utterance up, whole utterance down. Zero streaming. Whisper API is batch-only — no streaming endpoint exists. This is fine for voice notes, fatal for a call.

One bit of leftover code worth noting: src/voice/ is a half-built Python "Hey Cosmo" wake-word menu-bar daemon. Parked, never wired up. Different problem (text injection into focused app), not relevant to /call.

What's reusable for v1

The token-resolves-to-a-session premise forces a question we've been avoiding: what is a session? The codebase overloads the word at least three ways already:

1. Claude Agent SDK session — the sessionId string returned by query(). Stored as claudeSessionId on the chat doc.
2. Telegram chat state — the chats Firestore collection, with backwards-compat aliases getSession/saveSession in db.js:69-72.
3. Claude Code transcripts — JSONL files in ~/.claude/projects/<slug>/, each with their own sessionId.

A web call has nowhere to live in any of these. It needs at minimum a doc somewhere that holds: which originating context, when it started, which Cosmo session ID is active, optionally a name.

Two paths fall out of the research:

And there's the naming problem. Adding sessions as a fourth meaning of "session" is asking for grief. Calling them conversations or threads would disambiguate. The user's call: do we colonise the word back, or pick a new one?

Naming the conversations

Whatever we call them, they need human-readable names. Sidebar showing abc123 · 2 days ago is unsearchable. Showing "Memory v2 dashboard work · 2 days ago" is not. Three options:

• User-set — /name <text>. Same pattern as /project. Always available, always overrideable. Default fallback to the auto-name if user hasn't set one.
• Auto-named — Cosmo titles the conversation after the first 2-3 turns. Same trick ChatGPT does.
• Hybrid — auto-suggest after first few turns, user can override anytime. Probably the right answer.

The Claude Code session-start hook

Claude Code SessionStart hooks receive a JSON payload with session_id, transcript_path, cwd, source (startup | resume | clear | compact), and the project dir as $CLAUDE_PROJECT_DIR. Hooks can write context back into the conversation via stdout or via { hookSpecificOutput: { additionalContext: "..." } }. So a hook can:

1. POST { surface: 'claude-code', cwd, branch, transcriptPath, claudeSessionId } to a local Cosmo endpoint.
2. Endpoint mints / fetches the session doc, returns its ID.
3. Hook prints "Cosmo session: <id>. To call, run /call." back to the prompt context.

That gives Claude Code first-class membership in the same session universe Telegram has. The slash command /call then just shells out, looks at the most recent transcript line for the current sessionId, posts to /call/mint with that as the parent session, and prints the URL.

This is where the user wanted depth. The request was: even on web, make it feel like FaceTime / Google Meet, not like a Zoom side-tab.

PWA capabilities — what works in May 2026

Manifest pattern:

{
  "name": "Cosmo Calls",
  "short_name": "Cosmo",
  "display": "standalone",
  "orientation": "portrait",
  "theme_color": "#0f1115",
  "background_color": "#0f1115",
  "start_url": "/?source=pwa",
  "icons": [{ "src": "/icon-512.png", "sizes": "512x512", "type": "image/png" }]
}

The audio loop — WebRTC vs WebSocket+Opus

The honest take from the research is that for a single-user agent on a good network, WebSocket + Opus-in-AudioWorklet works. WebRTC is the better tech, but the floor latency improvement (~200-500ms) is dwarfed by LLM TTFT (~700ms+). Most production voice agents are migrating to WebRTC anyway, but it's a real infra step (STUN/TURN, optionally an SFU).

VAD — barge-in is what makes it feel like a call

The single biggest "this feels alive" feature is interruption. You start to talk, Cosmo stops talking immediately. Without that, every call feels like a walkie-talkie.

The solved pattern is browser-side Silero VAD via @ricky0123/vad-web:

const myvad = await vad.MicVAD.new({
  onSpeechStart: () => { /* user started — clear TTS ring buffer */ },
  onSpeechEnd:   (audio) => { /* Float32Array @ 16kHz — finalise STT */ },
  positiveSpeechThreshold: 0.5,
  negativeSpeechThreshold: 0.35,
  minSpeechFrames: 3,
  redemptionFrames: 8,
  frameSamples: 1536,
});

~1ms per frame, runs as ONNX in an AudioWorklet thread, doesn't block the UI.

Audio playback — the AudioWorklet ring buffer pattern

For low-latency streaming TTS chunks, the right pattern is Web Audio AudioWorklet with a ring buffer. PCM chunks come in over the WS, get pushed into a SharedArrayBuffer, the worklet drains it sample-by-sample. Cancellation = drain the buffer, one message. Same pattern OpenAI Realtime, LiveKit, and Pipecat browser clients all use.

The legacy options (HTMLAudioElement per chunk, MSE) all have audible gaps or codec-prefix requirements that don't suit our use.

The visual call screen

The bits that make a web page feel like a call screen, not like a settings page:

• Big avatar with audio-driven pulse. Route the TTS playback stream through an AnalyserNode, compute RMS in requestAnimationFrame, scale a CSS transform on the avatar. The avatar pulses when Cosmo is talking. Same for the user's mic ring when they're talking. This is the single highest-impact visual.
• Connection-state stages. dialing → ringing → connected with a subtle animation per stage. Fake it slightly if needed — even 400ms of "connecting" feels honest.
• Bright red end-call button, large, bottom-right. Material 3 style: oversized, unmistakable. Far from mute (which goes left).
• Mute button with strong visual feedback. No haptic on iOS, so the visual must carry it.
• Persistent call timer in MM:SS. Tiny touch, big effect.
• Wake lock on the whole time the call is active. Screen never dims.
• Status-bar tinted via theme-color so the iOS bar matches the call screen instead of clashing.
• Safe-area padding on the bottom so end-call sits above the home indicator.

Mic permissions UX

Mic permissions are fragile, especially on iOS Safari. Two non-negotiable rules:

• HTTPS only. getUserMedia doesn't prompt on http:// except localhost.
• Synchronous user-gesture. On iOS, getUserMedia must be called synchronously inside a tap handler. Don't await anything before it. Pattern: tap button → call getUserMedia first → await everything else after.

Best practice: a "Tap to start the call" splash that explicitly says mic access will be requested. No auto-prompt on page load — looks dodgy and gets denied.

STT — streaming, not Whisper

Whisper API is batch-only. For a real call we need streaming. The realistic options:

Streaming STT is non-negotiable for the call to feel right. Deepgram is the safe pick; AssemblyAI is cheaper. Whisper is out for the call path (still fine for voice notes).

TTS — the ElevenLabs vs OpenAI fork

This is the most interesting tradeoff in the stack.

The decision is: continuity vs latency vs voice quality vs new bill. There isn't a wrong answer; there's a preference.

Side note: Cartesia Sonic Turbo has the lowest model TTFB in the comparison set (~40ms) and is a strong technical choice, but the user explicitly named ElevenLabs and OpenAI as the two providers in scope, so it's a footnote, not a contender.

Brain — keeping it unchanged, with one switch

The Agent SDK supports token-level streaming via includePartialMessages: true. Today's executeClaudeCode() doesn't set it. With it on, the iterator additionally yields stream_event chunks containing raw Anthropic API streaming events. Per the docs:

for await (const message of query({ prompt, options: { includePartialMessages: true, ...existing } })) {
  if (message.type === "stream_event") {
    const event = message.event;
    if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
      // ship event.delta.text to TTS sentence buffer
    }
    if (event.type === "content_block_start" && event.content_block.type === "tool_use") {
      // emit "looking at your <tool name>..." filler audio
    }
  }
  // existing AssistantMessage and result handling unchanged
}

Telegram path is untouched (it ignores stream_event); call path adds the new branch. Per-turn voice-mode system prompt overlay is one extra append on the system prompt when the surface is call.

Voice-mode system prompt

From the prompting research (ElevenLabs guide, LiveKit prompting voice agents, Vapi guide), voice-tuned prompts produce responses 60-70% shorter than text equivalents. The overlay we'd append:

• You're on a phone call. Speak in 1-3 sentence turns. One question per turn.
• No markdown — no **bold**, no asterisks, no bullets, no headers, no code blocks.
• Spell out numbers when natural ("twenty-three dollars" not "$23"). Spell out small numbers, time-of-day, and money.
• Don't narrate the user's question back to them.
• Before any tool call, emit a short ack first ("looking now..." or similar), then call the tool.
• Use natural fillers ("hmm", "let me see") only when they emerge organically. Don't force it.
• If interrupted, stop cleanly and listen.

Aborting mid-stream

Today's interrupt path goes through Firestore (cancelRequest(), polled by the agent). For a call, the abort needs to be sub-100ms — too slow if it round-trips Firestore. The Agent SDK accepts an AbortController; the call path should hold it locally and abort directly when VAD fires onSpeechStart. Worth confirming in the SDK source before locking it in.

The same minting endpoint serves all surfaces. The only thing that varies is what context the surface ships in the mint request.

Telegram /call

1. User types /call in any chat.
2. Bot reads chat context (existing getChatContextWithMigration), POSTs { surface: 'telegram', telegramChatKey, recentTurns, project } to /call/mint.
3. Endpoint returns a single-use URL (TTL: 5 min to claim, then call session lives for the call).
4. Bot replies with the URL as a clickable link.
5. User taps → browser opens the call screen.

Claude Code /call

1. Slash command (~/.claude/commands/call.md) shells out to a node script.
2. Script reads the most recent transcript line for current sessionId, cwd, gitBranch.
3. POSTs { surface: 'claude-code', claudeSessionId, cwd, gitBranch, transcriptPath } to /call/mint.
4. Prints the URL into the response. User cmd-clicks.

An optional SessionStart hook (off by default until we've been running it a while) can pre-mint a session doc on Claude Code launch so even if you don't /call, your CC sessions are first-class in the same Firestore collection — useful for cross-session memory and "what was I doing yesterday" queries.

Mac menu-bar / native (later)

A small menu-bar app could open a fixed URL like cosmo-call.../c/menubar?fresh=1 to start an "ambient" call with no parent session — pure new conversation, no context inherited. Or it can read the system clipboard for a token URL the user just got and open straight to it. Out of scope for v1, easy to add later.

New components

• cosmo-voice — new PM2 process. Holds the WS connections. Runs the per-call loop. Talks to the existing cosmo-agent in-process (Node import) or over a tighter local IPC than Firestore (decision below).
• Mint endpoint — small HTTP API. Lives in cosmo-voice or cosmo-web.
• Web app — static site (Cloudflare Pages or served from cosmo-web). Manifest, service worker, the call screen UI, the AudioWorklet, the VAD setup.
• sessions Firestore collection (or whatever we end up calling it).
• /call in bot.js — registered command, mint + reply.
• /call Claude Code slash command — under ~/.claude/commands/call.md.

What stays exactly the same

• executeClaudeCode(). Adds includePartialMessages: true when called from voice surface. Prepends a voice-mode system prompt block when surface is call. Otherwise unchanged.
• Memory router (buildTurnTopicsContext). Same.
• All tools, all integrations, all skills.
• Telegram voice notes. The existing /voicemode path is untouched.

In rough order of impact:

When we add a real phone number later, the cleanest path is Twilio ConversationRelay, not raw Media Streams:

• Twilio handles STT and TTS server-side. Connects to our WebSocket as text in / text out.
• Our WS receives { type: "prompt", voicePrompt: "<user transcript>" } and replies with streamed { type: "text", token: "...", last: false } chunks.
• Interruption arrives as { type: "interrupt", utteranceUntilInterrupt: "..." } so our local conversation history can be rewound to exactly where the user heard.
• Architecturally identical to web-v1 from the server's perspective: same stream-of-tokens-from-Claude → stream-of-text-tokens-to-the-other-side loop. Twilio CR replaces the browser as the audio peer.

What would paint v1 into a corner:

• Doing TTS server-side and sending audio chunks to the browser. Twilio CR doesn't accept audio in that mode — it wants text. If we send text in v1 (browser does its own decode/playback) and a server-side TTS path lives parallel for browser audio output, both transports work cleanly later.
• Doing STT only in the browser. Twilio sends transcripts as text — server-side ingest path needs to exist regardless.

v1 should structure the agent-facing interface as (input: text-stream | audio-stream) → agent → (output: text-stream) so swapping browser ↔ Twilio is a transport adapter, not an agent change.

Full research findings at specs/research/voice-call-findings.md. Key external sources:

• openclaw-voice — community Claude voice chat add-on
• Anthropic cookbook — low-latency Claude voice with ElevenLabs
• LiveKit Agents — Anthropic plugin
• LiveKit — WebRTC vs WebSocket for voice agents
• Pipecat — Python voice agent framework
• Claude Agent SDK — streaming output docs
• Claude Code hooks reference
• @ricky0123/vad-web — Silero VAD in browser
• ElevenLabs WebSocket TTS
• ElevenLabs prompting guide for voice agents
• LiveKit — prompting voice agents to sound more realistic
• OpenAI Realtime WebRTC (for reference, not adoption)
• Twilio ConversationRelay — Anthropic streaming pattern
• WebKit bug 198277 — iOS PWA background audio
• WebKit bug 254545 — iOS PWA wake lock (resolved)

Doc lives at plans/voice-call.html. Generated Sun 3 May 2026 from research findings.