Browser WebRTC channel

Parlora.Channels.Browser connects one browser microphone/speaker pair to a Parlora session. Media uses Opus over WebRTC; Parlora decodes it to mono PCM16LE at 24 kHz for the voice provider. All audio passes through your server. Provider credentials stay on the server.

For a browser client you can run, start with OrderDesk.

Mount signaling in your application

Your application owns its HTTP server, routes, authentication, Origin checks, TLS, and session admission. After authorizing a Plug request, upgrade it:

Parlora.Channels.Browser.Signaling.upgrade(conn,
  start_session: fn channel ->
    Parlora.start_session(
      channel: channel,
      provider: {Parlora.Providers.OpenAI, api_key: key},
      backend: backend,
      context_policy: policy,
      instructions: "Your application's conversation rules",
      event_sink: self()
    )
  end,
  on_ready: &MyApp.greet/1,
  channel_options: [ice_servers: ice_servers, ice_transport_policy: :all]
)

key, backend, policy, and ice_servers come from your host configuration. The factory runs in the signaling socket process; event_sink: self() routes session events back to it. on_ready is optional application greeting logic. The signaling module starts no listener.

The browser creates an RTCPeerConnection, adds its microphone track, plays the remote audio track, and exchanges an offer and ICE candidates through the socket. Wait for session ready before inviting speech. Use HTTPS for remote microphone access; localhost is suitable for local development.

Signaling messages

Messages are JSON text frames. Each JSON object below is one frame. Values in angle brackets represent runtime SDP/ICE data. Use the values generated by the browser in place of these placeholders.

1. Open the WebSocket. Wait for the server's attached message before sending an offer. attached means the channel can accept signaling; it does not mean the session is ready for speech.
2. Add one microphone audio track to the peer, create an offer, and set it as the local description. Send the offer, then any local ICE candidates collected while creating it. Video, data channels, and renegotiation are unsupported.
3. Apply the server's answer as the remote description. Queue server candidates until that description is set, then apply them in order. Keep exchanging ICE candidates while negotiation completes.
4. Wait for ready before enabling the microphone and inviting speech. Readiness requires the WebRTC connection and all session adapters. Waiting for ready before sending the offer prevents negotiation from completing.

Client to server

{"type":"offer","sdp":"<offer.sdp>"}
{"type":"candidate","candidate":{"candidate":"<ICE candidate string>","sdpMid":"0","sdpMLineIndex":0,"usernameFragment":"<ICE username fragment>"}}

The offer contains one bidirectional Opus audio media section. Send candidates using event.candidate.toJSON(), wrapped as shown. sdpMLineIndex must be 0; sdpMid is a string or null, and usernameFragment is optional. Ignore a null event.candidate (end of gathering); do not send it as a candidate frame. Sending an offer before attached, candidates before the offer, or malformed messages can close the session.

Server to client

{"type":"attached","session_id":"<session ID>"}
{"type":"answer","sdp":"<answer SDP>"}
{"type":"candidate","candidate":{"candidate":"<ICE candidate string>","sdpMid":null,"sdpMLineIndex":0,"usernameFragment":null}}
{"type":"ready"}
{"type":"pong"}
{"type":"closed"}

Server candidates can arrive before the answer. A socket close can also occur without a final closed frame, including during failed startup. Handle both. Additional envelopes are:

Fields within caption/event payloads depend on the provider and event; do not assume every field is present. Captions are display data, not authoritative work results. Browser microphone mute disables the local track and is independent of output suppression and backend cancellation.

Minimal browser client

Mount the authorized signaling upgrade above at /voice, then serve this HTML from the same application origin (HTTPS, or localhost for development). It uses one microphone track and has no provider credentials or business logic. Set iceServers to the ICE configuration supplied by your application. An empty list is suitable for a test on the same machine. Configure the Elixir peer separately. If your Content Security Policy forbids inline scripts, move the script into a JavaScript asset that your policy allows and serve it from the same origin.

<!doctype html>
<html lang="en">
<meta charset="utf-8">
<title>Parlora voice</title>
<button id="connect">Connect microphone</button>
<button id="disconnect" disabled>Disconnect</button>
<p id="status" role="status">Disconnected.</p>
<audio id="speaker" autoplay controls></audio>
<script>
const connect = document.querySelector('#connect');
const disconnect = document.querySelector('#disconnect');
const status = document.querySelector('#status');
const speaker = document.querySelector('#speaker');
const iceServers = []; // Supply your host-issued STUN/TURN configuration here.
let current = null;

disconnect.onclick = () => current?.stop('Disconnected.');
window.addEventListener('pagehide', () => current?.stop('Disconnected.'));
connect.onclick = async () => {
  if (current) return;
  const call = {ended: false};
  current = call;
  connect.disabled = true;
  disconnect.disabled = false;
  status.textContent = 'Connecting…';

  const send = message => {
    if (!call.ended && call.socket?.readyState === WebSocket.OPEN) {
      call.socket.send(JSON.stringify(message));
    }
  };
  call.stop = message => {
    if (call.ended) return;
    send({type: 'disconnect'});
    call.ended = true;
    clearInterval(call.ping);
    call.socket?.close();
    call.peer?.close();
    call.microphone?.getTracks().forEach(track => track.stop());
    if (current === call) {
      current = null;
      speaker.srcObject = null;
      connect.disabled = false;
      disconnect.disabled = true;
      status.textContent = message;
    }
  };

  try {
    const microphone = await navigator.mediaDevices.getUserMedia({audio: true});
    if (call.ended) {
      microphone.getTracks().forEach(track => track.stop());
      return;
    }
    call.microphone = microphone;
    const peer = call.peer = new RTCPeerConnection({iceServers});
    const track = microphone.getAudioTracks()[0];
    track.enabled = false; // Keep caller speech out until the session is ready.
    peer.addTrack(track, microphone);
    const localCandidates = [], remoteCandidates = [];
    let offerSent = false;

    peer.onicecandidate = ({candidate}) => {
      if (!candidate || call.ended) return;
      if (offerSent) send({type: 'candidate', candidate: candidate.toJSON()});
      else localCandidates.push(candidate.toJSON());
    };
    peer.ontrack = ({track}) => {
      if (call.ended) return;
      speaker.srcObject = new MediaStream([track]);
      speaker.play().catch(() => {
        if (!call.ended) status.textContent = 'Press Play on the audio control to hear the assistant.';
      });
    };
    peer.onconnectionstatechange = () => {
      if (['failed', 'disconnected', 'closed'].includes(peer.connectionState)) {
        call.stop('Media connection closed. Connect again for a new session.');
      }
    };

    const scheme = location.protocol === 'https:' ? 'wss:' : 'ws:';
    const socket = call.socket = new WebSocket(`${scheme}//${location.host}/voice`);
    socket.onopen = () => {
      call.ping = setInterval(() => send({type: 'ping'}), 5000);
    };
    socket.onclose = () => call.stop('Session closed. Connect again for a new session.');
    socket.onerror = () => call.stop('Signaling failed. Check the server and route authorization.');

    // Serialize async handlers so candidates cannot overtake setRemoteDescription.
    let incoming = Promise.resolve();
    socket.onmessage = ({data}) => {
      incoming = incoming.then(async () => {
        if (call.ended) return;
        const message = JSON.parse(data);
        switch (message.type) {
          case 'attached': {
            const offer = await peer.createOffer();
            if (call.ended) return;
            await peer.setLocalDescription(offer);
            if (call.ended) return;
            send({type: 'offer', sdp: offer.sdp});
            offerSent = true;
            for (const candidate of localCandidates.splice(0)) send({type: 'candidate', candidate});
            break;
          }
          case 'answer':
            await peer.setRemoteDescription({type: 'answer', sdp: message.sdp});
            for (const candidate of remoteCandidates.splice(0)) await peer.addIceCandidate(candidate);
            break;
          case 'candidate':
            if (peer.remoteDescription) await peer.addIceCandidate(message.candidate);
            else remoteCandidates.push(message.candidate);
            break;
          case 'ready':
            track.enabled = true;
            status.textContent = 'Ready. You can speak.';
            break;
          case 'event':
            if (message.event === 'failed') call.stop('A voice adapter failed. Check the server.');
            break;
          case 'closed': call.stop('Session closed.'); break;
        }
      }).catch(() => call.stop('Connection setup failed. Check the server and ICE configuration.'));
    };
  } catch (_error) {
    call.stop('Could not start audio. Check microphone permission and server configuration.');
  }
};
</script>
</html>

Click Connect microphone, grant permission, and wait for Ready before speaking. A greeting depends on your on_ready callback. If autoplay is blocked, use the audio element's Play control. Disconnect, page navigation, socket closure, and setup failures release the local microphone, peer, and keepalive timer. Connecting again starts a fresh session. For captions, controls, and the complete OrderDesk UI, see the browser example and examples/order_desk/priv/browser/voice.js in the source checkout.

ICE and deployment

Supply ICE servers on both peers. ice_transport_policy: :relay requires TURN; :all is the default. Generate expiring TURN credentials in your application and send only those credentials to the page. Keep the shared TURN secret on the server.

A disconnect or the death of the signaling owner closes the channel and session. Reconnect starts a fresh session. Your application decides how to restore conversation context and reconcile work; it must not automatically replay uncertain writes.

The OrderDesk server binds to loopback and has no account login. Before adapting it for public access, provide authentication, HTTPS, appropriate ICE configuration, and admission/duration limits for your deployment.

Audio settings and controls

Browser audio requires the native libopus binding; see native build requirements. native_audio: false can omit bindings for local audio devices while retaining native_opus for the browser.

Use :fixed buffering for a constant startup target; zero initial cushion disables buffering. Recovery can release early under upstream pressure. Provider receive storage is separate from this cushion.

Clear advances the playback epoch and clears output managed by Parlora. Packets already admitted to ExWebRTC, the browser jitter buffer, or OS audio cannot be recalled. Queue/handoff observations report their boundary; they do not confirm audibility. Captions are text observations, not authoritative speech turns or work completion.