Browser SDK

Browser SDK

@casola/avatar-client is a framework-free TypeScript browser SDK that handles MSE video playback, mic capture, 48 kHz → 16 kHz resampling, and PCM16 framing against the Casola edge. Install it from npm:

npm install @casola/avatar-client

Quick example

import { AvatarSession, connectViaToken } from '@casola/avatar-client'

const { connect_url, session_token } = await fetch('/my-backend/start-session').then(r => r.json())

const session = new AvatarSession({
  videoEl: document.querySelector('video'),
  connect: connectViaToken({ connectUrl: connect_url, sessionToken: session_token }),
  workletUrl: '/mic-worklet.js',
  callbacks: {
    onStateChange(next) { console.log('state:', next) },
    onPartial(text)     { console.log('partial:', text) },
    onTurn(t)           { console.log('turn:', t.text) },
    onFirstFrame()      { console.log('live') },
    onClose(reason)     { console.log('ended:', reason) },
    onError(err)        { console.error(err) },
  },
})

await session.start()

---

AvatarSession

Constructor

new AvatarSession(opts: AvatarSessionOpts)

Option	Type	Required	Description
videoEl	HTMLVideoElement	yes	Video element that receives the avatar stream.
connect	ConnectStrategy	yes	Strategy returned by connectViaToken.
lang	string	no	BCP-47 language tag sent to the ASR engine (default "en").
workletUrl	string	no	URL of the mic-worklet script (default "/mic-worklet.js").
prewarm	() => Promise<void> | void	no	Called during the ready → connecting transition; use for persona selection or other prep. Errors are silently swallowed.
dev	boolean	no	Logs unexpected state-machine transitions to the console when true.
callbacks	object	no	Event callbacks (see below).

Callbacks

All callbacks are optional.

Callback	Signature	When fired
onStateChange	(next: WidgetState, prev: WidgetState) => void	Every state transition.
onQueueStatus	(s: {phase:'open'}) => void	Signals the edge connection opened. Not emitted by connectViaToken.
onPartial	(text: string) => void	Incremental ASR transcript from the avatar.
onTurn	(t: Turn) => void	Completed turn: {text, reply, language?, speechId?}.
onFirstFrame	() => void	First video frame decoded — the session is visually live.
onClose	(reason: EndReason) => void	Session ended normally or due to a server close.
onError	(err: unknown) => void	Unrecoverable error (mic permission denied, media failure).

Methods

session.start(): Promise<void>

Begins the session. Transitions from idle → waiting. Resolves immediately (connection is async). No-op if not in idle state.

session.leave(): void

Gracefully ends the session. Tears down media and WebSockets, transitions to idle, fires onClose('generic').

session.setMuted(muted: boolean): void

Mutes or unmutes the mic stream. Audio is captured but not transmitted while muted.

session.destroy(): void

Hard teardown — stops everything immediately without state transitions or callbacks. Use in component cleanup (e.g. useEffect return, onUnmount).

Getters

session.state: WidgetState

Current state (see state machine below).

session.sessionCapSeconds: number | undefined

Session cap in seconds as reported by the edge once the connection is ready. undefined until the cap is known (i.e. until ready state).

Static methods

AvatarSession.ensureMicPermission(): Promise<void>

Requests microphone access. Rejects with a DOMException if denied. Call from a user gesture. Throws NotAllowedError, SecurityError, NotFoundError, OverconstrainedError, or NotSupportedError.

AvatarSession.mediaSupported(): boolean

Returns true if ManagedMediaSource (Safari 17.1+) or MediaSource is available in the browser. Use to gate the CTA before starting a session.

---

State machine

AvatarSession transitions through these states (from WidgetState):

idle → waiting → ready → connecting → live → ended
                                           ↘ error

State	Meaning
idle	Initial state; no session in progress.
waiting	Session started; awaiting the edge target from the connect strategy (near-instant with connectViaToken).
ready	Slot granted; starting media setup.
connecting	Opening MSE and mic WebSockets.
live	First video frame received; session is active.
ended	Session closed by the server or leave().
error	Unrecoverable failure.

WidgetState also includes selecting and verifying, which are host application states (not used by AvatarSession directly — only emitted if the session is driven by custom host state-machine logic).

EndReason values

Passed to onClose:

Value	Meaning
cap	Session time cap reached.
edge_disconnect	MSE WebSocket closed unexpectedly.
kicked	Server ended the session.
expired	Session token expired before the connection was established.
dropped	Connection closed before a ready signal.
generic	leave() called, or unclassified server close.

---

Connect strategy

connectViaToken

Your server mints a session_token via POST /api/v1/sessions and passes it to the browser; the browser connects directly to the edge.

import { connectViaToken } from '@casola/avatar-client'

connectViaToken({
  connectUrl: string,       // edge base URL from POST /api/v1/sessions
  sessionToken: string,     // short-lived JWT from the same response
  edgePaths?: {
    mse?: string,           // default '/mse'
    micStream?: string,     // default '/mic_stream'
  },
  sessionCapSeconds?: number,
})

---

Worklet asset

audioWorklet.addModule() requires a separate file at a real URL — it cannot be inlined into the main bundle. Ship dist/worklet/mic-worklet.js from the package as a static asset and pass its URL to workletUrl.

Vite:

import workletUrl from '@casola/avatar-client/worklet?worker&url'
// pass workletUrl to AvatarSession opts

esbuild / custom bundler: copy node_modules/@casola/avatar-client/dist/worklet/mic-worklet.js to your public directory (e.g. via a build script or plugin) and pass the resulting path.

No bundler: serve the file from a CDN or static host and pass the absolute URL directly.