Tool calling

An avatar with tools can do things while it talks: look up an order, deal a hand of cards, generate a picture. The call runs outside the conversation loop, so the avatar keeps speaking while your backend works, and delivers the answer a few seconds later.

turn commits ──▶ background judge decides a tool is needed
                        │
   avatar keeps ◀───────┤ (conversation never blocks)
   talking              │
                        ├──▶ your app gets `calling` on /events   → show UI
                        └──▶ platform executes the tool           → your hook / MCP server
                                     │
                        result ◀─────┘
                                     │
                             appended to the conversation history
                                     │
                             avatar delivers it in its next line

A worked example: the tarot reader

Sibyl is a tarot-reading avatar. Drawing cards is not something a language model can do — the user picks them, in your app’s UI. So draw_cards is a tool.

1. The avatar covers, it does not stop

The user says “let’s start the reading.” Sibyl replies, out loud, in the ordinary streamed transcript:

“Alright — clear your mind, and let’s turn over the first card.”

That line is a cover line. Sibyl’s system prompt carries a short [BACKGROUND TOOLS] block — the name and one-line description of every tool this session resolved — telling her that tools run after she replies, that she should say she is checking, and that she must never invent the answer. So she produces a natural sentence and the conversation keeps moving.

The tool decision is a second, separate generation. Once the turn commits, the box runs one extra non-streamed call against the same model, showing it the tool list and the last few turns and demanding exactly one of:

<none/>
<tool name="draw_cards">{"count": 1, "spread": "past"}</tool>

The box’s side-agent parses that output — this is the marker, and it is how the call is recognized. What matters for your integration is where it lives: in the verdict generation, not in the spoken reply. The transcript your users see and your app receives on /mic_stream never contains a tool marker, so do not parse it for one.

Two reasons it is split this way. In-band tool calling would put a serial non-streamed model round (~0.3–0.6 s) on the critical path of every tool turn, and the box speculates replies continuously — a marker inside a speculative decode that never gets spoken must never fire a real side effect. Anchoring dispatch to the committed turn makes that impossible.

Boxes running with native tool calling enabled send the real JSON schemas and read back an OpenAI-style tool_calls array instead of the <tool> grammar. Same lifecycle, same guarantees; the difference is invisible to you.

2. Your app is told a call started

Having parsed the verdict, the box’s side-agent broadcasts a pre-event on the session’s /events WebSocket — before the tool round-trip, so your UI moves at the same moment the avatar’s cover line lands rather than seconds later:

{ "type": "agent_event", "kind": "calling", "tool": "draw_cards", "call_id": "tc_1a2b3c4d" }

This is the hook for your UI. Sibyl’s app fans a card spread across the screen and waits for the user to pick three. Meanwhile the conversation is still live — the user can talk, and Sibyl answers normally.

If the avatar happens to fall silent while the call is outstanding, the box plays a short filler line after ~2.5 s so the pause does not read as a freeze.

3. The platform executes the tool

In parallel with that broadcast, the box POSTs the call to the platform’s agent gateway, which forwards it to your hook (see Executing an inline tool):

{ "v": 1, "op": "tool_call", "sid": "019fe497-…", "conv": "reading-8812",
  "call_id": "tc_1a2b3c4d", "idem": "9f2c1ab4de70c518",
  "tool": "draw_cards", "args": { "count": 1, "spread": "past" },
  "history": [ { "role": "user", "content": "let's start the reading" } ] }

Your hook holds the HTTP response open until the user has picked, then answers:

{ "ok": true, "status": "done",
  "result": { "kind": "tool", "text": "Card 1 (past): The Tower, reversed.",
              "structured": { "cards": [ { "name": "The Tower", "reversed": true } ] } } }

Hold time is bounded: the tool’s timeout_ms (≤ 25 s), and the box gives up after 30 s regardless. For anything slower, see When the tool takes longer than the call.

The same flow works with an MCP server. Register the card backend once, mint with {"tools": {"servers": ["mcps_…"]}}, and the gateway calls tools/call instead of your hook — everything else is identical, including the calling broadcast that drives your UI. Match the call to the right user with _meta["casola/conv"], which carries the conv you minted with. The one thing you give up is result fidelity: MCP results are flattened to text plus structured, so a tool that needs to push an image card to the browser has to be inline. Dealing cards does not — the spread is drawn by your own frontend.

4. The result rejoins the conversation

The box appends the result to the conversation history as a bracketed note the model can read but never speaks aloud, then decides how to deliver it:

User has stayed silent and the result is fresh (under 60 s): the avatar speaks a follow-up line immediately — “The Tower, reversed — so something you were bracing for already broke, and you’re standing in the after.”
User spoke while the call was in flight: the result folds into context silently and colors the next reply instead. Nothing is interrupted.
Result is stale or the session ended: it is dropped from speech; the tenant hook and webhooks still receive it.

The note also instructs the model to use only the facts it contains, so Sibyl cannot invent a card that was never drawn.

The three kinds of tool

	Inline (your hook)	MCP server	Built-in
Declared as	`tools.inline[]` + `agent.hook`	`tools.servers: ["mcps_…"]`	`tools.servers: ["casola.image"]`
Registered	nowhere — the definition is the declaration	`POST …/mcp-servers` (once)	toggled per workspace
Runs on	your server	the platform, as an MCP client	the GPU box
Result fidelity	full — `kind`, `url`, `caption` reach the browser	normalized to text + `structured`	rich (`kind: "image"`)
Use it for	anything app-specific, anything with UI	an existing MCP backend	on-box image generation

Inline and MCP are interchangeable for most work — both reach your backend, both drive the same /events broadcast, both can hold the response while a user acts on screen. Pick MCP when you already run an MCP server and want one registration reused across sessions; pick inline when the tool is specific to one app, when you want the definition to travel with the mint, or when the result needs to reach the browser as an image card.

Regardless of source, the model only ever sees each tool’s name and description when deciding. Write the description for that reader: “Ask the user to draw cards from the on-screen spread” earns a call at the right moment; “card utility” does not.

Declaring tools on a mint

Tools resolve at mint and freeze into the session. Everything here is secret-key only — a publishable, device or trusted-issuer credential sending tools, extra_tools or context gets 403 tools_context_not_allowed.

{
  "avatar_id": "019fe463-…",
  "agent": {
    "hook": "https://tarot.example.com/casola/hook",
    "tok": "your-own-bearer",
    "conv": "reading-8812"
  },
  "tools": {
    "inline": [
      {
        "name": "draw_cards",
        "description": "Ask the user to draw cards from the on-screen spread.",
        "input_schema": {
          "type": "object",
          "properties": {
            "count": { "type": "integer" },
            "spread": { "type": "string", "enum": ["past", "present", "future"] }
          },
          "required": ["count"]
        },
        "timeout_ms": 20000
      }
    ],
    "max_calls": 12
  }
}

The `tools` object

Field	Type	Notes
`servers`	string[]	MCP server ids (`mcps_…`) and built-in keys (`casola.image`). Max 4
`inline`	object[]	Tool definitions executed by your own hook. Requires `agent.hook`
`allow`	string[]	Name filter applied to everything the servers and inline list resolve to
`max_calls`	integer	Per-session tool-call budget. Default 20, max 100
`history`	boolean	Default `true`. Send a recent-turn snapshot to MCP tools

tools replaces the avatar’s stored tool config; extra_tools takes the same shape and merges onto whatever resolved, with the extra winning a name collision. Full layering rules: Prompt & overrides.

A tool definition

Field	Type	Notes
`name`	string	Required. `[a-zA-Z0-9_-]{1,64}`, unique across the session
`description`	string	Required, ≤ 300 chars. The model picks tools from this — write it for a reader, not a schema
`input_schema`	object	Required. JSON Schema, root `type: "object"`, ≤ 2,048 bytes, depth ≤ 4, no `$ref` / `$defs`
`timeout_ms`	integer	Clamped to 1,000–25,000. Default 8,000
`idempotent`	boolean	Safe to retry once on a transport error
`fast`	boolean	Route through the pre-speech gate instead of the background lane — only for tools that answer in a few hundred ms
`speak_hint`	string	How to deliver the result out loud
`cover_line`	string	Suggested phrasing while the call runs

The `agent` object

Field	Notes
`hook`	Your HTTPS endpoint. Required to use `tools.inline`; also receives `session_start`, `turn_committed` and `session_end`
`tok`	Bearer the platform sends to your hook. Yours to choose and verify
`conv`	Your conversation id, echoed on every op. Defaults to the session id

Serialized agent is capped at 1,024 bytes.

What comes back

A successful mint echoes the resolved names:

{ "status": "ready", "session_id": "019fe497-…", "connect_url": "…", "session_token": "…",
  "tools_resolved": ["draw_cards"] }

Absent means no tools resolved. Assert on it in your integration tests — a silently tool-less session looks identical to a working one until the avatar starts making things up.

Executing an inline tool

Your hook is one HTTPS endpoint handling four ops, discriminated by op, authenticated with the tok you supplied:

`op`	When	Expected response
`session_start`	Session connects	`{"ok": true}`, optionally `turn_seq_base`
`tool_call`	A tool fires	`{"ok": true, "status": "done", "result": {…}}`
`turn_committed`	Each spoken turn	`{"ok": true}` — fire-and-forget transcript
`session_end`	Slot released	`{"ok": true}`

Every request carries { v, op, conv, sid, ts, … } and Authorization: Bearer <your tok>.

The `tool_call` response

The result envelope is strict. Anything other than ok: true and status: "done" and an object result is read as a failure, and the avatar apologizes gracefully instead of inventing an answer.

{ "ok": true, "status": "done",
  "result": {
    "kind": "tool",
    "text": "Card 1 (past): The Tower, reversed.",
    "structured": { "cards": [ … ] }
  } }

`result.kind`	Effect
`tool`	Text (plus serialized `structured`, truncated at 800 chars) enters the conversation
`image`	Also broadcast to `/events` with `url` and `caption` — the browser renders a card
`image_pending`	Broadcast immediately; your client polls the `url` for completion

Failure: return {"ok": false, "error": "…"}. The box turns that into a spoken apology and an offer to retry.

Idempotency

Every tool_call carries an idem key. The gateway records (session_id, idem) and replays the stored response byte-for-byte on a repeat, without touching your hook or spending budget. Retries after a network blip are therefore free of double side effects — as long as you treat idem, not call_id, as the deduplication key.

Verifying the caller

Compare the bearer against the tok you minted with, in constant time, and check conv matches a conversation you started. Everything else in the envelope is untrusted input.

MCP servers

Register a streamable-HTTP MCP server once per workspace; sessions then reference it by id. The platform is the MCP client — the GPU box never speaks MCP and never sees your credentials.

GET    /api/v1/workspaces/{wsId}/mcp-servers
POST   /api/v1/workspaces/{wsId}/mcp-servers
GET    /api/v1/workspaces/{wsId}/mcp-servers/{serverId}
POST   /api/v1/workspaces/{wsId}/mcp-servers/{serverId}/refresh
PATCH  /api/v1/workspaces/{wsId}/mcp-servers/{serverId}
DELETE /api/v1/workspaces/{wsId}/mcp-servers/{serverId}

Required scope: workspaces:write (or user:write). Reads included — a registration carries a credential, so listing is a privileged operation.

POST /api/v1/workspaces/ws_…/mcp-servers
{
  "name": "policy-backend",
  "url": "https://mcp.acme.example/mcp",
  "auth": { "kind": "bearer", "secret": "…" },
  "allowed_tools": ["lookup_policy"],
  "default_timeout_ms": 8000
}

auth.kind is none (default), bearer, or header (which also needs header). The URL must be https:, and must not be localhost, an IP literal, or a .local / .internal hostname.

Registration runs a live probe — initialize + tools/list — and the response tells you whether it worked:

{ "id": "mcps_019fe4…", "name": "policy-backend", "status": "active",
  "tools": [ { "name": "lookup_policy", "description": "Look up a policy by number." } ],
  "tools_cached_at": 1719000000, "validation": { "ok": true } }

A failed probe still creates the row, with status: "error" and last_error, so you can fix the server and call /refresh rather than re-registering. A server with no cached tool list is refused at mint with 409 mcp_server_unavailable.

The tool list is cached for 10 minutes and served stale-while-revalidate, so a mint never blocks on your server being up. PATCH … {"status": "disabled"} takes a server out of service without deleting it; DELETE always returns 204.

Each tools/call carries _meta with casola/conv, casola/sid, and (unless tools.history: false) casola/history. Use casola/conv to map a call back to the end user you minted the session for.

MCP results are normalized to text plus structured, capped at 8 KB. If you need url / caption to reach the browser as an image card, use an inline tool instead.

Built-in tools

Tools the GPU box runs itself. Enable per workspace, then opt in per session.

GET /api/v1/workspaces/{wsId}/builtin-servers
PUT /api/v1/workspaces/{wsId}/builtin-servers/{serverKey}     { "enabled": true }

Key	Tools
`casola.image`	`generate_image` (a picture from a prompt), `edit_avatar` (the avatar’s own face, restyled — identity preserved)

Reference it like any other server: {"tools": {"servers": ["casola.image"]}}. Not enabled → 403 builtin_server_not_enabled.

These never traverse the gateway on the way out; the box reports completion afterwards, which is what fires your webhooks and usage records. Results arrive on /events as {"kind": "image", "url": …, "caption": …} — image URLs are presigned and expire after an hour, so copy anything you want to keep.

Watching from the browser

The /events WebSocket on the session’s box carries tool activity:

const ws = new WebSocket(`${connectUrl.replace('https', 'wss')}/events?token=${sessionToken}`);
ws.onmessage = (e) => {
  const m = JSON.parse(e.data);
  if (m.type !== 'agent_event') return;      // the feed also carries status frames at 10 Hz
  // { type, kind, tool, call_id, url?, caption? }
  if (m.kind === 'calling') showSpread(m.tool);
  if (m.kind === 'image') showCard(m.url, m.caption);
};

Filter on type === 'agent_event'; everything else on that socket is telemetry. The frame is advisory — the authoritative record is your own hook, which is called on your server where you can trust it. @casola/avatar-client does not surface this feed yet, so open the socket directly.

When the tool takes longer than the call

The box holds a tool call for at most 30 s. Past that, return immediately with kind: "image_pending" (or a plain tool result saying the work started) and a url your client can poll. The browser drives completion; the avatar is not interrupted.

One honest limitation: a result that arrives after the call returned is not pushed back into the conversation. The avatar will not announce it. Show it in your UI, and if the user asks about it, they can simply ask — the next tool call can fetch the finished state.

Pacing and budget

Behaviour of the conversation loop, worth designing around:


Concurrency	One tool call in flight per session
Cooldown	~20 s between dispatches
Turn	At most one tool per turn
Hold	≤ 30 s per call (`timeout_ms` ≤ 25,000)
Session budget	`tools.max_calls`, default 20, max 100
Tools per session	16
MCP servers per session	4

Exhausting the budget returns budget_exhausted on the wire; the avatar apologizes rather than failing the call. Avatar-initiated turns never dispatch tools, so a tool result cannot trigger another tool.

Errors

At mint (400 unless noted):

Error	Cause
`tools_context_not_allowed` (403)	Non-secret credential sent `tools` / `extra_tools` / `context`
`workspace_id_required`	`tools.servers` used with a key that is not workspace-bound
`inline_tools_require_agent_hook`	`tools.inline` without `agent.hook`
`invalid_agent_hook`	`agent.hook` is not `https://…`
`agent_claim_too_large`	Serialized `agent` over 1,024 bytes
`too_many_tools` / `too_many_mcp_servers`	Over 16 tools / 4 servers
`tool_name_conflict` (409)	Two resolved tools share a name
`no_tools_resolved`	`tools` was sent but resolved to nothing
`invalid_tool_name` / `invalid_tool_description:<name>`	Name pattern, or description over 300 chars
`schema_root_must_be_object:<name>` / `schema_too_large:<name>` / `schema_too_deep:<name>` / `schema_ref_unsupported:<name>`	`input_schema` rejected
`invalid_max_calls`	Outside 1–100
`mcp_server_not_found` (400) / `mcp_server_unavailable` (409)	Unknown or disabled server / no cached tool list
`builtin_server_not_enabled` (403) / `builtin_server_not_found` (404)	Built-in key not toggled on / unknown
`agent_payload_too_large`	Tools + context over 128 KB

On the wire, returned in the body with HTTP 200: unknown_tool, budget_exhausted, server_disabled, builtin_tool_is_local, timeout, tool_error (with a message, first 300 chars).

If your hook returns any 4xx, the box disables the tool channel for the rest of that session — a bad token or an unknown conversation cannot be fixed by retrying. Three consecutive 5xx or transport failures do the same. The session continues; the avatar simply has no tools.

Webhooks

Subscribe with POST /api/v1/workspaces/{wsId}/webhooks (webhooks:write) to agent.tool_call.succeeded and agent.tool_call.failed:

{ "event": "agent.tool_call.succeeded",
  "workspace_id": "ws_…",
  "timestamp": 1719000000000,
  "data": { "session_id": "019…", "conv": "reading-8812", "tool": "draw_cards",
            "source": "hook", "ok": true, "ms": 4210, "result": { … } } }

Signed as x-signature-256: sha256=<hex HMAC-SHA256 of the raw body> with the endpoint’s signing secret. Delivery is best-effort with three attempts and 1/2/4 s backoff — use it for audit and analytics, not as the path your app depends on.

Retention

Tool configuration for a live session is stored until the session ends plus a 15-minute grace window, then deleted. The call ledger that backs idempotency and usage is kept for 30 days, and is purged on account deletion.

Sessions — where tools are declared
Prompt & overrides — how tools layers against stored defaults
Profiles & config — storing tool config as an avatar’s default