Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "fishjam-docs",
"version": "0.24.0",
"version": "0.30.0",
"private": true,
"scripts": {
"docusaurus": "docusaurus",
Expand Down
6 changes: 6 additions & 0 deletions versioned_docs/version-0.30.0/_common/agents/definition.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
## What is an Agent?

An agent is a piece of software that allows your backend server to participate in a Fishjam room, similar to how the Fishjam client SDKs allow your client-side application to participate in a Fishjam room.
They can be used to implement features such as real-time audio transcription, real-time content moderation, conversations with AI agents and more.

You can simply think of an agent as a peer running within your backend application.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
:::danger[Remember to disconnect your agents!]
It's important to disconnect agents, because **every connected agent generates usage** just as a normal peer.
:::
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
By default, agents **won't** receive peers' audio streams.
This is by design and aims to prevent unnecessary resource usage by the agents.

For an agent to start receiving a peer's audio, the peer must be **created** with the **subscribe** option set.
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
:::warning[A composition bills for the whole time it exists]
With that guard off, cleanup waits for the outputs to fall silent too, so one that keeps
publishing is never cleaned up for you. Delete it as soon as you are finished, and do not
leave one running after a test. See
[Cost and lifecycle](../../explanation/compositions#cost-and-lifecycle).
:::
48 changes: 48 additions & 0 deletions versioned_docs/version-0.30.0/ai-skill.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
sidebar_position: 1
title: Build Fishjam apps with AI
sidebar_label: Build with AI
description: "The Fishjam Skill gives your AI coding assistant up-to-date, correct guidance for building Fishjam apps: right SDK, real APIs, secure auth."
slug: /ai-skill
---

# Build Fishjam apps with AI

The **Fishjam Skill** gives any skills-compatible coding assistant accurate, always-current Fishjam expertise. With it installed, your assistant generates code that uses the right SDK, real API names, and secure auth patterns from the first prompt.

## What it covers

The skill is one entry in Software Mansion's `skills` plugin, and it spans the whole Fishjam stack — backend, web, and mobile:

- **Platform fundamentals** — rooms, peers, tracks, the two-tier auth model, notifications, and the REST API.
- **Server SDKs (Node & Python)** — including AI voice agents and the Gemini Live integration.
- **React web client** — the `@fishjam-cloud/react-client` hooks and providers.
- **React Native / Expo client** — permissions, foreground service, CallKit, screen share, and Picture-in-Picture.

## Install

Install the skill into Claude Code (recommended) or any skills-compatible assistant:

```bash
npx skills add software-mansion-labs/skills
```

## How it activates

No command needed. The skill auto-loads whenever you mention Fishjam or a Fishjam SDK/API in your prompts, like `@fishjam-cloud/react-client`, `FishjamClient`, or `managementToken`.

## Why it helps

The skill turns your assistant into a Fishjam specialist:

- **Always current** — it tracks the latest SDKs and APIs, so your assistant builds on today's Fishjam instead of stale examples.
- **Secure by default** — authentication and tokens follow the patterns we recommend for production, from the first line of code.
- **Full-stack in one place** — backend, web, and mobile share the same guidance, so features connect cleanly end to end.

:::tip
The skill follows the latest Fishjam release, so your assistant always works against current SDKs and APIs.
:::

## Links

- Source and installation details: [software-mansion-labs/skills](https://github.com/software-mansion-labs/skills) (MIT).
12 changes: 12 additions & 0 deletions versioned_docs/version-0.30.0/api/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"label": "API",
"position": 7,
"link": {
"type": "generated-index",
"description": "API documentation for Client and Server SDKs.",
"slug": "/api"
},
"customProps": {
"id": "generated-api"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# @fishjam-cloud/react-native-custom-video-source

## Modules

- [index](index/index.md)
- [webgpu](webgpu/index.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Function: useManagedForwardTrack()

> **useManagedForwardTrack**(`enabled`): [`ManagedForwardTrack`](../interfaces/ManagedForwardTrack.md)

Defined in: [src/internal/useManagedForwardTrack.ts:29](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedForwardTrack.ts#L29)

Owns the async lifecycle of a forwarding custom video track: creates it while `enabled`,
exposes the handle + stream once ready, and stops the tracks on disable/unmount (also when
creation resolves after the owner already unmounted).

## Parameters

| Parameter | Type |
| ------ | ------ |
| `enabled` | `boolean` |

## Returns

[`ManagedForwardTrack`](../interfaces/ManagedForwardTrack.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Function: useManagedPooledTrack()

> **useManagedPooledTrack**(`enabled`, `width`, `height`, `poolSize`): [`ManagedPooledTrack`](../interfaces/ManagedPooledTrack.md)

Defined in: [src/internal/useManagedPooledTrack.ts:77](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L77)

Owns the async lifecycle of a surface pool + pooled custom video track: allocates both while
`enabled` (re-allocates when the dimensions change), exposes worklet-ready descriptors, and
tears down in the correct order (stop tracks, then dispose the pool) on disable/unmount.

## Parameters

| Parameter | Type |
| ------ | ------ |
| `enabled` | `boolean` |
| `width` | `number` |
| `height` | `number` |
| `poolSize` | `number` |

## Returns

[`ManagedPooledTrack`](../interfaces/ManagedPooledTrack.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# index

Publish your own video frames to Fishjam.

Source-lifecycle hooks that create a custom video track, publish it, and clean it up — you
supply the frames from any source. Two modes, picked by how you produce frames:

- [useManagedForwardTrack](functions/useManagedForwardTrack.md) — you already have finished native buffers (a camera, a native
ML pipeline, a compositor); forward each buffer pointer with `forwardFrame` from
`@fishjam-cloud/react-native-webrtc`.
- [useManagedPooledTrack](functions/useManagedPooledTrack.md) — you render the frames yourself; allocate a surface pool, draw
into it, and hand each frame back with `pushFrame`. The `@fishjam-cloud/react-native-custom-video-source/webgpu`
entry point provides a WebGPU camera-rendering toolkit for this mode.

For a ready-made VisionCamera integration on top of this, use
`@fishjam-cloud/react-native-vision-camera-source`.

## Interfaces

- [ManagedForwardTrack](interfaces/ManagedForwardTrack.md)
- [ManagedPooledTrack](interfaces/ManagedPooledTrack.md)
- [WorkletBufferDescriptor](interfaces/WorkletBufferDescriptor.md)

## Functions

- [useManagedForwardTrack](functions/useManagedForwardTrack.md)
- [useManagedPooledTrack](functions/useManagedPooledTrack.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Interface: ManagedForwardTrack

Defined in: [src/internal/useManagedForwardTrack.ts:16](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedForwardTrack.ts#L16)

State of a forwarding custom video track managed by [useManagedForwardTrack](../functions/useManagedForwardTrack.md).
While the track is being created (or after an error) `track` and `stream` are `null`.

## Properties

### error

> **error**: `null` \| `Error`

Defined in: [src/internal/useManagedForwardTrack.ts:19](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedForwardTrack.ts#L19)

***

### stream

> **stream**: `null` \| `MediaStream`

Defined in: [src/internal/useManagedForwardTrack.ts:18](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedForwardTrack.ts#L18)

***

### track

> **track**: `null` \| `ForwardTrack`

Defined in: [src/internal/useManagedForwardTrack.ts:17](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedForwardTrack.ts#L17)
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Interface: ManagedPooledTrack

Defined in: [src/internal/useManagedPooledTrack.ts:26](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L26)

State of a pooled custom video track managed by [useManagedPooledTrack](../functions/useManagedPooledTrack.md).
While the pool and track are being created (or after an error) all fields are `null`.

## Properties

### bufferDescriptors

> **bufferDescriptors**: `null` \| [`WorkletBufferDescriptor`](WorkletBufferDescriptor.md)[]

Defined in: [src/internal/useManagedPooledTrack.ts:30](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L30)

Plain per-surface descriptors (the pool object itself is not worklet-serializable).

***

### error

> **error**: `null` \| `Error`

Defined in: [src/internal/useManagedPooledTrack.ts:31](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L31)

***

### stream

> **stream**: `null` \| `MediaStream`

Defined in: [src/internal/useManagedPooledTrack.ts:28](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L28)

***

### track

> **track**: `null` \| `PooledTrack`

Defined in: [src/internal/useManagedPooledTrack.ts:27](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L27)
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Interface: WorkletBufferDescriptor

Defined in: [src/internal/useManagedPooledTrack.ts:15](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L15)

One pooled output surface as plain values the frame worklet can capture and import itself.

## Properties

### height

> **height**: `number`

Defined in: [src/internal/useManagedPooledTrack.ts:19](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L19)

***

### index

> **index**: `number`

Defined in: [src/internal/useManagedPooledTrack.ts:16](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L16)

***

### surfaceHandle

> **surfaceHandle**: `bigint`

Defined in: [src/internal/useManagedPooledTrack.ts:17](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L17)

***

### width

> **width**: `number`

Defined in: [src/internal/useManagedPooledTrack.ts:18](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/internal/useManagedPooledTrack.ts#L18)
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
// @ts-check
/** @type {import("@docusaurus/plugin-content-docs").SidebarsConfig} */
const typedocSidebar = {items:[{type:"category",label:"index",items:[{type:"category",label:"Interfaces",items:[{type:"doc",id:"api/custom-video-source/index/interfaces/ManagedForwardTrack",label:"ManagedForwardTrack"},{type:"doc",id:"api/custom-video-source/index/interfaces/ManagedPooledTrack",label:"ManagedPooledTrack"},{type:"doc",id:"api/custom-video-source/index/interfaces/WorkletBufferDescriptor",label:"WorkletBufferDescriptor"}]},{type:"category",label:"Functions",items:[{type:"doc",id:"api/custom-video-source/index/functions/useManagedForwardTrack",label:"useManagedForwardTrack"},{type:"doc",id:"api/custom-video-source/index/functions/useManagedPooledTrack",label:"useManagedPooledTrack"}]}],link:{type:"doc",id:"api/custom-video-source/index/index"}},{type:"category",label:"webgpu",items:[{type:"category",label:"Interfaces",items:[{type:"doc",id:"api/custom-video-source/webgpu/interfaces/CameraPassthroughPipelineOptions",label:"CameraPassthroughPipelineOptions"},{type:"doc",id:"api/custom-video-source/webgpu/interfaces/CreateCameraShaderBindingsOptions",label:"CreateCameraShaderBindingsOptions"},{type:"doc",id:"api/custom-video-source/webgpu/interfaces/UseCameraWebGpuDeviceResult",label:"UseCameraWebGpuDeviceResult"}]},{type:"category",label:"Type Aliases",items:[{type:"doc",id:"api/custom-video-source/webgpu/type-aliases/WebGpuRuntime",label:"WebGpuRuntime"}]},{type:"category",label:"Functions",items:[{type:"doc",id:"api/custom-video-source/webgpu/functions/getWebGpuRuntime",label:"getWebGpuRuntime"},{type:"doc",id:"api/custom-video-source/webgpu/functions/useCameraWebGpuDeviceWithOverride",label:"useCameraWebGpuDeviceWithOverride"}]},{type:"category",label:"WebGPU",items:[{type:"doc",id:"api/custom-video-source/webgpu/functions/assertWebGpuDeviceSupportsCameraImport",label:"assertWebGpuDeviceSupportsCameraImport"},{type:"doc",id:"api/custom-video-source/webgpu/functions/computeAspectFillCrop",label:"computeAspectFillCrop"},{type:"doc",id:"api/custom-video-source/webgpu/functions/computeSquareCrop",label:"computeSquareCrop"},{type:"doc",id:"api/custom-video-source/webgpu/functions/createCameraBindGroup",label:"createCameraBindGroup"},{type:"doc",id:"api/custom-video-source/webgpu/functions/createCameraPassthroughPipeline",label:"createCameraPassthroughPipeline"},{type:"doc",id:"api/custom-video-source/webgpu/functions/createCameraShaderBindings",label:"createCameraShaderBindings"},{type:"doc",id:"api/custom-video-source/webgpu/functions/createCameraTextureResolver",label:"createCameraTextureResolver"},{type:"doc",id:"api/custom-video-source/webgpu/functions/encodeCameraPassthrough",label:"encodeCameraPassthrough"},{type:"doc",id:"api/custom-video-source/webgpu/functions/getOutputSurfaceFormat",label:"getOutputSurfaceFormat"},{type:"doc",id:"api/custom-video-source/webgpu/functions/getRequiredWebGpuCameraFeatures",label:"getRequiredWebGpuCameraFeatures"},{type:"doc",id:"api/custom-video-source/webgpu/functions/packFrameCropParams",label:"packFrameCropParams"},{type:"doc",id:"api/custom-video-source/webgpu/functions/resolveCameraTexture",label:"resolveCameraTexture"},{type:"doc",id:"api/custom-video-source/webgpu/functions/useCameraWebGpuDevice",label:"useCameraWebGpuDevice"},{type:"doc",id:"api/custom-video-source/webgpu/interfaces/CameraPassthroughPipeline",label:"CameraPassthroughPipeline"},{type:"doc",id:"api/custom-video-source/webgpu/interfaces/CameraShaderBindings",label:"CameraShaderBindings"},{type:"doc",id:"api/custom-video-source/webgpu/interfaces/CameraTextureResolver",label:"CameraTextureResolver"},{type:"doc",id:"api/custom-video-source/webgpu/interfaces/FrameCrop",label:"FrameCrop"},{type:"doc",id:"api/custom-video-source/webgpu/interfaces/WebGpuFrameRenderContext",label:"WebGpuFrameRenderContext"},{type:"doc",id:"api/custom-video-source/webgpu/type-aliases/WebGpuFrameRenderFunction",label:"WebGpuFrameRenderFunction"},{type:"doc",id:"api/custom-video-source/webgpu/variables/FrameCropParams",label:"FrameCropParams"},{type:"doc",id:"api/custom-video-source/webgpu/variables/sampleCamera",label:"sampleCamera"}]}],link:{type:"doc",id:"api/custom-video-source/webgpu/index"}}]};
module.exports = typedocSidebar.items;
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Function: assertWebGpuDeviceSupportsCameraImport()

> **assertWebGpuDeviceSupportsCameraImport**(`device`): `void`

Defined in: [src/webgpu/requiredFeatures.ts:31](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/webgpu/requiredFeatures.ts#L31)

Throws a descriptive error when `device` is missing any feature required to import camera
frames or Fishjam output surfaces on this platform. Called automatically on devices passed as
an override; call it yourself to validate a device early.

## Parameters

| Parameter | Type |
| ------ | ------ |
| `device` | `GPUDevice` |

## Returns

`void`
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Function: computeAspectFillCrop()

> **computeAspectFillCrop**(`sourceWidth`, `sourceHeight`, `targetAspect`, `uv00`, `uv01`, `uv10`, `uv11`): [`FrameCrop`](../interfaces/FrameCrop.md)

Defined in: [src/webgpu/cropUtilities.ts:86](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/webgpu/cropUtilities.ts#L86)

Center-crop of a (sourceWidth × sourceHeight) frame to the target aspect ratio (width/height).
A no-op full-frame crop when the source already has the target aspect. Worklet-safe.

Feed it the upright camera size from the render context:
`computeAspectFillCrop(context.cameraWidth, context.cameraHeight, context.outputWidth / context.outputHeight)`.

The optional `uv00..uv11` scalars form a UV-space transform applied around the frame center
(identity by default).

## Parameters

| Parameter | Type | Default value |
| ------ | ------ | ------ |
| `sourceWidth` | `number` | `undefined` |
| `sourceHeight` | `number` | `undefined` |
| `targetAspect` | `number` | `undefined` |
| `uv00` | `number` | `1` |
| `uv01` | `number` | `0` |
| `uv10` | `number` | `0` |
| `uv11` | `number` | `1` |

## Returns

[`FrameCrop`](../interfaces/FrameCrop.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Function: computeSquareCrop()

> **computeSquareCrop**(`sourceWidth`, `sourceHeight`, `uv00`, `uv01`, `uv10`, `uv11`): [`FrameCrop`](../interfaces/FrameCrop.md)

Defined in: [src/webgpu/cropUtilities.ts:123](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/webgpu/cropUtilities.ts#L123)

Square center-crop of a (sourceWidth × sourceHeight) frame — the usual shape for square model
inputs. Worklet-safe. The optional `uv00..uv11` scalars are as in [computeAspectFillCrop](computeAspectFillCrop.md).

## Parameters

| Parameter | Type | Default value |
| ------ | ------ | ------ |
| `sourceWidth` | `number` | `undefined` |
| `sourceHeight` | `number` | `undefined` |
| `uv00` | `number` | `1` |
| `uv01` | `number` | `0` |
| `uv10` | `number` | `0` |
| `uv11` | `number` | `1` |

## Returns

[`FrameCrop`](../interfaces/FrameCrop.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Function: createCameraBindGroup()

> **createCameraBindGroup**(`device`, `cameraShaderBindings`, `cameraTexture`): `GPUBindGroup`

Defined in: [src/webgpu/cameraShaderBindings.ts:134](https://github.com/fishjam-cloud/web-client-sdk/blob/be6bc1b673827e28bf8df7bed88de7c874f96db7/packages/react-native-custom-video-source/src/webgpu/cameraShaderBindings.ts#L134)

Builds the per-frame bind group for the live camera texture. The source hook already does this
for you when you pass `cameraShaderBindings` in its options (see the render context's
`cameraBindGroup`); call it yourself only for advanced multi-layout setups. Worklet-safe.

A camera texture expires when the frame ends, so a bind group referencing it must be rebuilt
every frame — never cache the result.

## Parameters

| Parameter | Type |
| ------ | ------ |
| `device` | `GPUDevice` |
| `cameraShaderBindings` | [`CameraShaderBindings`](../interfaces/CameraShaderBindings.md) |
| `cameraTexture` | `GPUExternalTexture` |

## Returns

`GPUBindGroup`
Loading
Loading