Skip to content

Commit 6274e79

Browse files
waleedlatif1waleedlatif1
committed
feat(knowledge): export knowledge bases as bundle archives
Adds a streaming zip export for a knowledge base across the v2 API, the internal API, the CLI, and the UI. The bundle carries the base's configuration and tag definitions, each workspace-visible document's original file and chunk text, and optionally the chunk vectors so an import into a deployment with the same embedding model can reuse them. Access-control lists, connector links, credentials, uploader identity, and storage keys never leave. - knowledge.export operation gated by a new Knowledge Base Export permission group setting; exports record an audit event - entries are appended one at a time so a large base never fans out storage reads and the manifest is written last with the counts actually produced - sim knowledge export <knowledgeBaseId> --output-file kb.zip [--no-vectors] - Export in the knowledge base row menu and base page header
1 parent fbcf13e commit 6274e79

43 files changed

Lines changed: 2545 additions & 16 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apps/docs/content/docs/cli/knowledge.mdx‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1099,6 +1099,34 @@ sim knowledge mv <knowledgeBaseId> <folder>
10991099

11001100
</CommandTable>
11011101

1102+
## Export a knowledge base as a .simkb.zip bundle
1103+
1104+
```bash
1105+
sim knowledge export <knowledgeBaseId> [options]
1106+
```
1107+
1108+
**Arguments**
1109+
1110+
<CommandTable>
1111+
1112+
| Argument | Required | Description |
1113+
| --- | --- | --- |
1114+
| `knowledgeBaseId` | Yes | Knowledge base to export |
1115+
1116+
</CommandTable>
1117+
1118+
**Options**
1119+
1120+
<CommandTable>
1121+
1122+
| Option | Required | Description |
1123+
| --- | --- | --- |
1124+
| `-o, --output-file <path>` | No | Write the bundle to this path instead of the name the server suggests; pass - to stream it to stdout. |
1125+
| `--force` | No | Overwrite --output-file if it already exists. |
1126+
| `--no-vectors` | No | Leave chunk vectors out of the bundle, so an import re-embeds every chunk. |
1127+
1128+
</CommandTable>
1129+
11021130
## List knowledge resources and child folders together
11031131

11041132
```bash

‎apps/docs/content/docs/cli/reference.mdx‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2405,6 +2405,36 @@ sim knowledge mv <knowledgeBaseId> <folder>
24052405

24062406
</CommandTable>
24072407

2408+
### sim knowledge export
2409+
2410+
Export a knowledge base as a .simkb.zip bundle
2411+
2412+
```bash
2413+
sim knowledge export <knowledgeBaseId> [options]
2414+
```
2415+
2416+
**Arguments**
2417+
2418+
<CommandTable>
2419+
2420+
| Argument | Required | Description |
2421+
| --- | --- | --- |
2422+
| `knowledgeBaseId` | Yes | Knowledge base to export |
2423+
2424+
</CommandTable>
2425+
2426+
**Options**
2427+
2428+
<CommandTable>
2429+
2430+
| Option | Required | Description |
2431+
| --- | --- | --- |
2432+
| `-o, --output-file <path>` | No | Write the bundle to this path instead of the name the server suggests; pass - to stream it to stdout. |
2433+
| `--force` | No | Overwrite --output-file if it already exists. |
2434+
| `--no-vectors` | No | Leave chunk vectors out of the bundle, so an import re-embeds every chunk. |
2435+
2436+
</CommandTable>
2437+
24082438
### sim knowledge ls
24092439

24102440
List knowledge resources and child folders together

‎apps/docs/content/docs/knowledgebase/export-import.mdx‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
title: Export
3+
description: Download a knowledge base as one archive of its documents, chunks, tags, and settings.
4+
---
5+
6+
An **export** is a single archive that holds a knowledge base: every document, the chunks Sim split it into, the tag definitions and values, and the chunking settings. Use it to keep a copy of a base outside Sim.
7+
8+
To export, right-click a base in the knowledge base list and choose **Export**, or open the base and click the **Export** button in the header. The download is named `<base name>.simkb.zip`.
9+
10+
## What the archive contains
11+
12+
```text
13+
<base name>.simkb.zip
14+
├── manifest.json
15+
├── files/
16+
│ └── <documentId>/<filename>
17+
└── chunks/
18+
└── <documentId>.ndjson
19+
```
20+
21+
- **`manifest.json`** describes the export (format version 1): the base name, description, and chunking config; the embedding model and dimension and whether vectors are included; the tag definitions; and a document list with each document's filename, MIME type, size, enabled flag, tag values, entry paths, and chunk, token, and character counts.
22+
- **`files/`** holds each document's original file, when it has one.
23+
- **`chunks/`** holds one NDJSON file per document, with one JSON line per chunk: its index, content, token count, start and end offsets, enabled flag, and the vector when vectors are included.
24+
25+
The export includes every document a workspace member can read, its file and its chunk text. Documents synced from a [connector](/knowledgebase/connectors) export as plain documents: their text and, where Sim stored it, their file.
26+
27+
## What stays behind
28+
29+
An export never includes:
30+
31+
- Access-control lists
32+
- Connector links and credentials
33+
- Who uploaded each document
34+
- Internal storage keys
35+
36+
Organization-wide search indexes cannot be exported.
37+
38+
## Vectors
39+
40+
Vectors are embeddings, the numbers a model produces so search can compare chunks. They are included by default. Each chunk's vector is stored as base64-encoded float32 values in its NDJSON line.
41+
42+
Vectors are only valid for a deployment that uses the same embedding model and dimension. The manifest records both for the base you exported. To skip vectors, set `vectors=false` on the API or pass `--no-vectors` to the CLI.
43+
44+
## Limits and governance
45+
46+
A base with more than 2,000 documents cannot be exported. The request returns `413`.
47+
48+
Organization admins can withhold export through the permission group setting **Knowledge Base Export** (enterprise). Every export records an audit event.
49+
50+
## API
51+
52+
```http
53+
GET /api/v2/knowledge/{knowledgeBaseId}/export?workspaceId={workspaceId}&vectors=true
54+
```
55+
56+
The response is the archive. See the [API reference](/api-reference/getting-started) for authentication.
57+
58+
## CLI
59+
60+
```bash
61+
sim knowledge export <knowledgeBaseId> --output-file kb.zip
62+
sim knowledge export <knowledgeBaseId> --output-file kb.zip --no-vectors
63+
```

‎apps/docs/content/docs/knowledgebase/meta.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
"connectors",
66
"tags",
77
"debugging-retrieval",
8-
"chunking-strategies"
8+
"chunking-strategies",
9+
"export-import"
910
]
1011
}

‎apps/docs/openapi-v2-knowledge.json‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -502,6 +502,105 @@
502502
}
503503
}
504504
},
505+
"/api/v2/knowledge/{knowledgeBaseId}/export": {
506+
"get": {
507+
"operationId": "exportKnowledgeBase",
508+
"summary": "Export Knowledge Base",
509+
"description": "Stream a knowledge base as a bundle archive: its configuration, tag definitions, each workspace-visible document's file and chunk text, and optionally chunk vectors. Access-control lists, connector links, and credentials never leave. More than 2000 documents returns `413`. Exports record an audit event. `HEAD` checks access with the same authorization as `GET` but skips side effects, returning an empty `200` without payload headers on success. `HEAD` omits `Content-Length`; use file metadata to size downloads.\n\nOAuth scope: `api:read`.",
510+
"x-sim-operation": "knowledge.export",
511+
"x-oauth-scope": "api:read",
512+
"tags": ["Knowledge Bases"],
513+
"parameters": [
514+
{
515+
"name": "knowledgeBaseId",
516+
"in": "path",
517+
"required": true,
518+
"description": "Unique knowledge base identifier.",
519+
"schema": {
520+
"type": "string",
521+
"minLength": 1,
522+
"description": "Unique knowledge base identifier."
523+
}
524+
},
525+
{
526+
"name": "workspaceId",
527+
"in": "query",
528+
"required": true,
529+
"description": "Workspace that owns the knowledge base.",
530+
"schema": {
531+
"type": "string",
532+
"minLength": 1,
533+
"maxLength": 128,
534+
"description": "Workspace that owns the knowledge base."
535+
}
536+
},
537+
{
538+
"name": "vectors",
539+
"in": "query",
540+
"required": false,
541+
"description": "Include chunk vectors so an import into a deployment with the same embedding model reuses them instead of re-embedding.",
542+
"schema": {
543+
"description": "Include chunk vectors so an import into a deployment with the same embedding model reuses them instead of re-embedding.",
544+
"type": "boolean"
545+
}
546+
}
547+
],
548+
"responses": {
549+
"200": {
550+
"description": "The knowledge base as a zip archive.",
551+
"headers": {
552+
"Content-Type": {
553+
"$ref": "#/components/headers/Content-Type"
554+
},
555+
"Content-Disposition": {
556+
"$ref": "#/components/headers/Content-Disposition"
557+
},
558+
"X-RateLimit-Limit": {
559+
"$ref": "#/components/headers/X-RateLimit-Limit"
560+
},
561+
"X-RateLimit-Remaining": {
562+
"$ref": "#/components/headers/X-RateLimit-Remaining"
563+
},
564+
"X-RateLimit-Reset": {
565+
"$ref": "#/components/headers/X-RateLimit-Reset"
566+
}
567+
},
568+
"content": {
569+
"application/zip": {
570+
"schema": {
571+
"type": "string",
572+
"format": "binary"
573+
}
574+
}
575+
}
576+
},
577+
"400": {
578+
"$ref": "#/components/responses/BadRequest"
579+
},
580+
"401": {
581+
"$ref": "#/components/responses/Unauthorized"
582+
},
583+
"403": {
584+
"$ref": "#/components/responses/Forbidden"
585+
},
586+
"404": {
587+
"$ref": "#/components/responses/NotFound"
588+
},
589+
"413": {
590+
"$ref": "#/components/responses/PayloadTooLarge"
591+
},
592+
"429": {
593+
"$ref": "#/components/responses/RateLimited"
594+
},
595+
"500": {
596+
"$ref": "#/components/responses/InternalError"
597+
},
598+
"503": {
599+
"$ref": "#/components/responses/ServiceUnavailable"
600+
}
601+
}
602+
}
603+
},
505604
"/api/v2/knowledge/{knowledgeBaseId}/connectors": {
506605
"get": {
507606
"operationId": "listKnowledgeConnectors",
@@ -4339,6 +4438,31 @@
43394438
}
43404439
},
43414440
"headers": {
4441+
"Content-Type": {
4442+
"description": "MIME type of the file, defaulting to application/octet-stream when the stored type is unavailable.",
4443+
"schema": {
4444+
"type": "string",
4445+
"title": "Content type",
4446+
"description": "MIME type of the file, defaulting to application/octet-stream when the stored type is unavailable."
4447+
}
4448+
},
4449+
"Content-Disposition": {
4450+
"description": "Attachment disposition containing sanitized and RFC 5987 encoded filenames.",
4451+
"schema": {
4452+
"type": "string",
4453+
"title": "Content disposition",
4454+
"description": "Attachment disposition containing sanitized and RFC 5987 encoded filenames."
4455+
}
4456+
},
4457+
"Content-Length": {
4458+
"description": "File size in bytes.",
4459+
"schema": {
4460+
"type": "string",
4461+
"pattern": "^(0|[1-9]\\d*)$",
4462+
"title": "Content length",
4463+
"description": "File size in bytes."
4464+
}
4465+
},
43424466
"X-RateLimit-Limit": {
43434467
"description": "Maximum requests allowed in the current window.",
43444468
"schema": {

0 commit comments

Comments
 (0)