diff --git a/TOC-ai.md b/TOC-ai.md index e4a251b30ec68..09e77c170248a 100644 --- a/TOC-ai.md +++ b/TOC-ai.md @@ -45,20 +45,10 @@ - [Overview](/ai/ti/ti-overview.md) - [Install, Configure, and Update TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md) - [Manage TiDB Cloud Starter Instances](/ai/ti/guides/manage-starter-instances.md) - - Manage TiDB Cloud Filesystem - - [Manage Filesystem Resources](/ai/ti/guides/manage-filesystem-resources.md) - - [Configure Filesystem AI Providers](/ai/ti/guides/configure-filesystem-ai-providers.md) - - [Manage Filesystem Tokens](/ai/ti/guides/manage-filesystem-tokens.md) - - [Work with Filesystem Data](/ai/ti/guides/work-with-filesystem-data.md) - - [Manage Filesystem Layers and Checkpoints](/ai/ti/guides/manage-filesystem-layers.md) - - [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md) - - [Manage Git Workspaces](/ai/ti/guides/manage-git-workspaces.md) - - [Use Filesystem Journals](/ai/ti/guides/use-filesystem-journals.md) - - [Manage Filesystem Vault Secrets](/ai/ti/guides/manage-filesystem-vault-secrets.md) + - [Manage TiDB Cloud Filesystem](/ai/ti/guides/manage-filesystems-via-cli.md) - Scenarios for Users and Automation - [Run a Daily TiDB Cloud CLI Workflow](/ai/ti/guides/ti-daily-workflow-example.md) - [Query TiDB Cloud Starter with Explicit SQL Roles](/ai/ti/guides/ti-query-sql-with-roles-example.md) - - [Share a TiDB Cloud Filesystem Across Machines](/ai/ti/guides/ti-share-filesystem-across-machines-example.md) - [Hand Off CI Artifacts Between Isolated Jobs with TiDB Cloud Filesystem](/ai/ti/guides/ti-ci-artifact-handoff-example.md) - Scenarios for AI Agents - [Use TiDB Cloud Filesystem in an Agent Sandbox](/ai/ti/guides/ti-agent-sandbox-example.md) diff --git a/TOC-tidb-cloud-filesystem.md b/TOC-tidb-cloud-filesystem.md new file mode 100644 index 0000000000000..5475cea7e845b --- /dev/null +++ b/TOC-tidb-cloud-filesystem.md @@ -0,0 +1,39 @@ + + + +# Table of Contents + +## GET STARTED + +- [Overview](/tidb-cloud-filesystem/filesystem-overview.md) +- [Quick Start](/tidb-cloud-filesystem/filesystem-quick-start.md) + +## CONCEPTS + +- [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md) +- [Layers and Checkpoints](/tidb-cloud-filesystem/filesystem-layers-checkpoints.md) + +## GUIDES + +- [Manage TiDB Cloud Filesystem](/tidb-cloud-filesystem/manage-filesystem-resources.md) +- [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md) +- [Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) +- [Manage Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) +- [Share a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-sharing.md) +- Mount a Filesystem + - [Overview](/tidb-cloud-filesystem/filesystem-mount.md) + - [Linux](/tidb-cloud-filesystem/filesystem-mount-linux.md) + - [macOS](/tidb-cloud-filesystem/filesystem-mount-macos.md) + - [Docker and Docker Compose](/tidb-cloud-filesystem/filesystem-mount-docker.md) +- [Manage Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md) +- [Manage Git Workspaces](/tidb-cloud-filesystem/manage-git-workspaces.md) +- [Use Filesystem Journals](/tidb-cloud-filesystem/use-filesystem-journals.md) +- [Manage Filesystem Vault Secrets](/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md) +- [Configure Filesystem AI Providers](/tidb-cloud-filesystem/configure-filesystem-ai-providers.md) +- [Automation and AI Agent Workflows](/tidb-cloud-filesystem/use-filesystem-for-automation-and-ai-agents.md) + +## REFERENCES + +- [Filesystem CLI Commands](/ai/ti/reference/ti-filesystem.md) +- [Regions and Limitations](/tidb-cloud-filesystem/filesystem-regions-and-limitations.md) +- [Troubleshooting](/tidb-cloud-filesystem/filesystem-troubleshooting.md) diff --git a/ai/_index.md b/ai/_index.md index 78b4767293763..039edf068fd3f 100644 --- a/ai/_index.md +++ b/ai/_index.md @@ -8,7 +8,7 @@ summary: Build AI applications and agent workflows with TiDB using SQL, integrat TiDB provides data and workspace capabilities for building AI applications and running AI agent workflows. - For application development, you can use SQL or [Python SDK for TiDB AI (`pytidb`)](https://github.com/pingcap/pytidb) with structured data, vector search, full-text search, hybrid search, and AI-powered retrieval. -- For AI agents and automation, you can use [TiDB Cloud CLI (`ti`)](https://github.com/tidbcloud/ti-cli) to manage TiDB Cloud Starter instances and SQL workflows, and use TiDB Cloud Filesystems as persistent, shared storage across local machines, CI jobs, and ephemeral agent sandboxes. Filesystems also support mounted workspaces, Git workflows, journals, and delegated secrets. +- For AI agents and automation, you can use [TiDB Cloud CLI (`ti`)](https://github.com/tidbcloud/ti-cli) to manage TiDB Cloud Starter instances and SQL workflows, and use [TiDB Cloud Filesystems](/tidb-cloud-filesystem/_index.md) as persistent, shared storage across local machines, CI jobs, and ephemeral agent sandboxes. Filesystems also support mounted workspaces, Git workflows, journals, and delegated secrets. ## Get started @@ -101,8 +101,8 @@ The TiDB Cloud CLI (`ti`) lets users, scripts, CI jobs, and AI agents manage TiD | Understand what `ti` manages and when to use it | [TiDB Cloud CLI Overview](/ai/ti/ti-overview.md) | | Install and configure `ti`, then complete a first workflow | [Get Started with TiDB Cloud CLI](/ai/ti/ti-quick-start.md) | | Automate TiDB Cloud Starter instance, branch, and SQL operations | [Manage TiDB Cloud Starter Instances](/ai/ti/guides/manage-starter-instances.md) | -| Persist and share files across machines, CI jobs, and sandboxes | [Manage Filesystem Resources](/ai/ti/guides/manage-filesystem-resources.md) and [Work with Filesystem Data](/ai/ti/guides/work-with-filesystem-data.md) | -| Use mounted workspaces, Git workspaces, journals, or delegated secrets | [Mount a Filesystem](/ai/ti/guides/mount-filesystem.md), [Manage Git Workspaces](/ai/ti/guides/manage-git-workspaces.md), [Use Filesystem Journals](/ai/ti/guides/use-filesystem-journals.md), and [Manage Filesystem Vault Secrets](/ai/ti/guides/manage-filesystem-vault-secrets.md) | +| Persist and share files across machines, CI jobs, and sandboxes | [Use TiDB Cloud Filesystem with TiDB Cloud CLI](/ai/ti/guides/manage-filesystems-via-cli.md) | +| Use mounted workspaces, Git workspaces, journals, or delegated secrets | [Mount a Filesystem](/tidb-cloud-filesystem/filesystem-mount.md), [Manage Git Workspaces](/tidb-cloud-filesystem/manage-git-workspaces.md), [Use Filesystem Journals](/tidb-cloud-filesystem/use-filesystem-journals.md), and [Manage Filesystem Vault Secrets](/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md) | | Follow an end-to-end automation or agent example | [Run a Daily TiDB Cloud CLI Workflow](/ai/ti/guides/ti-daily-workflow-example.md) or [Use TiDB Cloud Filesystem in an Agent Sandbox](/ai/ti/guides/ti-agent-sandbox-example.md) | | Look up commands, global options, output behavior, and errors | [TiDB Cloud CLI Command Reference](/ai/ti/reference/ti-cli-reference.md) | diff --git a/ai/ti/guides/configure-filesystem-ai-providers.md b/ai/ti/guides/configure-filesystem-ai-providers.md deleted file mode 100644 index 138ee6f720ee4..0000000000000 --- a/ai/ti/guides/configure-filesystem-ai-providers.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -title: Configure TiDB Cloud Filesystem AI Providers -summary: Learn how to inspect and configure media extraction and embedding providers for a TiDB Cloud Filesystem by using the CLI. ---- - -# Configure TiDB Cloud Filesystem AI Providers - -A TiDB Cloud Filesystem can optionally extract text from media files and generate embeddings for stored content. To enable these capabilities, you can configure one or more AI providers through the CLI. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- Obtain the target Filesystem ID. -- Obtain the API key required by your selected AI provider. - -The configuration commands require TiDB Cloud API credentials and an explicit Filesystem ID. Set the provider key through `TI_FS_AI_PROVIDER_API_KEY`. The CLI does not persist the key locally. The Filesystem service stores it encrypted and returns only a masked value in subsequent configuration output. - -For an interactive shell, read and export the provider key without placing it in shell history: - -```bash -printf 'Provider API key: ' >&2 -read -r -s TI_FS_AI_PROVIDER_API_KEY -printf '\n' >&2 -export TI_FS_AI_PROVIDER_API_KEY -``` - -In CI, inject `TI_FS_AI_PROVIDER_API_KEY` from a masked secret. Unset the variable after you finish configuring providers. - -> **Note:** -> -> When you enable, re-enable, or replace a provider configuration, the Filesystem service sends a small built-in request to the provider endpoint to validate the credentials, connectivity, and model response. This validation request might incur a provider charge. A disable-only or prompt-only update does not make a validation request. - -## Inspect media extraction configuration - -Read the effective extraction configuration for a media type: - -```shell -ti fs describe-file-system-extract-configuration \ - --file-system-id "" \ - --media-type image -``` - -## Update media extraction configuration - -Use [`update-file-system-extract-configuration`](/ai/ti/reference/ti-fs-update-file-system-extract-configuration.md) to enable, update, or disable image, audio, or video extraction. For example, configure image extraction through an OpenAI-compatible provider: - -```shell -ti fs update-file-system-extract-configuration \ - --file-system-id "" \ - --media-type image \ - --enabled true \ - --provider-api-base https://api.openai.com/v1 \ - --provider-model "" \ - --provider-protocol openai -``` - -The `openai` protocol supports image, audio, and video extraction. The `qwen-asr` protocol is supported only for audio extraction through Alibaba Cloud Model Studio. An endpoint from another provider might work if it implements the required OpenAI-compatible API contract. Native interfaces for Anthropic, Gemini, Vertex AI, Amazon Bedrock, and Azure OpenAI are not supported. - -To disable extraction for a media type: - -```shell -ti fs update-file-system-extract-configuration \ - --file-system-id "" \ - --media-type image \ - --enabled false -``` - -## Inspect embedding configuration - -Read whether embeddings are managed by the application or database: - -```shell -ti fs describe-file-system-embedding-configuration \ - --file-system-id "" -``` - -## Update embedding configuration - -Use [`update-file-system-embedding-configuration`](/ai/ti/reference/ti-fs-update-file-system-embedding-configuration.md) to update the optional application-managed embedding configuration. For example: - -```shell -ti fs update-file-system-embedding-configuration \ - --file-system-id "" \ - --enabled true \ - --provider-api-base https://api.openai.com/v1 \ - --provider-model text-embedding-3-small -``` - -Application-managed embeddings require an OpenAI-compatible endpoint that returns 1024-dimensional vectors. They are available for Shared Filesystems and Native Filesystems whose effective embedding mode is `fts_only`. If a Native Filesystem uses database-managed automatic embeddings, the service rejects this update and reports `source=database_auto`. - -After you finish configuring providers, remove the key from the current shell: - -```shell -unset TI_FS_AI_PROVIDER_API_KEY -``` - -To disable that configuration: - -```shell -ti fs update-file-system-embedding-configuration \ - --file-system-id "" \ - --enabled false -``` - -## Data flow after configuration - -After you enable extraction, the Filesystem service sends media content to the configured extraction provider. It sends the extracted text or descriptions to the configured embedding provider. Choose provider accounts and retention policies appropriate for your data. - -If an update fails because of a timeout, lost response, or another ambiguous network error, run the matching `describe-file-system-*-configuration` command before retrying. The provider validation request might have succeeded and incurred a charge even if the CLI did not receive the response. - -## What's next - -- [Work with TiDB Cloud Filesystem Data](/ai/ti/guides/work-with-filesystem-data.md) -- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/ai/ti/guides/manage-filesystem-layers.md b/ai/ti/guides/manage-filesystem-layers.md deleted file mode 100644 index 1ece63ff88283..0000000000000 --- a/ai/ti/guides/manage-filesystem-layers.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -title: Manage TiDB Cloud Filesystem Layers and Checkpoints -summary: Learn how to safely create, inspect, fork, checkpoint, roll back, commit, pack, and restore TiDB Cloud Filesystem layers. ---- - -# Manage TiDB Cloud Filesystem Layers and Checkpoints - -Use layers to record isolated changes over a Filesystem base path before you commit or discard them. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- Select a Filesystem by passing `--file-system-id`, setting `TI_FS_FILE_SYSTEM_ID`, or supplying an FS token that identifies the Filesystem. -- Provide an FS token with the required read or write permission by using `--fs-token`, `TI_FS_TOKEN`, or the local credential stored for the selected Filesystem. -- Choose the base path whose data the layer overlays. - -## Create and inspect a layer - -```shell -ti fs create-layer \ - --base-root-path /workspace \ - --layer-name agent-task \ - --durability-mode restore-safe \ - --tag task=review -``` - -Use the returned layer ID to write and inspect changes: - -```shell -ti fs copy-file \ - --from-local ./proposal.md \ - --to-remote /workspace/proposal.md \ - --layer-id "" - -ti fs describe-layer --layer-id "" -ti fs diff-layer --layer-id "" -``` - -> **Note:** -> -> `copy-file` with `--layer-id` does not support recursive copy. To seed a directory tree into a layer, mount the layer as a writable FUSE mount and copy files through the mount path. - -Do not mount the same writable layer at multiple local paths concurrently. Reuse its existing mount, or unmount it before mounting the layer elsewhere. - -## Create a checkpoint and fork a layer - -```shell -ti fs create-layer-checkpoint \ - --layer-id "" \ - --checkpoint-id seed \ - --label "before review" - -ti fs fork-layer \ - --parent-layer-ref "" \ - --layer-name experiment \ - --checkpoint-id seed -``` - -Use `list-layer-chain` to inspect the pinned ancestry of the fork: - -```shell -ti fs list-layer-chain --layer-ref experiment -``` - -A checkpoint mount is read-only. To continue working from a checkpoint, fork a new writable layer from it. - -## Finish work in a layer - -> **Warning:** -> -> Before you create a checkpoint for a layer with a writable FUSE mount, run [`drain-file-system`](/ai/ti/guides/mount-filesystem.md#drain-or-unmount). A checkpoint includes only changes that have reached the service. Before you roll back or commit the layer, drain and then [`unmount-file-system`](/ai/ti/guides/mount-filesystem.md#drain-or-unmount). The CLI does not perform these steps automatically. - -Choose one outcome for a layer: - -- Roll back the layer to discard its changes: - - ```shell - ti fs rollback-layer --layer-id "" - ``` - -- Commit the layer to apply its changes to the base path: - - ```shell - ti fs commit-layer --layer-id "" - ``` - -> **Note:** -> -> Do not run both `rollback-layer` and `commit-layer` in sequence for the same layer. - -## Move local state to another machine - -When a FUSE mount uses write-back cache, some data can remain in its local overlay directory. To move this local state to another machine, pack it to an explicit remote archive path: - -```shell -ti fs pack-file-system \ - --mount-path /path/to/workspace \ - --archive-path /workspace-overlay.tar.gz -``` - -On the destination machine, restore the archive into a local overlay root: - -```shell -ti fs unpack-file-system \ - --local-root /path/to/local-overlay \ - --remote-root /workspace \ - --mount-profile portable \ - --archive-path /workspace-overlay.tar.gz -``` - -Use the same local overlay root when you mount the Filesystem on the destination machine. For all pack and unpack options, see the [`pack-file-system`](/ai/ti/reference/ti-fs-pack-file-system.md) and [`unpack-file-system`](/ai/ti/reference/ti-fs-unpack-file-system.md) references. - -## What's next - -- [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md) -- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/ai/ti/guides/manage-filesystem-resources.md b/ai/ti/guides/manage-filesystem-resources.md deleted file mode 100644 index 0ba5d578fd957..0000000000000 --- a/ai/ti/guides/manage-filesystem-resources.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -title: Manage TiDB Cloud Filesystem Resources -summary: Learn how to safely create, inspect, check, select, and delete TiDB Cloud Filesystem resources by using TiDB Cloud CLI. ---- - -# Manage TiDB Cloud Filesystem Resources - -TiDB Cloud Filesystem is a serverless distributed file system designed for AI agents and automation workloads. It provides a persistent, shareable file namespace that remains available independently of the local machine, sandbox, or CI runner that accesses it. - -You can access files directly through TiDB Cloud CLI commands or mount a Filesystem into a supported environment and work with it like a local file system. This makes it useful for preserving agent state, sharing files across isolated environments, handing off CI artifacts, and maintaining reusable workspaces. - -This document describes how to use [`ti fs` commands](/ai/ti/reference/ti-filesystem.md) to create, inspect, select, and delete Filesystem resources. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- Configure a profile with TiDB Cloud API credentials. -- Install `jq`, or use another JSON processor to capture command output safely. - -## Create a Filesystem - -Create a Filesystem and save the returned ID and one-time owner token in a file that is not world-readable. The `--wait` flag tells the CLI to poll until data-plane access is ready before returning: - -```shell -umask 077 -ti fs create-file-system \ - --display-name agent-workspace \ - --label environment=development \ - --wait > ./filesystem.json - -export TI_FS_FILE_SYSTEM_ID="$(jq -r '.file_system_id' ./filesystem.json)" -export TI_FS_TOKEN="$(jq -r '.fs_token' ./filesystem.json)" -``` - -> **Warning:** -> -> The JSON response includes `fs_token` only once. The CLI also stores this token in its local credential directory automatically. However, if the local storage is lost, you cannot retrieve the token again. Store a backup copy in a secret manager, and then delete `filesystem.json`. - -> **Note:** -> -> Do not put credentials, connection strings, private paths, or personal data in Filesystem labels. - -## List and inspect Filesystems - -List the Filesystems available in the effective region: - -```shell -ti fs list-file-systems --output text -``` - -Read authoritative metadata for one Filesystem: - -```shell -ti fs describe-file-system --file-system-id "" -``` - -If you have access to more than one Filesystem, pass `--file-system-id` explicitly or set the `TI_FS_FILE_SYSTEM_ID` environment variable. The CLI does not automatically select a Filesystem for you. - -## Check access - -Verify resource selection, endpoint resolution, credentials, and companion access: - -```shell -ti fs check-file-system --file-system-id "" -``` - -## Delete a Filesystem - -> **Warning:** -> -> Before deleting a Filesystem, drain and unmount any active local mounts for it. The CLI does not do this automatically. - -Delete a Filesystem by explicit ID: - -```shell -ti fs delete-file-system --file-system-id "" -``` - -Filesystem deletion is asynchronous. After the service accepts the request, the CLI reports the Filesystem status as `deleting` and removes the matching local credential. This output does not mean that remote deletion has finished. - -## What's next - -- [Manage TiDB Cloud Filesystem Tokens](/ai/ti/guides/manage-filesystem-tokens.md) -- [Work with TiDB Cloud Filesystem Data](/ai/ti/guides/work-with-filesystem-data.md) -- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/ai/ti/guides/manage-filesystem-tokens.md b/ai/ti/guides/manage-filesystem-tokens.md deleted file mode 100644 index b7a861d775410..0000000000000 --- a/ai/ti/guides/manage-filesystem-tokens.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Manage TiDB Cloud Filesystem Tokens -summary: Learn how to import, generate, scope, inspect, disable, refresh, and revoke access tokens for a TiDB Cloud Filesystem. ---- - -# Manage TiDB Cloud Filesystem Tokens - -You can use Filesystem tokens to give users or automation access to a TiDB Cloud Filesystem without sharing TiDB Cloud API credentials. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- For owner-token generation and TiDB Cloud-authenticated token management, configure TiDB Cloud API credentials and obtain the Filesystem ID. -- For scoped-token generation or bearer-authenticated token management, obtain an owner FS token. You can pass it through `--fs-token`, set `TI_FS_TOKEN`, or use the local token stored for an explicitly selected Filesystem. - -> **Note:** -> -> To avoid security risks, treat token plaintext as a secret. Token creation and rotation commands return plaintext only once when the token is issued. You cannot retrieve the plaintext later. - -## Import an existing token - -When you run `import-file-system-token`, the CLI validates the token format, extracts the Filesystem ID embedded in it, verifies connectivity by making a remote stat request, and stores the token in the local credential directory: - -```shell -ti fs import-file-system-token --from-file ./fs-token --region aws-us-east-1 -``` - -## Generate a token - -Generate another owner token by using TiDB Cloud API credentials. The CLI does not store the generated token locally by default, so you must capture its one-time plaintext response securely: - -```shell -umask 077 -ti fs generate-file-system-token \ - --file-system-id "" \ - --token-name ci \ - --ttl 24h > ./ci-token.json -``` - -To have the CLI store the generated token locally, add `--store-locally`. Use `--replace` if a different token is already stored for this Filesystem. - -For least-privilege access, generate a path-and-operation-limited token from an owner token: - -```shell -ti fs generate-file-system-scoped-token \ - --file-system-id "" \ - --ttl 24h \ - --allow /workspace:read,list > ./scoped-token.json -``` - -## Inspect and change token status - -List non-secret token metadata: - -```shell -ti fs list-file-system-tokens --file-system-id "" -``` - -Use [`disable-file-system-token`](/ai/ti/reference/ti-fs-disable-file-system-token.md) to suspend a token temporarily and [`enable-file-system-token`](/ai/ti/reference/ti-fs-enable-file-system-token.md) to restore it. - -## Rotate or revoke a token - -Use [`refresh-file-system-token`](/ai/ti/reference/ti-fs-refresh-file-system-token.md) to rotate a token. When you refresh the locally stored token, the CLI automatically updates the local credential file. When you refresh a token provided through `--fs-token` or `TI_FS_TOKEN`, the CLI returns the new token in the command output without storing it. - -> **Note:** -> -> Refresh is non-idempotent. If a request might have succeeded but its response was lost, do not retry with the old token. Instead, generate a new owner token using TiDB Cloud credentials. - -Use [`delete-file-system-token`](/ai/ti/reference/ti-fs-delete-file-system-token.md) to revoke a token permanently. If the deleted token matches the locally stored token, the CLI automatically removes the local credential. - -> **Note:** -> -> Before you rotate, disable, or delete a token used by an active local mount, run [`drain-file-system`](/ai/ti/guides/mount-filesystem.md#drain-or-unmount) and then [`unmount-file-system`](/ai/ti/guides/mount-filesystem.md#drain-or-unmount). The CLI checks for known active mounts and refuses the operation if the token is still in use. - -## What's next - -- [Share a TiDB Cloud Filesystem Across Machines](/ai/ti/guides/ti-share-filesystem-across-machines-example.md) -- [Use TiDB Cloud Filesystem in an Agent Sandbox](/ai/ti/guides/ti-agent-sandbox-example.md) -- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/ai/ti/guides/manage-filesystem-vault-secrets.md b/ai/ti/guides/manage-filesystem-vault-secrets.md deleted file mode 100644 index 4b222e12983b4..0000000000000 --- a/ai/ti/guides/manage-filesystem-vault-secrets.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: Manage TiDB Cloud Filesystem Vault Secrets -summary: Learn how to store, read, delegate, inject, audit, revoke, and mount secrets securely with TiDB Cloud Filesystem Vault. ---- - -# Manage TiDB Cloud Filesystem Vault Secrets - -TiDB Cloud Filesystem Vault lets you store secrets, delegate narrowly scoped and time-limited access to users or agents, and inject credentials into processes without writing plaintext to disk. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- Select a Filesystem by passing `--file-system-id`, setting `TI_FS_FILE_SYSTEM_ID`, or supplying an FS token that identifies the Filesystem. -- For owner operations, provide an owner FS token through `--fs-token`, `TI_FS_TOKEN`, or the local credential stored for the selected Filesystem. - -> **Note:** -> -> To avoid security risks, never print, log, or commit owner or delegated tokens. - -## Create and read a secret - -```shell -ti fs-vault create-secret \ - --secret-name db-prod \ - --field DB_URL=mysql://example \ - --field PASSWORD=@./password.txt - -ti fs-vault read-secret --secret-name db-prod -``` - -> **Note:** -> -> All `read-secret` output formats, including the default JSON format, contain plaintext secret values. Direct the output only to the intended process. - -## Delegate limited access - -Create a short-lived read grant and capture its token: - -```shell -export TI_VAULT_TOKEN="$(ti fs-vault create-grant \ - --agent-id deploy-agent \ - --scope db-prod/DB_URL \ - --permission read \ - --ttl 10m \ - --token-only)" -``` - -Prefer `TI_VAULT_TOKEN` to a command-line token because command-line values can remain in process listings or shell history. - -## Inject a secret into a process - -The CLI can inject secret fields as environment variables into a child process without writing plaintext to disk. When you run the following command, the CLI reads the secret, sets each field as an environment variable (for example, `DB_URL`, `PASSWORD`), removes its own credential environment variables from the child, and then executes the specified command: - -```shell -ti fs-vault run-with-secret --secret-path /n/vault/db-prod -- -``` - -Prefer process injection to writing plaintext to disk. - -Field names injected by `run-with-secret` must match `[A-Z_][A-Z0-9_]*`. The command rejects the entire injection if any field name does not match this pattern or any field value contains an unsupported control character. Use uppercase environment-variable-style names when creating fields that you plan to inject. - -## Audit and revoke access - -```shell -ti fs-vault list-audit-events \ - --secret-name db-prod \ - --agent-id deploy-agent \ - --since 24h \ - --limit 20 - -ti fs-vault delete-grant \ - --grant-id "" \ - --revoked-by operator \ - --reason rotated -``` - -Revocation prevents new authorized operations but cannot erase a value that a process already read. - -## Mount a read-only Vault view - -On macOS or Linux with FUSE support, you can mount a read-only FUSE view of Vault secrets. The CLI creates the mount and serves secret fields as files under the mount path (for example, `/path/to/vault/db-prod/DB_URL`): - -Before mounting, set `TI_VAULT_TOKEN` to a delegated Vault token, such as the token created in [Delegate limited access](#delegate-limited-access). The mount command requires either `TI_VAULT_TOKEN` or `--vault-token`. - -```shell -mkdir -p /path/to/vault -ti fs-vault mount-vault \ - --mount-path /path/to/vault -``` - -Stop any processes that use the mount before you unmount it: - -```shell -ti fs-vault unmount-vault --mount-path /path/to/vault -``` - -Vault mounts are unavailable on Windows. Direct secret reads and process injection do not require a mount. - -## Security recommendations - -- Grant the narrowest field scope and shortest practical TTL. -- Do not store delegated tokens in CLI configuration or operation logs. -- Revoke grants after their tasks finish. - -## What's next - -- [Delegate TiDB Cloud Filesystem Vault Secrets to an Agent](/ai/ti/guides/ti-vault-agent-secrets-example.md) -- [TiDB Cloud Filesystem Vault CLI Command Reference](/ai/ti/reference/ti-filesystem-vault.md) diff --git a/ai/ti/guides/manage-filesystems-via-cli.md b/ai/ti/guides/manage-filesystems-via-cli.md new file mode 100644 index 0000000000000..696fbda84a297 --- /dev/null +++ b/ai/ti/guides/manage-filesystems-via-cli.md @@ -0,0 +1,41 @@ +--- +title: Manage TiDB Cloud Filesystem +summary: Learn how to create a TiDB Cloud Filesystem and write and read a file with `ti fs`, then find guides for other Filesystem tasks. +--- + +# Manage TiDB Cloud Filesystem + +With TiDB Cloud CLI (`ti`), you can create and manage TiDB Cloud Filesystems and work with their files from a terminal or automation workflow. + +## Create and use a Filesystem + +Before you begin, [install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md) with API keys that can create a Filesystem in your organization. Choose a supported Filesystem region when you run `ti configure`. + +Create a Filesystem and wait until it is ready: + +```shell +ti fs create-file-system --display-name my-workspace --wait +``` + +Copy the returned `file_system_id` and replace `` in the following commands. The CLI stores the Filesystem token locally. Treat the returned `fs_token` as a secret; do not share the command output publicly. + +```shell +echo "Hello from my workspace" | ti fs copy-file --file-system-id "" --from-stdin --to-remote /hello.txt +ti fs read-file --file-system-id "" --path /hello.txt +``` + +The read returns `Hello from my workspace`. For supported regions, detailed setup, and cleanup instructions, follow [Get Started with TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-quick-start.md). + +## More Filesystem tasks + +For common tasks, see the following guides in the **TiDB Cloud Filesystem** documentation: + +| What you want to do | Guide | +| --- | --- | +| Create, inspect, or delete a Filesystem | [Manage TiDB Cloud Filesystems](/tidb-cloud-filesystem/manage-filesystem-resources.md) | +| Manage access tokens | [Manage Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) | +| Copy, read, and organize files | [Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) | +| Access files through a local mount | [Mount a Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) | +| Isolate changes with layers and checkpoints | [Manage Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md) | + +For syntax, flags, and output fields, see the [`ti fs` command reference](/ai/ti/reference/ti-filesystem.md). diff --git a/ai/ti/guides/manage-git-workspaces.md b/ai/ti/guides/manage-git-workspaces.md deleted file mode 100644 index ad2ba0ec29752..0000000000000 --- a/ai/ti/guides/manage-git-workspaces.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: Manage Git Workspaces on TiDB Cloud Filesystem -summary: Learn how to clone, hydrate, create linked worktrees, and remove Git workspaces on a mounted TiDB Cloud Filesystem. ---- - -# Manage Git Workspaces on TiDB Cloud Filesystem - -Use `ti fs-git` to accelerate Git workspace setup on a mounted TiDB Cloud Filesystem while continuing to use ordinary Git commands for daily work. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md) through FUSE. -- Select the mounted Filesystem by passing `--file-system-id`, setting `TI_FS_FILE_SYSTEM_ID`, or supplying an FS token that identifies it. Provide an FS token with Git workspace permissions. -- Install Git and configure repository credentials independently. - -## Clone a workspace - -```shell -ti fs-git clone-git-workspace \ - --repo-url https://github.com/pingcap/tidb.git \ - --target-path /path/to/workspace/tidb -``` - -For a large repository, add `--blobless --hydrate background` to make the directory tree available immediately. The CLI starts a background process that downloads clean file content and Git objects after the clone command returns. Use `--hydrate sync` when your workflow requires hydration to finish before the command returns. - -## Hydrate an existing workspace - -If a workspace was cloned with `--blobless`, you can explicitly fetch the missing Git objects by running `hydrate-git-workspace`: - -```shell -ti fs-git hydrate-git-workspace \ - --target-path /path/to/workspace/tidb \ - --timeout 30m -``` - -Hydration fetches missing blob data from the remote repository without discarding your working-tree changes. - -## Add and use a linked worktree - -```shell -ti fs-git add-git-worktree \ - --base-path /path/to/workspace/tidb \ - --worktree-path /path/to/workspace/tidb-feature \ - --branch-name feature-x -``` - -After creation, use ordinary Git commands in the linked worktree. - -## Remove a worktree - -```shell -ti fs-git remove-git-worktree \ - --worktree-path /path/to/workspace/tidb-feature -``` - -The CLI checks for uncommitted changes and rejects the removal if the worktree is dirty. Use `--force` only after you decide that local changes in the worktree can be discarded. - -> **Note:** -> -> Before terminating an ephemeral machine, preserve required changes, remove unused worktrees, and gracefully unmount the Filesystem. - -## What's next - -- [Prepare a Git Workspace for Agents on TiDB Cloud Filesystem](/ai/ti/guides/ti-git-workspace-for-agents-example.md) -- [TiDB Cloud Filesystem Git CLI Command Reference](/ai/ti/reference/ti-filesystem-git.md) diff --git a/ai/ti/guides/mount-filesystem.md b/ai/ti/guides/mount-filesystem.md deleted file mode 100644 index 52a73e39ba801..0000000000000 --- a/ai/ti/guides/mount-filesystem.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: Mount a TiDB Cloud Filesystem -summary: Learn how to safely mount, use, drain, and unmount a TiDB Cloud Filesystem on macOS, Linux, or in a container. ---- - -# Mount a TiDB Cloud Filesystem - -In TiDB Cloud CLI, you can mount a TiDB Cloud Filesystem when an application needs to access remote data through a local filesystem path. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- Select a Filesystem by passing `--file-system-id`, setting `TI_FS_FILE_SYSTEM_ID`, or supplying an FS token that identifies the Filesystem. -- Provide an FS token through `--fs-token`, `TI_FS_TOKEN`, or the local credential stored for the selected Filesystem. -- On Linux, install FUSE3 and provide access to `/dev/fuse`. - -## Choose a mount driver - -| Platform | `--driver auto` | Notes | -|---|---|---| -| macOS | WebDAV | Install macFUSE and select `--driver fuse` for FUSE support. | -| Linux | FUSE | WebDAV mounting is not supported. | -| Windows | Not supported | Use `ti fs` data-plane commands without a mount. | - -## Mount the Filesystem - -On macOS or Linux, create a local path and mount the Filesystem in the background: - -```shell -mkdir -p /path/to/workspace -ti fs mount-file-system \ - --file-system-id "" \ - --mount-path /path/to/workspace -``` - -The CLI starts a background mount process and writes a local mount locator so that the drain and unmount commands can find the correct process. - -Use `--remote-path` to expose a subtree or `--read-only` to prevent writes. To mount a layer or checkpoint, select the FUSE driver and pass the appropriate layer options described in the [`mount-file-system` reference](/ai/ti/reference/ti-fs-mount-file-system.md). - -## Mount in a container - -Installing FUSE3 in an image is not sufficient. The host must expose `/dev/fuse`, and the container must be allowed to perform the mount. For Docker, provide settings equivalent to the following: - -```shell -docker run --rm -it \ - --device /dev/fuse \ - --cap-add SYS_ADMIN \ - --security-opt apparmor=unconfined \ - --env TI_FS_TOKEN \ - --env TI_REGION_CODE \ - --env TI_FS_FILE_SYSTEM_ID \ - -``` - -For Docker Compose, pass the same device, capability, security, and environment settings: - -```yaml -services: - agent: - image: - devices: - - /dev/fuse:/dev/fuse - cap_add: - - SYS_ADMIN - security_opt: - - apparmor=unconfined - environment: - TI_FS_TOKEN: ${TI_FS_TOKEN} - TI_REGION_CODE: ${TI_REGION_CODE} - TI_FS_FILE_SYSTEM_ID: ${TI_FS_FILE_SYSTEM_ID} -``` - -> **Warning:** -> -> `SYS_ADMIN` and an unconfined AppArmor profile weaken container isolation. Use them only for a dedicated, trusted container. When FUSE access is unavailable, use `ti fs` data commands without a mount. - -## Ubuntu 26.04 mount paths - -Ubuntu 26.04 applies an AppArmor profile to `/usr/bin/fusermount3`. By default, use a path under the current user's home directory, `/mnt`, `/media`, `/tmp`, or `/run/user/` instead of `/workspace`. - -For example: - -```shell -mkdir -p "$HOME/workspace" -ti fs mount-file-system \ - --file-system-id "" \ - --mount-path "$HOME/workspace" -``` - -If an application requires `/workspace`, add the following rules to `/etc/apparmor.d/local/fusermount3`: - -```text -mount fstype=@{fuse_types} options=(nosuid,nodev) options in (ro,rw,noatime,dirsync,nodiratime,noexec,sync) -> /workspace/{,**/}, -umount /workspace/{,**/}, -``` - -Then reload the profile: - -```shell -sudo apparmor_parser -r /etc/apparmor.d/fusermount3 -``` - -For related errors, see [Troubleshoot TiDB Cloud CLI](/ai/ti/reference/ti-troubleshooting.md). - -## Drain or unmount - -When you run `unmount-file-system`, the CLI automatically flushes open file handles and pending FUSE work before stopping the mount: - -```shell -ti fs unmount-file-system --mount-path /path/to/workspace -``` - -If you need a durability barrier while keeping a FUSE mount online (for example, before creating a layer checkpoint), run `drain-file-system` explicitly. This command flushes pending writes and waits for them to complete without unmounting: - -```shell -ti fs drain-file-system --mount-path /path/to/workspace --timeout 30s -``` - -> **Note:** -> -> Drain is supported only for FUSE mounts. WebDAV mounts flush writes through normal file close operations. - -> **Warning:** -> -> Do not terminate a machine while writes remain pending or after an unmount returns an error. In-memory writes and local-only overlay files can be lost. For a FUSE mount, run `drain-file-system` before shutdown to confirm that pending writes have reached the remote Filesystem. For a WebDAV mount, close files in the application and verify that `unmount-file-system` succeeds. - -## What's next - -- [Manage TiDB Cloud Filesystem Layers and Checkpoints](/ai/ti/guides/manage-filesystem-layers.md) -- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/ai/ti/guides/ti-agent-sandbox-example.md b/ai/ti/guides/ti-agent-sandbox-example.md index 98e01ecf5d5cb..d1366bb154e4b 100644 --- a/ai/ti/guides/ti-agent-sandbox-example.md +++ b/ai/ti/guides/ti-agent-sandbox-example.md @@ -79,7 +79,7 @@ ti fs mount-file-system \ cat "$HOME/workspace/sandbox/status.txt" ``` -On macOS, omit `--driver fuse` to use WebDAV, which requires no FUSE installation. Install macFUSE and select FUSE when you need FUSE-specific capabilities such as Git workspaces, layers, or online drain. For platform requirements and mount-path restrictions, see [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md). +On macOS, omit `--driver fuse` to use WebDAV, which requires no FUSE installation. Install macFUSE and select FUSE when you need FUSE-specific capabilities such as Git workspaces, layers, or online drain. For platform requirements and mount-path restrictions, see [Mount a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-mount.md). After mounting, you can use `ti fs-git`, `ti fs-journal`, and owner-authorized `ti fs-vault` commands with the same FS environment. Give agents a delegated `TI_VAULT_TOKEN` instead of the owner token when they need only selected secret fields. @@ -91,7 +91,7 @@ Stop writers and unmount. A graceful FUSE unmount automatically flushes and drai ti fs unmount-file-system --mount-path "$HOME/workspace" ``` -For a FUSE mount, use `ti fs drain-file-system --mount-path "$HOME/workspace"` separately when you need to verify remote durability while keeping the mount online. `drain-file-system` is not supported for WebDAV. For more information, see [Drain or unmount](/ai/ti/guides/mount-filesystem.md#drain-or-unmount). Back on the trusted machine: +For a FUSE mount, use `ti fs drain-file-system --mount-path "$HOME/workspace"` separately when you need to verify remote durability while keeping the mount online. `drain-file-system` is not supported for WebDAV. For more information, see [Finish safely](/tidb-cloud-filesystem/filesystem-mount.md#finish-safely). Back on the trusted machine: ```bash ti fs delete-file-system \ diff --git a/ai/ti/guides/ti-ci-artifact-handoff-example.md b/ai/ti/guides/ti-ci-artifact-handoff-example.md index a0a39282375a3..d0ff031a19596 100644 --- a/ai/ti/guides/ti-ci-artifact-handoff-example.md +++ b/ai/ti/guides/ti-ci-artifact-handoff-example.md @@ -17,7 +17,7 @@ The pipeline injects one Filesystem token and region into both jobs. The token i ## Prerequisites -[Create a Filesystem](/ai/ti/guides/manage-filesystem-resources.md#create-a-filesystem) on a trusted machine, and store these values as protected CI secrets or variables: +[Create a Filesystem](/tidb-cloud-filesystem/manage-filesystem-resources.md#create-a-filesystem) on a trusted machine, and store these values as protected CI secrets or variables: ```text TI_FS_TOKEN diff --git a/ai/ti/guides/use-filesystem-journals.md b/ai/ti/guides/use-filesystem-journals.md deleted file mode 100644 index e3ea5e7154873..0000000000000 --- a/ai/ti/guides/use-filesystem-journals.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Use TiDB Cloud Filesystem Journals -summary: Learn how to create, append, read, search, and verify append-only journals for agent and automation events in a Filesystem. ---- - -# Use TiDB Cloud Filesystem Journals - -Journals provide append-only, hash-chained event logs for agent workflows and automation pipelines running on a TiDB Cloud Filesystem. Use [`ti fs-journal` commands](/ai/ti/reference/ti-filesystem-journal.md) to create a journal, append ordered events, search or read them, and verify the hash chain. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- Select a Filesystem by passing `--file-system-id`, setting `TI_FS_FILE_SYSTEM_ID`, or supplying an FS token that identifies the Filesystem. -- Provide an FS token with journal permissions through `--fs-token`, `TI_FS_TOKEN`, or the local credential stored for the selected Filesystem. - -## Create a journal - -```shell -ti fs-journal create-journal \ - --journal-kind agent \ - --title "review task" \ - --actor agent:reviewer -``` - -Save the returned journal ID. - -## Append entries - -```shell -ti fs-journal append-journal-entries \ - --journal-id "" \ - --entry-json '{"type":"review_started"}' -``` - -For supported input forms and entry fields, see the [`append-journal-entries` reference](/ai/ti/reference/ti-fs-journal-append-journal-entries.md). - -## Read and search entries - -Read entries in sequence order: - -```shell -ti fs-journal read-journal-entries --journal-id "" -``` - -Search across journals and entries: - -```shell -ti fs-journal search-journal-entries \ - --entry-type review_started \ - --include-entries -``` - -## Verify a journal - -Verify that the journal's hash chain is intact: - -```shell -ti fs-journal verify-journal --journal-id "" -``` - -## What's next - -- [Record an Agent Workflow in a TiDB Cloud Filesystem Journal](/ai/ti/guides/ti-journal-agent-workflow-example.md) -- [TiDB Cloud Filesystem Journal CLI Command Reference](/ai/ti/reference/ti-filesystem-journal.md) diff --git a/ai/ti/guides/work-with-filesystem-data.md b/ai/ti/guides/work-with-filesystem-data.md deleted file mode 100644 index 8b75a0708ced6..0000000000000 --- a/ai/ti/guides/work-with-filesystem-data.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: Work with TiDB Cloud Filesystem Data -summary: Learn how to copy, read, organize, search, and inspect files and directories in TiDB Cloud Filesystem by using the CLI. ---- - -# Work with TiDB Cloud Filesystem Data - -In TiDB Cloud CLI, you can use [`ti fs` commands](/ai/ti/reference/ti-filesystem.md) to transfer data between local storage and TiDB Cloud Filesystem and to manage its remote namespace. - -## Prerequisites - -- [Install and configure TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). -- [Create a Filesystem](/ai/ti/guides/manage-filesystem-resources.md) or obtain access to an existing one. -- Select the Filesystem by passing `--file-system-id`, setting `TI_FS_FILE_SYSTEM_ID`, or supplying an FS token that identifies it. Provide an FS token with the permissions required for each operation. - -## Copy data - -Upload a local file to a remote path: - -```shell -ti fs copy-file --from-local ./report.md --to-remote /reports/report.md -``` - -[`copy-file`](/ai/ti/reference/ti-fs-copy-file.md) also supports downloads, streaming, appending, resuming, and recursive copies. - -## Read and inspect data - -Read a file or byte range to standard output: - -```shell -ti fs read-file --path /reports/report.md --offset 0 --length 1024 -``` - -List a directory and inspect one path: - -```shell -ti fs list-files --path /reports --output text -ti fs describe-file --path /reports/report.md -``` - -## Organize the namespace - -Create a directory, move a file, and remove data with the corresponding commands: - -```shell -ti fs create-directory --path /reports/archive -ti fs move-file --from-remote /draft.md --to-remote /reports/final.md -ti fs delete-file --path /scratch --recursive -``` - -You can also use `chmod-file`, `create-symlink`, and `create-hardlink` to manage POSIX-style metadata and links. - -> **Warning:** -> -> `delete-file --recursive` permanently deletes the target directory and its contents. Verify the remote path before you run the command. - -## Search for data - -Search file content below a path: - -```shell -ti fs search-file-content --path /reports --pattern "TODO" -``` - -Find paths by name, type, tags, size, or timestamps: - -```shell -ti fs find-files --path /reports --file-name-pattern "*.md" --tag stage=review -``` - -## What's next - -- [Manage TiDB Cloud Filesystem Layers and Checkpoints](/ai/ti/guides/manage-filesystem-layers.md) -- [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md) -- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/ai/ti/reference/ti-filesystem-git.md b/ai/ti/reference/ti-filesystem-git.md index d5567ac92c7cb..52de33fe513e0 100644 --- a/ai/ti/reference/ti-filesystem-git.md +++ b/ai/ti/reference/ti-filesystem-git.md @@ -18,5 +18,5 @@ summary: Reference every `ti fs-git` command for cloning, hydrating, and managin ## See also -- [Manage Git Workspaces on TiDB Cloud Filesystem](/ai/ti/guides/manage-git-workspaces.md) +- [Manage Git Workspaces on TiDB Cloud Filesystem](/tidb-cloud-filesystem/manage-git-workspaces.md) - [Prepare a Git Workspace for Agents on TiDB Cloud Filesystem](/ai/ti/guides/ti-git-workspace-for-agents-example.md) diff --git a/ai/ti/reference/ti-filesystem-journal.md b/ai/ti/reference/ti-filesystem-journal.md index 9e721d7129301..8dd50b49c05ea 100644 --- a/ai/ti/reference/ti-filesystem-journal.md +++ b/ai/ti/reference/ti-filesystem-journal.md @@ -19,5 +19,5 @@ summary: Reference every `ti fs-journal` command for creating, appending, readin ## See also -- [Use TiDB Cloud Filesystem Journals](/ai/ti/guides/use-filesystem-journals.md) +- [Use TiDB Cloud Filesystem Journals](/tidb-cloud-filesystem/use-filesystem-journals.md) - [Record an Agent Workflow in a TiDB Cloud Filesystem Journal](/ai/ti/guides/ti-journal-agent-workflow-example.md) diff --git a/ai/ti/reference/ti-filesystem-vault.md b/ai/ti/reference/ti-filesystem-vault.md index cece96304ca50..7dc479df1d4f2 100644 --- a/ai/ti/reference/ti-filesystem-vault.md +++ b/ai/ti/reference/ti-filesystem-vault.md @@ -27,5 +27,5 @@ Most secret-management commands identify a secret by its name, such as `db-prod` ## See also -- [Manage TiDB Cloud Filesystem Vault Secrets](/ai/ti/guides/manage-filesystem-vault-secrets.md) +- [Manage TiDB Cloud Filesystem Vault Secrets](/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md) - [Delegate TiDB Cloud Filesystem Vault Secrets to an Agent](/ai/ti/guides/ti-vault-agent-secrets-example.md) diff --git a/ai/ti/reference/ti-filesystem.md b/ai/ti/reference/ti-filesystem.md index efa09ce87b292..9408dce5b9c6d 100644 --- a/ai/ti/reference/ti-filesystem.md +++ b/ai/ti/reference/ti-filesystem.md @@ -124,12 +124,12 @@ Aliases use the same options, authentication, output, query, and error behavior ## See also -- [Manage TiDB Cloud Filesystem Resources](/ai/ti/guides/manage-filesystem-resources.md) -- [Configure TiDB Cloud Filesystem AI Providers](/ai/ti/guides/configure-filesystem-ai-providers.md) -- [Manage TiDB Cloud Filesystem Tokens](/ai/ti/guides/manage-filesystem-tokens.md) -- [Work with TiDB Cloud Filesystem Data](/ai/ti/guides/work-with-filesystem-data.md) -- [Manage Filesystem Layers and Checkpoints](/ai/ti/guides/manage-filesystem-layers.md) -- [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md) -- [Manage Git Workspaces on TiDB Cloud Filesystem](/ai/ti/guides/manage-git-workspaces.md) -- [Use TiDB Cloud Filesystem Journals](/ai/ti/guides/use-filesystem-journals.md) -- [Manage TiDB Cloud Filesystem Vault Secrets](/ai/ti/guides/manage-filesystem-vault-secrets.md) +- [Manage TiDB Cloud Filesystems](/tidb-cloud-filesystem/manage-filesystem-resources.md) +- [Configure TiDB Cloud Filesystem AI Providers](/tidb-cloud-filesystem/configure-filesystem-ai-providers.md) +- [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) +- [Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) +- [Manage Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md) +- [Mount a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) +- [Manage Git Workspaces on TiDB Cloud Filesystem](/tidb-cloud-filesystem/manage-git-workspaces.md) +- [Use TiDB Cloud Filesystem Journals](/tidb-cloud-filesystem/use-filesystem-journals.md) +- [Manage TiDB Cloud Filesystem Vault Secrets](/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md) diff --git a/ai/ti/reference/ti-install-configure-update.md b/ai/ti/reference/ti-install-configure-update.md index 1c5335c6e482a..2bbe56f17f2da 100644 --- a/ai/ti/reference/ti-install-configure-update.md +++ b/ai/ti/reference/ti-install-configure-update.md @@ -171,7 +171,7 @@ ti update --dry-run > ti fs unmount-file-system --mount-path > ``` > -> For a Vault mount, use `ti fs-vault unmount-vault --mount-path `. For details, see [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md) and [Manage Filesystem Vault Secrets](/ai/ti/guides/manage-filesystem-vault-secrets.md). +> For a Vault mount, use `ti fs-vault unmount-vault --mount-path `. For details, see [Mount a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) and [Manage Filesystem Vault Secrets](/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md). Apply the latest update: @@ -209,7 +209,7 @@ ti fs unmount-file-system --mount-path ti fs-vault unmount-vault --mount-path ``` -For details, see [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md) and [Manage Filesystem Vault Secrets](/ai/ti/guides/manage-filesystem-vault-secrets.md). +For details, see [Mount a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) and [Manage Filesystem Vault Secrets](/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md). diff --git a/ai/ti/reference/ti-migrate-from-tdc.md b/ai/ti/reference/ti-migrate-from-tdc.md index 7a87e1e49a59b..042b60d7e37f6 100644 --- a/ai/ti/reference/ti-migrate-from-tdc.md +++ b/ai/ti/reference/ti-migrate-from-tdc.md @@ -92,5 +92,5 @@ During the v0.2.x transition, `ti` accepts a legacy `TDC_*` environment variable - [Install, Configure, and Update TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md) - [TiDB Cloud CLI Configuration and Credentials](/ai/ti/reference/ti-configuration-and-credentials.md) - [Manage TiDB Cloud Starter Instances](/ai/ti/guides/manage-starter-instances.md) -- [Manage Filesystem Resources](/ai/ti/guides/manage-filesystem-resources.md) +- [Manage Filesystem Resources](/tidb-cloud-filesystem/manage-filesystem-resources.md) - [Troubleshoot TiDB Cloud CLI](/ai/ti/reference/ti-troubleshooting.md) diff --git a/ai/ti/reference/ti-regions-security-and-limitations.md b/ai/ti/reference/ti-regions-security-and-limitations.md index 0be90892f8d60..94edb15819e11 100644 --- a/ai/ti/reference/ti-regions-security-and-limitations.md +++ b/ai/ti/reference/ti-regions-security-and-limitations.md @@ -1,11 +1,11 @@ --- title: TiDB Cloud CLI Regions, Security, and Limitations -summary: Reference supported regions, authentication boundaries, platform dependencies, preview constraints, and Filesystem companion behavior. +summary: Reference supported regions, authentication boundaries, security best practices, and current TiDB Cloud CLI limitations. --- # TiDB Cloud CLI Regions, Security, and Limitations -This reference describes current placement, authentication, platform, and preview boundaries. +This reference describes current regions, authentication, platform, and preview boundaries for TiDB Cloud CLI. For Filesystem regions and limitations, see [TiDB Cloud Filesystem Regions and Limitations](/tidb-cloud-filesystem/filesystem-regions-and-limitations.md). > **Note:** > @@ -51,56 +51,22 @@ TiDB Cloud API calls use Digest authentication. SQL HTTPS execution uses generat - Create TiDB Cloud API keys with only the access required for the workflow. Do not reuse a personal administrator key in unattended automation. - Inject automation credentials from a CI secret store or runtime secret manager. Do not place credentials in source control, container images, shell scripts, or command-line arguments that can appear in process listings and shell history. - Do not copy the complete `~/.ti/` directory into an agent sandbox. For an existing Filesystem, pass only `TI_FS_TOKEN` and `TI_REGION_CODE`; use `TI_FS_FILE_SYSTEM_ID` only as an optional assertion. -- Treat an FS owner token as full access to that Filesystem. When an agent needs only selected secrets, create a vault grant with the narrowest field scope and shortest practical TTL, and pass the delegated vault token instead. -- Use a separate Filesystem token for each machine, CI workflow, or sandbox class so that one environment can be disabled or revoked without interrupting others. Token names are operational labels, not unique identifiers; mutate tokens only by `token_id`. -- Capture generated and refreshed token plaintext immediately because it is returned only once. A token refreshed from `TI_FS_TOKEN` is not written back to an external secret manager. Refresh is non-idempotent, so do not retry after an ambiguous network failure. -- For shared-token rotation, generate and distribute a replacement, validate access, then disable and delete the old token. Allow approximately 10 seconds for authentication caches to converge after a state change. -- Pass an AI provider key only through `TI_FS_AI_PROVIDER_API_KEY`. The TiDB Cloud CLI does not persist this value locally, and the Filesystem service returns it only in masked form. Do not retry an AI configuration update after an ambiguous failure until you describe the effective configuration. -- Enabling extraction shares Filesystem media with the configured extraction provider. Enabling app-managed embedding shares text or extracted descriptions with the configured embedding provider. Review that provider's data retention and security terms before enabling either feature. - Use `--read-only` for SQL inspection by untrusted or exploratory agents. Use `--admin` only for DDL or privilege management, and use `--read-write` only when data changes are intended. - Use `--dry-run` before destructive control-plane operations. Keep `~/.ti/credentials`, resource credentials, and DB SQL credentials owner-readable only. -- Grant Docker access to `/dev/fuse`, `SYS_ADMIN`, and an unconfined AppArmor profile only to dedicated, trusted containers. These settings reduce container isolation. - Review local operation logs before sharing diagnostics. The logs exclude SQL text, paths, payloads, and credential values, but command names, flag names, profile and region metadata, status codes, and operational timing can still be sensitive. -## Mount platform limitations - -| Platform | Filesystem mount | Vault mount | Requirements and alternatives | -| --- | --- | --- | --- | -| macOS | WebDAV by default; FUSE with explicit `--driver fuse` | FUSE | The built-in WebDAV helper supports Filesystem mounts. Install macFUSE and approve its system extension for FUSE or Vault mounts. | -| Linux | FUSE | FUSE | Install FUSE3 and provide access to `/dev/fuse`. WebDAV mounting is not supported. | -| Windows | Not supported | Not supported | Use `ti fs` data-plane commands and non-mount Vault commands instead. | - -FUSE and WebDAV are implemented by the bundled [Drive9](https://github.com/mem9-ai/drive9) companion. The TiDB Cloud CLI does not fall back to a separate native mount implementation. - -Ubuntu 26.04 additionally confines `fusermount3` with AppArmor. Use a mount path under `$HOME` or `/mnt`; `/workspace` requires an explicit local AppArmor rule even when `ti` runs as root. - -## Durability limitations - -- Default FUSE behavior uses local buffering and asynchronous remote work where permitted by the companion. -- A successful `unmount-file-system` gracefully flushes and drains FUSE work; a separate drain is not required first. -- `drain-file-system` is a FUSE-only online durability barrier that leaves the mount active. -- Abruptly killing the mount process or deleting a machine can lose uncommitted memory/write-back state. -- The default coding-agent mount profile stores dependency trees, generated output, caches, and Git internals locally. Local-only data disappears when its disk disappears unless it is packed or otherwise preserved. -- A running mount remains on the companion version loaded at mount time. Unmount and remount after updating the TiDB Cloud CLI. -- Remote-committed Filesystem data survives client or sandbox deletion; deleting the machine does not delete the remote resource. +For Filesystem token, mount, Vault, and AI provider security, see [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md), [Manage Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md), and [Configure Filesystem AI Providers](/tidb-cloud-filesystem/configure-filesystem-ai-providers.md). ## Product limitations - The TiDB Cloud CLI is in preview, and command contracts can change. -- Database management targets TiDB Cloud Starter, not every TiDB Cloud cluster tier. +- Database management targets TiDB Cloud Starter instances, not other TiDB Cloud database plans. - SQL execution accepts one statement per invocation. - Read-write is the default SQL role; use explicit role flags in security-sensitive automation. -- Journals are append-only and the current public command surface has no journal delete command. -- Filesystem list and describe commands query the region-scoped remote inventory with TiDB Cloud credentials. They do not aggregate across regions. -- The local credential store keeps one selected token per profile and Filesystem. It does not mirror all remote tokens. Older create/import credentials without a known token ID remain usable, but cannot be correlated with remote token metadata. -- Filesystem extraction and embedding provider configuration is optional. Leaving it unconfigured does not block resource administration, file access, search, layers, Git, journal, vault, or mount workflows. -- OpenAI provider interfaces are supported for embedding and image, audio, and video extraction. Alibaba Cloud Model Studio Qwen ASR is supported only for audio extraction. Other vendors are conditionally compatible only through the exact OpenAI-compatible contract; native Anthropic, Gemini, Vertex AI, Bedrock, and Azure OpenAI interfaces are not supported. -- App-managed embedding requires a provider model that returns exactly 1024 dimensions. Filesystems that report `source=database_auto` use database-managed embedding and reject app-managed configuration. - Telemetry management commands are intentionally not implemented. Control telemetry through `~/.ti/.preferences` or `TI_TELEMETRY`; serverless-function deployment, Homebrew, and Scoop distribution are not implemented. -- The TiDB Cloud CLI depends on its installed `ti-drive9` companion for all public Filesystem runtime behavior, including direct file operations, layers, mounts, Git workspaces, journals, and Vault operations. ## Related documentation -- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) +- [TiDB Cloud Filesystem Regions and Limitations](/tidb-cloud-filesystem/filesystem-regions-and-limitations.md) - [TiDB Cloud CLI Configuration and Credentials](/ai/ti/reference/ti-configuration-and-credentials.md) - [Troubleshoot TiDB Cloud CLI](/ai/ti/reference/ti-troubleshooting.md) diff --git a/ai/ti/reference/ti-troubleshooting.md b/ai/ti/reference/ti-troubleshooting.md index b8f35127d3be8..9ffe01f136643 100644 --- a/ai/ti/reference/ti-troubleshooting.md +++ b/ai/ti/reference/ti-troubleshooting.md @@ -1,11 +1,11 @@ --- title: Troubleshoot TiDB Cloud CLI -summary: Diagnose TiDB Cloud CLI authentication, project, Filesystem selection, companion, quota, SQL user, mount, and interrupted-cleanup failures. +summary: Learn how to diagnose TiDB Cloud CLI API authentication, Starter quota, SQL credential, and interrupted-command failures safely. --- # Troubleshoot TiDB Cloud CLI -Use this reference to diagnose common current TiDB Cloud CLI failures. Add `--debug` only when needed; debug output is redacted but should still be reviewed before sharing. +Use this reference to diagnose CLI authentication, Starter, SQL, and interrupted-command failures. For Filesystem tokens, regions, companion processes, and mounts, see [Troubleshoot TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-troubleshooting.md). Add `--debug` only when needed; review redacted output before sharing it. > **Note:** > @@ -31,107 +31,12 @@ ti db list-db-clusters --db-cluster-type starter --profile default An API key can authenticate successfully but still lack the permission declared by a command. Use a key with the access required by that operation. `ti configure` validates and stores local values without contacting TiDB Cloud, so credential failures first appear on a remote command. -## Filesystem token is missing +## Starter creation reaches quota -For a clean sandbox, provide the token and region. `ti` derives the file system ID from the token: - -```bash -export TI_FS_TOKEN="" -export TI_REGION_CODE="" -ti fs check-file-system -``` - -The FS token is not the TiDB Cloud API private key. `TI_FS_FILE_SYSTEM_ID` is optional when a token is supplied; set it only when you want `ti` to verify that a separately distributed ID matches the token. - -If the token is known but is not stored on the current machine, import it and then select the derived ID: - -```bash -# Store a known token without requiring TiDB Cloud API keys. -chmod 600 ./fs-token -ti fs import-file-system-token --from-file ./fs-token --region -ti fs list-files --file-system-id --path / -``` - -If every known token is lost or revoked, use TiDB Cloud API keys to generate another owner token: - -```bash -ti fs generate-file-system-token \ - --file-system-id "" \ - --token-name recovery \ - --ttl 24h -``` - -The new plaintext appears once in the response. Store it securely or add `--store-locally` to select it on the current machine. - -## Filesystem token is rejected - -A data-plane HTTP 401 cannot distinguish a token that was disabled, expired, refreshed on another machine, or revoked. Inspect remote metadata with TiDB Cloud API keys: - -```bash -ti fs list-file-system-tokens \ - --file-system-id "" \ - --include-expired \ - --output text -``` - -Token names are not unique. Use the immutable `token_id` from this output for enable, disable, or delete operations. Old credentials created or imported without token lifecycle metadata can remain valid, but `ti` cannot safely identify their list row and never guesses a match. - -After enable, disable, delete, or refresh, allow approximately 10 seconds for authentication caches to converge. If refresh reports `fs.token_refresh_ambiguous`, the server might have rotated the token even though the response was lost. The outcome is unknown: the old token might still work if the refresh did not commit, or it might already be invalid. The replacement token from a committed refresh cannot be recovered because its response was lost. Do not retry the refresh with the old token. Instead, use TiDB Cloud credentials to generate an independent owner token. - -If token mutation reports `fs.token_mount_active`, use the exact mount path in the error: - -```bash -ti fs drain-file-system --mount-path /path/to/workspace -ti fs unmount-file-system --mount-path /path/to/workspace -``` - -Then retry the token operation. A mount on another machine is not visible locally; coordinate rotation with that machine separately. - -## Filesystem selection is missing - -List remote resources in the configured region with TiDB Cloud API keys and select one explicitly: - -```bash -ti fs list-file-systems --output text -ti fs list-files --file-system-id --path / -``` - -Or select the Filesystem for subsequent commands in the current shell: - -```bash -export TI_FS_FILE_SYSTEM_ID="" -``` - -The TiDB Cloud CLI intentionally does not infer a Filesystem from local credential count, including when only one credential exists. Supply its ID or an FS token whose embedded ID can be derived. - -## Filesystem region is unsupported - -The configured TiDB Cloud region might not be one of the Filesystem endpoints built into the installed TiDB Cloud CLI release. Compare it with [current Filesystem regions](/ai/ti/reference/ti-regions-security-and-limitations.md#supported-regions). Change placement with a valid profile or command-scoped `--region`; do not configure a raw server URL. - -## Companion is missing or incompatible - -The release installer places `ti-drive9`, the companion runtime for Filesystem commands, next to `ti`. You do not invoke `ti-drive9` directly. Re-run the current installer when the TiDB Cloud CLI reports a missing companion: - -```bash -curl -fsSL https://github.com/tidbcloud/ti-cli/releases/latest/download/install.sh | sh -s -- --yes -``` - -Verify that `PATH` resolves the expected `ti`: - -```bash -command -v ti -ti --version -``` - -Do not copy an arbitrary standalone Drive9 binary into place. - -## Starter or Filesystem creation reaches quota - -Quota and capacity errors can mean the organization has reached its free Starter limit. List existing resources before creating another: +Quota and capacity errors can mean the organization has reached its free Starter limit. List existing Starter resources before creating another: ```bash ti db list-db-clusters --db-cluster-type starter --output text -ti fs list-file-systems --output text ``` Never delete an unrelated resource to make automation pass. A Starter spending limit can require configured billing. @@ -155,72 +60,6 @@ ti db execute-sql-statement \ Deleting `~/.ti/db_users//credentials` removes local passwords. Run the create/repair command rather than inventing credentials. -## Mount does not become ready - -Background mount success prints the TiDB Cloud CLI result without Drive9 startup messages. If startup fails or times out, inspect the companion log path in the error. Confirm: - -- the mount path exists and is writable; -- no existing mount covers the path; -- the FS token and region are valid; -- FUSE prerequisites or the WebDAV helper are installed; -- the remote region is reachable. - -macOS defaults to WebDAV. To request FUSE after installing macFUSE: - -```bash -ti fs mount-file-system \ - --mount-path /path/to/workspace \ - --driver fuse -``` - -Linux needs FUSE3 and access to `/dev/fuse`. Filesystem and Vault mounts are not supported on Windows; use `ti fs` data-plane commands or non-mount Vault commands instead. - -## Ubuntu 26.04 rejects a FUSE mount under `/workspace` - -Ubuntu 26.04 applies an AppArmor profile to `fusermount3`. Its default mount-path allowlist does not include `/workspace`, so root and non-root users can both receive: - -```text -/usr/bin/fusermount3: mount failed: Permission denied -``` - -Confirm the denial: - -```bash -sudo journalctl -k --since "10 minutes ago" | - grep 'profile="fusermount3"' -``` - -An entry with `operation="mount"`, `name="/workspace/"`, and `info="failed mntpnt match"` identifies this restriction. Mount under `$HOME` or `/mnt` instead: - -```bash -mkdir -p "$HOME/workspace" -ti fs mount-file-system --mount-path "$HOME/workspace" -``` - -Changing the owner or mode of `/workspace` does not bypass AppArmor. If the path cannot change, add explicit `/workspace` mount and unmount rules to `/etc/apparmor.d/local/fusermount3` as described in [Mount a TiDB Cloud Filesystem](/ai/ti/guides/mount-filesystem.md#ubuntu-2604-mount-paths). - -## Mount becomes stale after a process crash - -If the companion is killed without graceful unmount, FUSE access can return `EIO` or `Transport endpoint is not connected`. Stop processes with open files, then try: - -```bash -ti fs unmount-file-system \ - --mount-path /path/to/workspace \ - --force -``` - -Use `--ignore-absent` when cleanup should succeed if no locator remains. Abrupt cleanup cannot guarantee recovery of pending writes from a deleted local disk. - -## Unmount reports busy - -Close editors, shells whose working directory is inside the mount, and other open file handles, and then retry: - -```bash -ti fs unmount-file-system --mount-path /path/to/workspace -``` - -Unmount performs the graceful FUSE drain automatically. Running `drain-file-system` separately does not close file descriptors or resolve a busy mount; use it only when you need to flush pending work while leaving the mount online. Drain is not supported for WebDAV. - ## An interrupted command leaves resources List resources and identify only those created by your workflow. Use describe before delete: diff --git a/ai/ti/ti-overview.md b/ai/ti/ti-overview.md index ef567bfc2b802..0b7a5a6d7d08d 100644 --- a/ai/ti/ti-overview.md +++ b/ai/ti/ti-overview.md @@ -84,7 +84,7 @@ If you are new to the TiDB Cloud CLI, start with the [Quick Start](/ai/ti/ti-qui Then continue based on what you want to do: - [Manage TiDB Cloud Starter Instances](/ai/ti/guides/manage-starter-instances.md) -- [Manage TiDB Cloud Filesystems](/ai/ti/guides/manage-filesystem-resources.md) +- [Use TiDB Cloud Filesystem with TiDB Cloud CLI](/ai/ti/guides/manage-filesystems-via-cli.md) - **Follow end-to-end workflows**: Start with [Run a Daily TiDB Cloud CLI Workflow](/ai/ti/guides/ti-daily-workflow-example.md) or [Use TiDB Cloud Filesystem in an Agent Sandbox](/ai/ti/guides/ti-agent-sandbox-example.md) - **Look up a specific command**: Check the [TiDB Cloud CLI Command Reference](/ai/ti/reference/ti-cli-reference.md) - **See what’s new in TiDB Cloud CLI**: Check the [TiDB Cloud CLI (`ti`) Release Notes](https://github.com/tidbcloud/ti-cli/releases) diff --git a/ai/ti/ti-quick-start.md b/ai/ti/ti-quick-start.md index 37fd7198be9ca..c39578746759b 100644 --- a/ai/ti/ti-quick-start.md +++ b/ai/ti/ti-quick-start.md @@ -192,6 +192,6 @@ A TiDB Cloud Filesystem is a persistent, shareable cloud file system that you ca ## What's next - Read the [TiDB Cloud Command Line Interface Overview](/ai/ti/ti-overview.md) to understand what `ti` manages and when to use it. -- Follow the task guides to manage [TiDB Cloud Starter](https://docs.pingcap.com/tidbcloud/select-cluster-tier/?plan=starter#starter) or [Filesystem resources](/ai/ti/guides/manage-filesystem-resources.md). +- Follow the task guides to manage [TiDB Cloud Starter](https://docs.pingcap.com/tidbcloud/select-cluster-tier/?plan=starter#starter) or [Filesystem resources](/tidb-cloud-filesystem/manage-filesystem-resources.md). - Explore the [TiDB Cloud CLI Command Reference](/ai/ti/reference/ti-cli-reference.md) for command groups, global options, and shared CLI behavior. - Learn about [TiDB Cloud CLI Configuration and Credentials](/ai/ti/reference/ti-configuration-and-credentials.md) to set up multiple profiles or non-interactive authentication. diff --git a/tidb-cloud-filesystem/_index.md b/tidb-cloud-filesystem/_index.md new file mode 100644 index 0000000000000..5a926ca3d3a9b --- /dev/null +++ b/tidb-cloud-filesystem/_index.md @@ -0,0 +1,66 @@ +--- +title: TiDB Cloud Filesystem Documentation +hide_sidebar: true +hide_commit: true +summary: TiDB Cloud Filesystem provides persistent, shared file storage for applications, automation, and AI agents across machines and sessions. +--- + + + + + +[TiDB Cloud Filesystem Overview](/tidb-cloud-filesystem/filesystem-overview.md) + +[Layers and Checkpoints](/tidb-cloud-filesystem/filesystem-layers-checkpoints.md) + + + + + +[Quick Start](/tidb-cloud-filesystem/filesystem-quick-start.md) + + + + + +[Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md) + +[Share a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-sharing.md) + +[Mount a Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) + + + + + +[Manage TiDB Cloud Filesystem](/tidb-cloud-filesystem/manage-filesystem-resources.md) + +[Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) + +[Manage Git Workspaces](/tidb-cloud-filesystem/manage-git-workspaces.md) + +[Automation and AI Agent Workflows](/tidb-cloud-filesystem/use-filesystem-for-automation-and-ai-agents.md) + + + + + +[Authorization](/tidb-cloud-filesystem/filesystem-authorization.md) + +[Manage Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) + +[Manage Filesystem Vault Secrets](/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md) + + + + + +[Filesystem CLI Commands](/ai/ti/reference/ti-filesystem.md) + +[Regions and Limitations](/tidb-cloud-filesystem/filesystem-regions-and-limitations.md) + +[Troubleshooting](/tidb-cloud-filesystem/filesystem-troubleshooting.md) + + + + diff --git a/tidb-cloud-filesystem/access-filesystem.md b/tidb-cloud-filesystem/access-filesystem.md new file mode 100644 index 0000000000000..1264cbc39c883 --- /dev/null +++ b/tidb-cloud-filesystem/access-filesystem.md @@ -0,0 +1,96 @@ +--- +title: Access an Existing TiDB Cloud Filesystem +summary: Learn how to access an existing TiDB Cloud Filesystem from your current machine, another machine, CI job, or agent environment. +--- + +# Access an Existing TiDB Cloud Filesystem + +If a TiDB Cloud Filesystem already exists, you can access it from your current machine or another environment. + +How you connect depends on where you are working: + +- If you created or imported a Filesystem on the current machine, TiDB Cloud CLI (`ti`) can use the token already stored locally. +- If you are working from another machine, CI job, or agent environment, provide a Filesystem token and region for that environment. + +## Prerequisites + +Before you begin, [install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli) and make sure you have access to the Filesystem. For information about owner and scoped tokens and their permissions, see [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md). + +## Continue using a Filesystem on the same machine + +If you created the Filesystem by using `ti` on the current machine, or previously imported its token, the CLI already has a token stored locally. + +Select the Filesystem for the current shell: + +```shell +export TI_FS_FILE_SYSTEM_ID="" +``` + +You can then run Filesystem commands without providing the Filesystem ID or token each time: + +```shell +ti fs list-files --path / +``` + +Setting `TI_FS_FILE_SYSTEM_ID` selects the Filesystem for subsequent commands in the current shell. It does not change or revoke any Filesystem tokens. + +Alternatively, you can select the Filesystem for an individual command: + +```shell +ti fs list-files \ + --file-system-id "" \ + --path / +``` + +## Access a Filesystem from another environment + +If you are accessing the Filesystem from another machine, CI job, agent sandbox, or other environment without its locally stored credential, provide a Filesystem token and the Filesystem region: + +```shell +export TI_FS_TOKEN="" +export TI_REGION_CODE="" +``` + +The token identifies its Filesystem, so you do not need to provide the Filesystem ID separately. + +You can then run commands that the token permits. For example: + +```shell +ti fs list-files --path "" +``` + +A scoped token can access only the paths and operations included in its scope. If another user or administrator gave you the token, check which paths and operations you are allowed to use. + +Treat Filesystem tokens as secrets. For CI jobs and agent environments, inject the token from a secret manager instead of storing it in source code, scripts, or container images. + +If you need to create a token for another environment, see [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md). + +## Switch between Filesystems + +If you have access to multiple Filesystems with locally stored tokens, change `TI_FS_FILE_SYSTEM_ID` to select the Filesystem you want to use in the current shell: + +```shell +export TI_FS_FILE_SYSTEM_ID="" +``` + +The CLI uses the locally stored token for the selected Filesystem. + +A Filesystem can have multiple remote tokens, while each CLI profile stores at most one selected local token for each Filesystem. Changing which Filesystem or local token the CLI uses does not disable or revoke other remote tokens. + +## Credential selection + +In most workflows, use one of the approaches above rather than specifying credentials on every command. + +If multiple token sources are available, `ti` selects the token in the following order: + +1. `--fs-token` +2. `TI_FS_TOKEN` +3. The locally stored token for the selected Filesystem + +For the complete Filesystem and credential selection rules, see [TiDB Cloud CLI Configuration and Credentials](/ai/ti/reference/ti-configuration-and-credentials.md#filesystem-credentials-and-remote-inventory). + +## What's next + +- [Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) +- [Mount TiDB Cloud Filesystem Locally](/tidb-cloud-filesystem/filesystem-mount.md) +- [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) diff --git a/tidb-cloud-filesystem/configure-filesystem-ai-providers.md b/tidb-cloud-filesystem/configure-filesystem-ai-providers.md new file mode 100644 index 0000000000000..5569b31cfae10 --- /dev/null +++ b/tidb-cloud-filesystem/configure-filesystem-ai-providers.md @@ -0,0 +1,157 @@ +--- +title: Configure TiDB Cloud Filesystem AI Providers +summary: Configure AI providers for media extraction and embeddings in a TiDB Cloud Filesystem. +aliases: ['/ai/configure-filesystem-ai-providers'] +--- + +# Configure TiDB Cloud Filesystem AI Providers + +Use this guide when you want TiDB Cloud Filesystem to extract searchable content from images, audio, or video, or when you want to configure a custom embedding provider for semantic search. + +- **Media extraction** uses an AI provider to extract text or descriptions from images, audio, or video so that the content can be searched. +- **Embeddings** represent text and extracted descriptions as vectors for semantic search. + +These capabilities are optional and can be configured separately. For example, you can configure media extraction without configuring a custom embedding provider. + +This guide shows you how to check the current configuration, configure providers for media extraction and embeddings, and disable a custom configuration when it is no longer needed. + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- Have access to an existing TiDB Cloud Filesystem and obtain its Filesystem ID. +- Configure TiDB Cloud API credentials. The commands in this guide require TiDB Cloud API credentials and an explicit Filesystem ID; they do not use a Filesystem token. +- If you want to enable or replace a provider configuration, obtain the provider endpoint, model name, and API key. + +> **Note:** +> +> When you configure a custom provider, TiDB Cloud Filesystem sends content to that provider for processing. Make sure that the provider account and its data retention and privacy policies are appropriate for your data. + +## Provide the API key of your AI provider + +When you enable or replace a provider configuration, provide the API key of your AI provider through `TI_FS_AI_PROVIDER_API_KEY`. + +For an interactive shell, read and export the key without placing it in shell history: + +```bash +printf 'Provider API key: ' >&2 +read -r -s TI_FS_AI_PROVIDER_API_KEY +printf '\n' >&2 +export TI_FS_AI_PROVIDER_API_KEY +``` + +The TiDB Cloud CLI does not store the key locally. The Filesystem service stores it encrypted and returns only a masked value when you inspect the configuration later. + +In CI, provide `TI_FS_AI_PROVIDER_API_KEY` through your CI secret-management mechanism. + +> **Note:** +> +> When you enable, re-enable, or replace a provider configuration, the Filesystem service sends a small request to the provider to validate the credentials, connectivity, and model response. The provider might charge for this validation request. Disabling a provider or updating only an extraction prompt does not send a validation request. + +## Configure media extraction + +Media extraction lets TiDB Cloud Filesystem process images, audio, or video and make the extracted text or descriptions available for content search. + +### Check the current configuration + +Before changing the configuration, check the current extraction configuration for the media type you want to process: + +```shell +ti fs describe-file-system-extract-configuration \ + --file-system-id "" \ + --media-type image +``` + +Replace `image` with `audio` or `video` to inspect another media type. + +### Enable or update media extraction + +For example, to enable image extraction with an OpenAI-compatible provider: + +```shell +ti fs update-file-system-extract-configuration \ + --file-system-id "" \ + --media-type image \ + --enabled true \ + --provider-api-base https://api.openai.com/v1 \ + --provider-model "" \ + --provider-protocol openai +``` + +The `openai` protocol supports image, audio, and video extraction. For audio extraction, you can also use the `qwen-asr` protocol with Alibaba Cloud Model Studio. + +Other provider endpoints can be used only if they implement the required OpenAI-compatible API contract. Native interfaces for Anthropic, Gemini, Vertex AI, Amazon Bedrock, and Azure OpenAI are not supported. + +For all available options, see [`update-file-system-extract-configuration`](/ai/ti/reference/ti-fs-update-file-system-extract-configuration.md). + +### Disable media extraction + +To disable extraction for a media type: + +```shell +ti fs update-file-system-extract-configuration \ + --file-system-id "" \ + --media-type image \ + --enabled false +``` + +## Configure embeddings + +Embeddings represent text content and extracted media descriptions as vectors for semantic search. + +### Check the current configuration + +Before configuring a custom embedding provider, check the current embedding configuration: + +```shell +ti fs describe-file-system-embedding-configuration \ + --file-system-id "" +``` + +Check the `source` field in the output. If it is `database_auto`, embeddings are managed by the service and you cannot replace the configuration with a custom provider. + +### Enable or update a custom embedding provider + +If the current configuration allows a custom provider, configure an OpenAI-compatible embedding endpoint: + +```shell +ti fs update-file-system-embedding-configuration \ + --file-system-id "" \ + --enabled true \ + --provider-api-base https://api.openai.com/v1 \ + --provider-model "" +``` + +The provider must implement the required OpenAI-compatible embeddings API and return exactly 1024-dimensional vectors. Make sure that the selected model and provider configuration return 1024 dimensions; other vector dimensions are not supported. + +For all available options, see [`update-file-system-embedding-configuration`](/ai/ti/reference/ti-fs-update-file-system-embedding-configuration.md). + +### Disable the custom embedding configuration + +To disable the custom embedding configuration: + +```shell +ti fs update-file-system-embedding-configuration \ + --file-system-id "" \ + --enabled false +``` + +## Finish configuring providers + +After you finish configuring providers, remove the provider API key from the current shell: + +```shell +unset TI_FS_AI_PROVIDER_API_KEY +``` + +When custom media extraction is enabled, TiDB Cloud Filesystem sends the relevant media content to the configured extraction provider. When a custom embedding provider is enabled, text content and extracted media descriptions are sent to the embedding provider. + +If an update fails because of a timeout, lost response, or another error where you cannot tell whether the update succeeded, do not immediately retry the command. Run the corresponding `describe-file-system-*-configuration` command first. + +The Filesystem service might already have saved the configuration and sent the provider validation request even if the CLI did not receive the response. + +## What's next + +- [Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) to search Filesystem content. +- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) for complete command syntax and options. diff --git a/tidb-cloud-filesystem/filesystem-authorization.md b/tidb-cloud-filesystem/filesystem-authorization.md new file mode 100644 index 0000000000000..13e628a53d3ef --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-authorization.md @@ -0,0 +1,107 @@ +--- +title: TiDB Cloud Filesystem Authorization +summary: Learn how TiDB Cloud API credentials, owner tokens, and scoped tokens provide different levels of access to TiDB Cloud Filesystem. +--- + +# TiDB Cloud Filesystem Authorization + +TiDB Cloud Filesystem uses different credentials for managing a Filesystem and accessing its data. This lets you keep high-privilege credentials in a trusted environment while giving users, applications, and agents only the access they need. + +In general, access becomes more limited at each level: + +```plaintext +TiDB Cloud API credentials + ↓ +Manage Filesystem resources + ↓ +Owner token + ↓ +Access one Filesystem + ↓ +Scoped token + ↓ +Access selected paths and operations +``` + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Authorization model + +TiDB Cloud Filesystem uses the following credential types: + +| Credential | Access | Typical use | +| -------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | +| TiDB Cloud API credentials | Manage Filesystem resources and owner tokens according to the account's permissions | Filesystem administration in a trusted environment | +| Owner token | Broad access to one Filesystem, including its data and scoped-token management | Trusted workflows that need full Filesystem access | +| Scoped token | Access only to specified paths and operations in one Filesystem | Users, applications, agents, or other machines that need limited access | + +These credentials are not interchangeable. TiDB Cloud API credentials manage the Filesystem resource itself, while Filesystem tokens control access within a Filesystem. + +## TiDB Cloud API credentials + +TiDB Cloud API credentials consist of a public and private API key pair. Their permissions are determined by the associated TiDB Cloud account. + +Use TiDB Cloud API credentials for resource-level operations such as creating, listing, describing, and deleting Filesystems. They are also required to generate owner tokens. + +Keep these credentials in a trusted administrative or automation environment rather than distributing them to applications or agents that only need access to Filesystem data. + +Configure these credentials with `ti configure`, or provide `TIDB_CLOUD_PUBLIC_KEY` and `TIDB_CLOUD_PRIVATE_KEY` together. For details, see [TiDB Cloud CLI Configuration and Credentials](/ai/ti/reference/ti-configuration-and-credentials.md). + +## Owner tokens + +An owner token provides broad access to one Filesystem. It can read, write, and delete files and can create and manage scoped tokens for that Filesystem. + +An owner token applies only to its Filesystem. It does not replace TiDB Cloud API credentials for resource-level operations such as creating or deleting Filesystems. An owner token also cannot create another owner token; generating another owner token requires TiDB Cloud API credentials. + +Because an owner token has broad privileges within a Filesystem, keep it on a trusted machine or in a secret manager. When another user, application, or agent needs only limited access, create a scoped token instead of sharing the owner token. + +A Filesystem can have multiple active owner tokens. This lets different users, applications, or environments use separate credentials instead of sharing the same token. + +For instructions on generating and managing owner tokens, see [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md). + +## Scoped tokens + +A scoped token limits access to specified paths and operations within one Filesystem. + +The supported operations are: + +- `read`: read file content and metadata. +- `list`: list entries under a directory. +- `search`: search for files under the allowed path. `search` also requires `read`. +- `write`: create or modify files and directories. +- `delete`: delete files or directories. + +For example, a reporting application could have: + +- `read,list` access to `/inputs` +- `read,list,write` access to `/reports` + +Another user that only needs to review generated reports could receive `read,list` access to `/reports`. + +Scoped permissions are enforced by the Filesystem service, including when the Filesystem is accessed through a mount. A scoped token cannot broaden its own permissions or create or manage other tokens. A scoped token can be refreshed while it is valid, but refreshing it does not broaden its permissions. + +After a scoped token is issued, disabling or revoking the owner token that created it does not automatically revoke the scoped token. Revoke scoped tokens separately when their access is no longer needed. + +Use scoped tokens when giving Filesystem access to users, applications, agents, or other machines that do not need full access to the Filesystem. + +For instructions on creating and delegating scoped tokens, see [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md#generate-and-delegate-a-scoped-token). + +## Credential boundaries + +Use the credential with the minimum access required for a workflow: + +* Use **TiDB Cloud API credentials** to manage the Filesystem resource. +* Use an **owner token** when a trusted workflow needs broad access to one Filesystem or needs to delegate access. +* Use a **scoped token** when a user, application, or agent needs access only to specific paths and operations. + +Having multiple credential types available in the same environment does not combine their permissions. For example, a scoped token remains scoped even if TiDB Cloud API credentials are also configured. + +TiDB Cloud CLI can store a selected Filesystem token locally for convenience. Local credential selection does not change, disable, or revoke remote tokens. For details about how `ti` selects Filesystems and credentials, see [TiDB Cloud CLI Configuration and Credentials](/ai/ti/reference/ti-configuration-and-credentials.md). + +## What's next + +- [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) +- [Share a TiDB Cloud Filesystem Across Machines](/tidb-cloud-filesystem/filesystem-sharing.md) +- [Mount TiDB Cloud Filesystem Locally](/tidb-cloud-filesystem/filesystem-mount.md) diff --git a/tidb-cloud-filesystem/filesystem-layers-checkpoints.md b/tidb-cloud-filesystem/filesystem-layers-checkpoints.md new file mode 100644 index 0000000000000..864f8b36d0fa0 --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-layers-checkpoints.md @@ -0,0 +1,67 @@ +--- +title: TiDB Cloud Filesystem Layers and Checkpoints +summary: TiDB Cloud Filesystem layers isolate changes from the base Filesystem, while forks and checkpoints preserve layer history until changes are committed. +--- + +# TiDB Cloud Filesystem Layers and Checkpoints + +A Filesystem layer records changes over a path in the base Filesystem while keeping those changes separate from the shared base data. Changes in a layer become part of the base Filesystem only when the layer is committed. + +Layers can also be checkpointed or forked, which lets you preserve a point in a layer's history or continue work in an independent child layer. + +Filesystem layers are independent of Git branches and do not require a Git repository. + +For commands to create, inspect, fork, checkpoint, commit, or roll back layers, see [Manage TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md). + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Core concepts + +- **Base Filesystem:** the shared Filesystem state outside a layer. A layer records changes over a base path without immediately modifying this state. +- **Layer:** a writable set of changes over a base path. Until the layer is committed, its changes remain separate from the base Filesystem. +- **Checkpoint:** a named point in a layer's durable history. Creating a checkpoint preserves that point in the layer history but does not apply the layer's changes to the base Filesystem. +- **Fork:** a new writable child layer created from the current state of another layer or from one of its checkpoints. Changes made after the fork are independent between the parent and child layers. +- **Commit:** applies the effective changes in a layer to the base Filesystem. +- **Rollback:** discards the layer's uncommitted changes without applying them to the base Filesystem. Rollback does not reset a layer to an earlier checkpoint. + +## Layers and the base Filesystem + +Changes made in a layer are visible through that layer but are not visible in the base Filesystem until the layer is committed. + +Operations that do not specify a layer access the base Filesystem. This means that an uncommitted layer does not change what other users or applications see when they access the base directly. + +A layer is an overlay on a base path rather than a complete snapshot of the Filesystem. A fork or checkpoint preserves the relevant layer history, but it does not freeze the entire base Filesystem. Data that continues to be resolved from the base can therefore reflect later changes to the base. + +## Forks and checkpoints + +A checkpoint preserves a point in one layer's durable history. It does not create another writable layer and does not publish changes to the base Filesystem. + +A checkpoint can be accessed as a read-only view. To continue making changes from a checkpoint, fork a new writable layer from it. + +A fork preserves the parent layer history at the point where the fork is created. After that, new changes in the parent and child layers are independent. + +For operational requirements when creating checkpoints or working with mounted layers, see [Manage TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md). + +## Commit and rollback + +Committing a layer applies all of its effective changes to the base Filesystem. A commit is not a Git commit and does not selectively publish individual files. + +A commit applies changes to the base Filesystem, not to the layer's parent layer. If the base Filesystem contains conflicting changes, the commit can fail rather than automatically merging the changes. + +Rolling back a layer discards its uncommitted changes instead of publishing them. It does not reset the layer to an earlier checkpoint. + +## Boundaries and limitations + +Layers provide change isolation, but they are not full Filesystem snapshots, backups, or authorization boundaries. To restrict access to Filesystem paths or operations, use [scoped tokens](/tidb-cloud-filesystem/filesystem-authorization.md#scoped-tokens). + +The current layer model has the following limitations: + +- Git-style merge and rebase operations are not supported. +- A layer cannot be reset in place to a checkpoint. To continue from a checkpoint, fork a new layer. +- Committing a child layer does not merge its changes into its parent layer. +- Deleting a layer logically abandons it, but history referenced by descendant layers might remain available while those descendants depend on it. +- Complex histories involving repeated changes to layer-created files or inherited metadata might have limitations during public preview. + +For command syntax, flags, and output fields, see the [`ti fs` command reference](/ai/ti/reference/ti-filesystem.md). diff --git a/tidb-cloud-filesystem/filesystem-mount-docker.md b/tidb-cloud-filesystem/filesystem-mount-docker.md new file mode 100644 index 0000000000000..e7295c1d6d9d6 --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-mount-docker.md @@ -0,0 +1,213 @@ +--- +title: Mount TiDB Cloud Filesystem in Docker +summary: Mount a TiDB Cloud Filesystem inside a Docker container on a Linux host by enabling FUSE and the required container permissions. +--- + +# Mount TiDB Cloud Filesystem in Docker + +To mount a TiDB Cloud Filesystem inside a Docker container, the container must have access to FUSE on the Linux host. In addition to installing `fuse3` inside the container, you need to expose `/dev/fuse` and grant the container permission to create the mount. + +This guide covers both Docker and Docker Compose on a Linux host. If your environment does not provide FUSE access or allow the required container privileges, use direct `ti fs` commands such as `copy-file`, `read-file`, and `list-files` instead. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Prerequisites + +Before you begin: + +- Use a Linux Docker host where `/dev/fuse` is available. +- Have permission to start containers with access to `/dev/fuse` and the capabilities required to create a mount. +- Have a Filesystem token and region code for an existing TiDB Cloud Filesystem. See [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). + +Docker Desktop and managed container or sandbox environments might not expose `/dev/fuse` or allow the required privileges. + +> **Warning:** +> +> The examples in this guide grant the container `SYS_ADMIN` and disable its AppArmor profile with `apparmor=unconfined`. These settings give the container broader privileges than a standard Docker container. +> +> Use them only when permitted by your environment's security policy. Do not run untrusted code with these privileges unless your environment provides an appropriate additional isolation boundary. A scoped Filesystem token can restrict access to Filesystem data, but it does not reduce the privileges granted to the container by Docker. + +## Provide Filesystem access to the container + +On the Docker host, set the Filesystem token and region code: + +```bash +export TI_FS_TOKEN="" +export TI_REGION_CODE="" +``` + +Treat the Filesystem token as a secret. Do not put it in a Dockerfile or commit it to source control. + +The Docker and Docker Compose examples below pass these environment variables from the host into the container. + +## Start the container + +Choose either Docker or Docker Compose. + +### Use Docker + +On the Docker host, start an interactive Ubuntu container with access to FUSE: + +```bash +docker run --rm -it \ + --device /dev/fuse \ + --cap-add SYS_ADMIN \ + --security-opt apparmor=unconfined \ + --env TI_FS_TOKEN \ + --env TI_REGION_CODE \ + ubuntu:24.04 bash +``` + +This command does the following: + +- Exposes the host's `/dev/fuse` device to the container; +- Grants the capability required to create the mount; and +- Passes the Filesystem token and region code into the container. + +After the container starts, continue with [Install and mount inside the container](#install-and-mount-inside-the-container). + +### Use Docker Compose + +Alternatively, create the following `compose.yaml` file on the Docker host: + +```yaml +services: + agent: + image: ubuntu:24.04 + command: ["sleep", "infinity"] + devices: + - /dev/fuse:/dev/fuse + cap_add: + - SYS_ADMIN + security_opt: + - apparmor=unconfined + environment: + TI_FS_TOKEN: ${TI_FS_TOKEN:?Set TI_FS_TOKEN} + TI_REGION_CODE: ${TI_REGION_CODE:?Set TI_REGION_CODE} +``` + +Start the container and open a shell in it: + +```bash +docker compose up -d +docker compose exec agent bash +``` + +Then continue with the steps below inside the container. + +## Install and mount inside the container + +The following steps run inside the container. They use the `ubuntu:24.04` image from the preceding examples, which runs as `root` by default. + +1. Install `fuse3` and the tools required to install `ti`: + + ```bash + apt-get update + apt-get install -y --no-install-recommends ca-certificates curl fuse3 + ``` + +2. Install TiDB Cloud CLI: + + ```bash + curl -fsSL https://github.com/tidbcloud/ti-cli/releases/latest/download/install.sh | sh -s -- --yes + ``` + +3. Add `ti` to the current shell and create a local directory for the mount: + + ```bash + export PATH="$HOME/.ti/bin:$PATH" + + mkdir -p "$HOME/workspace" + ``` + +4. Mount the Filesystem: + + ```bash + ti fs mount-file-system --mount-path "$HOME/workspace" + ``` + + On Linux, `ti` uses FUSE by default. After the command succeeds, you can access the Filesystem through `$HOME/workspace`. If the mount fails to start, inspect the diagnostic log path reported by `ti`. + + If your Filesystem token grants access only to a specific remote path, use the following command instead of the preceding mount command: + + ```bash + ti fs mount-file-system \ + --remote-path /workspace \ + --mount-path "$HOME/workspace" + ``` + + In this example, the remote `/workspace` directory becomes the root of the local mount. For more information, see [Mount only part of the Filesystem](/tidb-cloud-filesystem/filesystem-mount.md#mount-only-part-of-the-filesystem). + + To prevent writes through the local mount, add `--read-only` to the mount command. For example, to mount the Filesystem root as read-only: + + ```bash + ti fs mount-file-system \ + --mount-path "$HOME/workspace" \ + --read-only + ``` + + The `--read-only` option affects this local mount only. Use a scoped token with read-only permissions to enforce read-only access at the Filesystem service. + +5. Verify that you can access the mounted Filesystem: + + ```bash + ls "$HOME/workspace" + ``` + +Run the mount and the application that accesses it as the same OS user. If your application image uses a non-root user, install the required packages when building the image and create the mount as the application user at runtime. + +## Flush FUSE writes without unmounting + +If you need pending writes to reach the Filesystem while keeping the mount running, stop applications from writing to the relevant files and close those files first. Then drain the mount inside the container: + +```bash +ti fs drain-file-system \ + --mount-path "$HOME/workspace" \ + --timeout 30s +``` + +A successful drain confirms that pending writes have reached the Filesystem while leaving the mount running. If the drain times out or returns an error, keep the container and Docker host available, resolve the error, and verify the files before stopping or removing the container. For command syntax, see [`drain-file-system`](/ai/ti/reference/ti-fs-drain-file-system.md). + +## Stop the container safely + +Before stopping or removing the container: + +1. Inside the container, stop applications that are writing to the mounted directory and close open files. + +2. Unmount the Filesystem: + + ```bash + ti fs unmount-file-system --mount-path "$HOME/workspace" + ``` + + Wait for the unmount to succeed before stopping or removing the container. A successful FUSE unmount flushes pending writes. + +3. Stop the container: + + - If you started it with `docker run --rm -it`, exit the shell after the unmount succeeds: + + ```bash + exit + ``` + + Docker removes the container automatically because it was started with `--rm`. + + - If you used Docker Compose, exit the container shell, and then run the following command on the Docker host: + + ```bash + docker compose down + ``` + +Stopping or removing the container does not delete the remote Filesystem or its data. + +> **Warning:** +> +> If unmounting fails, do not stop or remove the container. Pending writes might still exist only inside the container. Keep the container running, resolve the error, and verify that the required files have reached the Filesystem first. + +## What's next + +- [Mount TiDB Cloud Filesystem Locally](/tidb-cloud-filesystem/filesystem-mount.md) for read-only mounts, mounting layers or checkpoints, and other common mount options. +- [Share a TiDB Cloud Filesystem Across Machines](/tidb-cloud-filesystem/filesystem-sharing.md) to give another user or environment access to the same Filesystem. +- [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) to restrict access to specific paths and operations. diff --git a/tidb-cloud-filesystem/filesystem-mount-linux.md b/tidb-cloud-filesystem/filesystem-mount-linux.md new file mode 100644 index 0000000000000..093ae624a73a4 --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-mount-linux.md @@ -0,0 +1,223 @@ +--- +title: Mount TiDB Cloud Filesystem on Linux +summary: Install FUSE userspace tools, mount a TiDB Cloud Filesystem on Linux, verify access, and troubleshoot common mount permission issues. +--- + +# Mount TiDB Cloud Filesystem on Linux + +On Linux, TiDB Cloud Filesystem uses FUSE to make Filesystem data available through a local directory. After mounting, your applications and tools can access the Filesystem by using ordinary local file paths. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI (`ti`)](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- Make the Filesystem and its token available to `ti`. See [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). +- Use a Linux host where you can install the `fuse3` package. For a container, follow [Docker and Docker Compose](/tidb-cloud-filesystem/filesystem-mount-docker.md) instead. + +Run the mount and the application that accesses it as the same OS user. + +## Install FUSE userspace tools + +1. On Ubuntu or Debian, install the `fuse3` package: + + ```bash + sudo apt-get update + sudo apt-get install -y --no-install-recommends fuse3 + ``` + +2. Check that the FUSE mount helper is installed: + + ```bash + command -v fusermount3 + ``` + + The command prints the path to `fusermount3`, such as: + + ```text + /usr/bin/fusermount3 + ``` + +3. Check that the FUSE device is available: + + ```bash + ls -l /dev/fuse + ``` + + The command shows an entry for `/dev/fuse`. + +If `fusermount3` is not found, make sure the `fuse3` package is installed. If `/dev/fuse` does not exist or you cannot use it when mounting, ask the host administrator to enable FUSE and grant your user access to the device. + +On another Linux distribution, install the FUSE package that provides `fusermount3` and perform the same checks. + +## Mount and verify the Filesystem + +1. Create a local directory for the mount: + + ```bash + mkdir -p "$HOME/workspace" + ``` + + Use a directory owned by the same OS user that will run the applications accessing the mount. + +2. Mount the Filesystem: + + ```bash + ti fs mount-file-system --mount-path "$HOME/workspace" + ``` + + On Linux, `ti` uses FUSE by default. + + The command waits until the mount is ready before returning. The mount continues running in the background after the command returns, so closing the terminal does not unmount it. + + After the mount succeeds, you can access the Filesystem through `$HOME/workspace`. + + If your Filesystem token grants access only to a specific remote path, use the following command instead of the preceding mount command: + + ```bash + ti fs mount-file-system \ + --remote-path /workspace \ + --mount-path "$HOME/workspace" + ``` + + In this example, the remote `/workspace` directory becomes the root of the local mount. For more information, see [Mount only part of the Filesystem](/tidb-cloud-filesystem/filesystem-mount.md#mount-only-part-of-the-filesystem). + + To prevent writes through the local mount, add `--read-only` to the mount command. For example, to mount the Filesystem root as read-only: + + ```bash + ti fs mount-file-system \ + --mount-path "$HOME/workspace" \ + --read-only + ``` + + The `--read-only` option affects this local mount only. Use a scoped token with read-only permissions to enforce read-only access at the Filesystem service. + +3. Verify that you can access the mounted Filesystem: + + ```bash + ls "$HOME/workspace" + ``` + + The command lists the files and directories at the root of the mounted path. + + If you used a writable mount and your token has write permission, you can also create and read a test file: + + ```bash + TEST_FILE="mount-check-$(date +%s).txt" + + printf 'Hello from Linux\n' > "$HOME/workspace/$TEST_FILE" + cat "$HOME/workspace/$TEST_FILE" + ``` + + Example output: + + ```text + Hello from Linux + ``` + +4. When you are finished, stop applications that are writing to the mounted directory, close open files, and unmount the Filesystem: + + ```bash + ti fs unmount-file-system --mount-path "$HOME/workspace" + ``` + + A successful unmount flushes pending FUSE writes before stopping the mount. + + If you created the test file above, you can optionally verify after unmounting that the file is available directly from the Filesystem: + + ```bash + ti fs read-file --path "/$TEST_FILE" + ``` + + Example output: + + ```text + Hello from Linux + ``` + +Unmounting removes the local mount but does not delete the Filesystem or its data. + +> **Warning:** +> +> If unmounting fails, keep the mount and machine running until you resolve the error and verify that the required files have reached the Filesystem. Some pending writes might still exist only on that machine. + +## Flush FUSE writes without unmounting + +If you need pending writes to reach the Filesystem while keeping the mount running, stop applications from writing to the relevant files and close those files first. Then drain the mount: + +```bash +ti fs drain-file-system \ + --mount-path "$HOME/workspace" \ + --timeout 30s +``` + +A successful drain confirms that pending writes have reached the Filesystem while leaving the mount running. If the drain times out or returns an error, keep the mount and machine available, resolve the error, and verify the files before ending the session or telling another user that the updates are ready. For command syntax, see [`drain-file-system`](/ai/ti/reference/ti-fs-drain-file-system.md). + +## Troubleshoot mount permission errors + +If the mount fails with `Permission denied`, check the following in order: + +1. Make sure the mount directory is writable by the current user: + + ```bash + ls -ld "$HOME/workspace" + ``` + +2. Make sure `/dev/fuse` exists: + + ```bash + ls -l /dev/fuse + ``` + +3. Make sure you are creating the mount as the same OS user that will run the application. + + Do not create the FUSE mount as `root` and then try to give another user access by changing the ownership of the mount directory. Instead, create the mount as the application user. + +4. Check whether another mount is already using the same path: + + ```bash + mount | grep "$HOME/workspace" + ``` + + If a mount is listed, unmount it with `ti fs unmount-file-system --mount-path "$HOME/workspace"` or `fusermount3 -u "$HOME/workspace"` before creating a new mount. + +5. If `ti` reports a diagnostic log path, inspect that log for the underlying error. + +On systems with additional security controls, such as AppArmor, the operating system can reject a mount even when the directory permissions look correct. + +For additional mount errors, see [Troubleshoot TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-troubleshooting.md). + +## Ubuntu 26.04 mount-path restrictions + +On Ubuntu 26.04, the AppArmor profile for `fusermount3` can prevent FUSE mounts at some paths. In particular, a top-level directory such as `/workspace` can fail with `Permission denied` even when its file permissions appear correct. + +For the commands in this guide, use `$HOME/workspace` as the mount directory instead of `/workspace`. + +Changing the owner of `/workspace` or running the mount as `root` does not bypass the AppArmor restriction. + +If your application specifically requires `/workspace`, ask the host administrator to allow that path in `/etc/apparmor.d/local/fusermount3`: + +```text +mount fstype=@{fuse_types} options=(nosuid,nodev) options in (ro,rw,noatime,dirsync,nodiratime,noexec,sync) -> /workspace/{,**/}, +umount /workspace/{,**/}, +``` + +The administrator can then reload the AppArmor profile: + +```shell +sudo apparmor_parser -r /etc/apparmor.d/fusermount3 +``` + +After the profile is updated, retry the mount at `/workspace`. + +For help checking whether AppArmor caused the failure, see [Ubuntu 26.04 rejects a FUSE mount under `/workspace`](/tidb-cloud-filesystem/filesystem-troubleshooting.md#ubuntu-2604-rejects-a-fuse-mount-under-workspace). + +## What's next + +- [Mount TiDB Cloud Filesystem Locally](/tidb-cloud-filesystem/filesystem-mount.md) for read-only mounts, mounting layers or checkpoints, and other common mount options. +- [Manage TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md) to mount and work with layers or checkpoints. +- [Share a TiDB Cloud Filesystem Across Machines](/tidb-cloud-filesystem/filesystem-sharing.md) to give another user or environment access to the Filesystem. diff --git a/tidb-cloud-filesystem/filesystem-mount-macos.md b/tidb-cloud-filesystem/filesystem-mount-macos.md new file mode 100644 index 0000000000000..ddf1676969ecf --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-mount-macos.md @@ -0,0 +1,207 @@ +--- +title: Mount TiDB Cloud Filesystem on macOS +summary: Mount a TiDB Cloud Filesystem as a local directory on macOS, and use macFUSE when you need FUSE-specific features. +--- + +# Mount TiDB Cloud Filesystem on macOS + +On macOS, you can mount a TiDB Cloud Filesystem as a local directory and access its files with your usual applications and tools. + +For most workflows, you can mount a TiDB Cloud Filesystem with WebDAV. It lets you access Filesystem files through normal local file paths and does not require additional mount software. + +Use FUSE with macFUSE when you also need FUSE-specific features, such as mounting a layer or checkpoint, or making pending writes reach the Filesystem without unmounting it. + +Without macFUSE, TiDB Cloud CLI (`ti`) uses WebDAV. If macFUSE is installed, `ti` prefers FUSE when the mount driver is selected automatically. The commands in this guide specify the driver explicitly so that you know which mount method is being used. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI (`ti`)](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- Make the Filesystem and its token available to `ti`. See [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). + +The write examples below require a token with write permission. + +## Mount with WebDAV + +For most workflows, you can use WebDAV without installing additional mount software. + +1. Create a local directory for the mount: + + ```bash + mkdir -p "$HOME/workspace" + ``` + +2. Mount the Filesystem with WebDAV: + + ```bash + ti fs mount-file-system \ + --mount-path "$HOME/workspace" \ + --driver webdav + ``` + + The mount continues running in the background after the command returns, so closing the terminal does not unmount it. + + After the command succeeds, you can access the Filesystem through `$HOME/workspace`. + + If your Filesystem token grants access only to a specific remote path, use the following command instead of the preceding mount command: + + ```bash + ti fs mount-file-system \ + --remote-path /workspace \ + --mount-path "$HOME/workspace" \ + --driver webdav + ``` + + In this example, the remote `/workspace` directory becomes the root of the local mount. For more information, see [Mount only part of the Filesystem](/tidb-cloud-filesystem/filesystem-mount.md#mount-only-part-of-the-filesystem). + + To prevent writes through the local mount, add `--read-only` to the mount command. For example, to mount the Filesystem root as read-only: + + ```bash + ti fs mount-file-system \ + --mount-path "$HOME/workspace" \ + --driver webdav \ + --read-only + ``` + + The `--read-only` option affects this local mount only. Use a scoped token with read-only permissions to enforce read-only access at the Filesystem service. + +3. Verify that you can access the mounted Filesystem: + + ```bash + ls "$HOME/workspace" + ``` + + If you used a writable mount and your token has write permission, you can also create and read a test file: + + ```bash + TEST_FILE="mount-check-$(date +%s).txt" + + printf 'Hello from macOS\n' > "$HOME/workspace/$TEST_FILE" + cat "$HOME/workspace/$TEST_FILE" + ``` + + Example output: + + ```text + Hello from macOS + ``` + +4. When you are finished, close files that are open in applications and unmount the Filesystem: + + ```bash + ti fs unmount-file-system --mount-path "$HOME/workspace" + ``` + + WebDAV does not support `drain-file-system`. Complete a normal unmount before shutting down the machine or handing updated files to another user or environment. + + If you created the test file above, you can optionally verify after unmounting that the file is available directly from the Filesystem: + + ```bash + ti fs read-file --path "/$TEST_FILE" + ``` + + Example output: + + ```text + Hello from macOS + ``` + +Unmounting removes the local mount but does not delete the Filesystem or its data. + +## Use macFUSE when you need FUSE features + +Use FUSE instead of WebDAV when you need to: + +- mount a layer or checkpoint; or +- make pending writes reach the Filesystem while keeping the mount running with `drain-file-system`. + +To use FUSE on macOS: + +1. Install [macFUSE](https://macfuse.github.io/) and complete any installation or security approval steps required by your macOS version. + + Installing `ti` does not install macFUSE. + +2. Prepare the local mount directory: + + - If a mount is already active at `$HOME/workspace`, unmount it before reusing the same directory: + + ```bash + ti fs unmount-file-system --mount-path "$HOME/workspace" + ``` + + - Ensure that `$HOME/workspace` exists: + + ```bash + mkdir -p "$HOME/workspace" + ``` + +3. Mount the Filesystem with FUSE: + + ```bash + ti fs mount-file-system \ + --mount-path "$HOME/workspace" \ + --driver fuse + ``` + + The mount continues running in the background after the command returns, so closing the terminal does not unmount it. + + Run the FUSE mount and the applications that access it as the same OS user. + + If your Filesystem token grants access only to a specific remote path, add `--remote-path` as described in [Mount only part of the Filesystem](/tidb-cloud-filesystem/filesystem-mount.md#mount-only-part-of-the-filesystem) instead of mounting the Filesystem root. + + Layer and checkpoint mounts require FUSE, and checkpoint mounts are always read-only. For details, see [Manage TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md). + + If you need pending writes to reach the Filesystem while keeping the mount running, see [FUSE write behavior](/tidb-cloud-filesystem/filesystem-mount.md#fuse-write-behavior). For command syntax, see [`drain-file-system`](/ai/ti/reference/ti-fs-drain-file-system.md). + +4. When you are finished, stop applications from writing to the mount, close open files, and unmount it: + + ```bash + ti fs unmount-file-system --mount-path "$HOME/workspace" + ``` + + A successful FUSE unmount flushes pending writes. You do not need to run `drain-file-system` before a normal unmount. + +Unmounting removes the local mount but does not delete the Filesystem or its data. + +## Flush FUSE writes without unmounting + +If you need pending writes to reach the Filesystem while keeping the FUSE mount running, stop applications from writing to the relevant files and close those files first. Then drain the mount: + +```bash +ti fs drain-file-system \ + --mount-path "$HOME/workspace" \ + --timeout 30s +``` + +A successful drain confirms that pending writes have reached the Filesystem while leaving the mount running. If the drain times out or returns an error, keep the mount and machine available, resolve the error, and verify the files before ending the session or telling another user that the updates are ready. WebDAV does not support `drain-file-system`. For command syntax, see [`drain-file-system`](/ai/ti/reference/ti-fs-drain-file-system.md). + +## Troubleshoot mount issues + +If a WebDAV mount fails to start: + +- Make sure the local mount directory exists and is writable. +- Make sure another mount is not already using the same local directory. +- Check the diagnostic log path reported by `ti` for the underlying error. + +If a FUSE mount fails to start: + +- Make sure macFUSE is installed. +- Complete any macOS security approvals required by macFUSE. +- Make sure the mount and the application that accesses it run as the same OS user. +- Make sure another mount is not already using the same local directory. +- Check the diagnostic log path reported by `ti`. + +If unmounting fails, keep the mount process and machine running until you resolve the error and verify that required files have reached the Filesystem. Do not remove local mount data while pending writes might remain. + +For additional mount errors, see [Troubleshoot TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-troubleshooting.md). + +## What's next + +- [Mount TiDB Cloud Filesystem Locally](/tidb-cloud-filesystem/filesystem-mount.md) for read-only mounts, mounting layers or checkpoints, and other common mount options. +- [Manage TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md) to work with layers and checkpoints through FUSE. +- [Share a TiDB Cloud Filesystem Across Machines](/tidb-cloud-filesystem/filesystem-sharing.md) to give another user or environment access to the Filesystem. diff --git a/tidb-cloud-filesystem/filesystem-mount.md b/tidb-cloud-filesystem/filesystem-mount.md new file mode 100644 index 0000000000000..be9abd3712c10 --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-mount.md @@ -0,0 +1,90 @@ +--- +title: Mount TiDB Cloud Filesystem Overview +summary: Learn when to mount a TiDB Cloud Filesystem, which mount method to use, and the capabilities and limitations of Filesystem mounts. +aliases: ['/ai/mount-filesystem'] +--- + +# Mount TiDB Cloud Filesystem Overview + +You can work with files in TiDB Cloud Filesystem either by using `ti fs` commands directly or by mounting the Filesystem as a local directory. + +Mount your TiDB Cloud Filesystem when an editor, application, agent, or other tool needs to access its files through local file paths. After mounting, the Filesystem appears as a local directory, so the tool can read and write its files using normal filesystem operations. + +If you only need to perform file operations through the CLI, such as reading, copying, organizing, or searching files, use `ti fs` commands directly instead of mounting the Filesystem. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Choose a mount method + +The available mount method depends on your environment: + +| Environment | Mount method | Detailed Guide | +| --- | --- | --- | +| Linux | FUSE | [Mount TiDB Cloud Filesystem on Linux](/tidb-cloud-filesystem/filesystem-mount-linux.md) | +| macOS | WebDAV for general file access, or FUSE with macFUSE for FUSE-specific features | [Mount TiDB Cloud Filesystem on macOS](/tidb-cloud-filesystem/filesystem-mount-macos.md) | +| Docker on Linux | FUSE with access to `/dev/fuse` and additional container privileges | [Mount TiDB Cloud Filesystem in Docker](/tidb-cloud-filesystem/filesystem-mount-docker.md) | + +Native Filesystem mounting is not supported on Windows. On Windows, use direct commands such as `ti fs copy-file`, `ti fs read-file`, and `ti fs list-files` instead. + +On macOS, WebDAV is sufficient for general file access and does not require additional mount software. FUSE is required for features such as mounting layers or checkpoints and using `drain-file-system`. + +## Use a token without configuring a profile + +If another user or system administrator gives you a Filesystem token, you can use that token from the current machine without configuring a `ti` profile or using the creator's API keys: + +```bash +export TI_FS_TOKEN="" +export TI_REGION_CODE="" +``` + +Then follow the guide for your environment. The token identifies the Filesystem and limits the paths and operations available to the current environment. Treat the token as a secret. + +## Common mount capabilities + +Filesystem mounts support several options that change what is exposed through the local mount: + +### Mount only part of the Filesystem + +By default, a mount exposes the Filesystem root `/`. Use `--remote-path` to expose a specific remote directory instead. This is also useful when a scoped token grants access only to a specific path. + +### Create a read-only mount + +Use `--read-only` to prevent writes through a particular mount. This option does not change the permissions of the Filesystem token. To enforce read-only access at the service level, use a scoped token with read-only permissions. + +### Mount layers and checkpoints + +Layers and checkpoints require FUSE. Checkpoint mounts are always read-only. + +For layer and checkpoint workflows, see [Manage TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md). + +## Finish safely + +### Unmount when you are finished + +Stop applications from writing to the mounted directory, close open files, and follow your platform guide to run `ti fs unmount-file-system`. A successful FUSE unmount flushes pending writes before stopping the mount. Unmounting removes the local mount but does not delete the Filesystem or its data. + +### FUSE write behavior + +FUSE mounts can temporarily have writes that have not yet reached the remote Filesystem. + +A normal successful unmount flushes pending writes, so you do not need to run `drain-file-system` before unmounting. + +Use `drain-file-system` only when you need pending writes to reach the Filesystem while keeping the mount running, such as before creating a checkpoint or making updated files available to another environment. + +WebDAV mounts do not support `drain-file-system`. + +> **Warning:** +> +> If a FUSE drain or unmount fails, keep the mount and machine available until you resolve the error and verify that required files have reached the Filesystem. Some pending writes might still exist only on that machine. + +## What's next + +Choose the guide for your environment: + +- [Mount TiDB Cloud Filesystem on Linux](/tidb-cloud-filesystem/filesystem-mount-linux.md) +- [Mount TiDB Cloud Filesystem on macOS](/tidb-cloud-filesystem/filesystem-mount-macos.md) +- [Mount TiDB Cloud Filesystem in Docker](/tidb-cloud-filesystem/filesystem-mount-docker.md) + +For all mount options, see the [`mount-file-system` command reference](/ai/ti/reference/ti-fs-mount-file-system.md). diff --git a/tidb-cloud-filesystem/filesystem-overview.md b/tidb-cloud-filesystem/filesystem-overview.md new file mode 100644 index 0000000000000..3c012e2be0b30 --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-overview.md @@ -0,0 +1,64 @@ +--- +title: TiDB Cloud Filesystem Overview +summary: Learn what TiDB Cloud Filesystem is, when to use it, and how to access and share persistent files across applications, automation, and AI agents. +--- + +# TiDB Cloud Filesystem Overview + +TiDB Cloud Filesystem is persistent, shared file storage that applications, automation, and AI agents can access across machines and sessions. + +Use a Filesystem when files need to remain available after a local process or temporary environment ends, or when multiple environments need to work with the same files without repeatedly copying them between machines. + +For example, you can use a Filesystem to: + +- continue working with the same files from another machine or environment; +- share files or results with another user, application, CI job, or agent; +- mount shared files as a local directory for tools that expect local file paths; and +- isolate changes in layers before applying selected changes to shared files. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## How TiDB Cloud Filesystem works + +Files in a TiDB Cloud Filesystem are stored independently of the machine or process that accesses them. When you move to another environment, you can access the same Filesystem instead of copying or recreating its files. + +You use TiDB Cloud CLI (`ti`) to create and manage Filesystems and to work with their files. You do not need to provision or manage a separate database to use TiDB Cloud Filesystem. + +You can work with Filesystem data in two main ways: + +- **Use `ti fs` commands** to upload, download, read, organize, and search files directly without mounting the Filesystem. +- **Mount the Filesystem as a local directory** so applications and tools can work with its files through normal filesystem paths. + +For more information, see [Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) and [Mount a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-mount.md). + +## Share files across environments + +A Filesystem can be accessed from multiple machines or environments. Each environment can use its own credential instead of sharing the credentials of the user who created the Filesystem. + +For example: + +- a CI job can write build output to a Filesystem and another environment can read it later; +- an agent can continue working with files created during an earlier session; or +- a reviewer can receive read-only access to a specific directory. + +TiDB Cloud Filesystem provides owner tokens and scoped tokens so you can control which files and operations each user or workload can access. + +For more information, see [TiDB Cloud Filesystem Authorization](/tidb-cloud-filesystem/filesystem-authorization.md) and [Share a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-sharing.md). + +## Isolate and manage changes + +When multiple tasks need to work from the same files without immediately changing the shared base, you can create Filesystem layers. + +Each layer provides an isolated view where changes can be made independently. You can create checkpoints of a layer and later commit selected changes to the base Filesystem. + +For more information, see [Layers and Checkpoints](/tidb-cloud-filesystem/filesystem-layers-checkpoints.md). + +## Get started + +Choose the path that matches what you want to do: + +- **Create a new Filesystem:** follow [Get Started with TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-quick-start.md). +- **Use a Filesystem that someone else has shared with you:** see [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). +- **Check supported regions, platforms, and current limitations:** see [Regions and Limitations](/tidb-cloud-filesystem/filesystem-regions-and-limitations.md). diff --git a/tidb-cloud-filesystem/filesystem-quick-start.md b/tidb-cloud-filesystem/filesystem-quick-start.md new file mode 100644 index 0000000000000..111c297d6509e --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-quick-start.md @@ -0,0 +1,136 @@ +--- +title: Get Started with TiDB Cloud Filesystem +summary: Create a TiDB Cloud Filesystem with the CLI, write and read a file, and keep the workspace available for another session. +--- + +# Get Started with TiDB Cloud Filesystem + +Create a workspace when your application's or agent's files need to outlive the machine that produces them. This quick start writes and reads a remote file without requiring a mount. You can mount the same Filesystem afterward. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Prerequisites + +Obtain a TiDB Cloud API public key and private key from the [API Keys page](https://tidbcloud.com/org-settings/api-keys). The keys must have permission to create a Filesystem in your organization. + +If someone has already supplied you with an FS token, skip resource creation and follow [Mounting Locally](/tidb-cloud-filesystem/filesystem-mount.md#use-a-token-without-configuring-a-profile). + +## Step 1. Install the CLI + + + +
+ +Run the installer: + +```bash +# Install the CLI and its bundled Filesystem runtime. +curl -fsSL https://github.com/tidbcloud/ti-cli/releases/latest/download/install.sh | sh -s -- --yes +``` + +After installation, add the binary directory to your current shell: + +```bash +# Make the installed CLI available in this terminal. +export PATH="$HOME/.ti/bin:$PATH" +ti --version +``` + +Add the same `export PATH` line to your shell profile to use `ti` in future terminals. + +
+ +
+ +Run the installer: + +```powershell +# Download and run the PowerShell installer. +$script = "$env:TEMP\install-ti.ps1" +Invoke-WebRequest https://github.com/tidbcloud/ti-cli/releases/latest/download/install.ps1 -OutFile $script +powershell -ExecutionPolicy Bypass -File $script -Yes +``` + +After installation, add the binary directory to your current session: + +```powershell +# Make the installed CLI available in this PowerShell session. +$env:Path = "$HOME\.ti\bin;$env:Path" +ti --version +``` + +Add `$HOME\.ti\bin` to your user `PATH` for future sessions. + +
+ + +> **Note:** +> +> On Windows, the direct file commands in this quick start are supported, but native Filesystem mounts through `ti` are not. + +For other installation and upgrade details, see [Install, Configure, and Update TiDB Cloud CLI](/ai/ti/reference/ti-install-configure-update.md). + +## Step 2. Configure access + +```shell +# Follow the prompts to save your API keys and default region. +ti configure +``` + +Choose one of these Filesystem regions: + +- `aws-us-east-1` +- `aws-ap-southeast-1` +- `aws-us-west-2` +- `alicloud-ap-southeast-1` + +Choose a region where you want to store the Filesystem's data. For the provider and location of each region, see [Supported regions](/tidb-cloud-filesystem/filesystem-regions-and-limitations.md#supported-regions). + +Configuration saves the inputs locally. Your first remote command verifies the credentials with the service. + +## Step 3. Create the Filesystem + +```shell +# Wait until the new Filesystem's root is readable. +ti fs create-file-system --display-name my-workspace --wait +``` + +Copy the returned `file_system_id` for the next step. The CLI stores this Filesystem's token locally, so you do not need to export a token. Treat the returned `fs_token` as a secret; do not paste the output into a public issue or log. + +Display names help identify resources but are not unique selectors. Subsequent commands use the ID. A failed wait does not automatically delete the created Filesystem; inspect the reported resource before creating another one. If creation reports a free-plan quota error, follow its billing link to add a payment method. + +## Step 4. Write and read a file + +Replace `` with the ID returned by creation: + +```shell +# Store a file in the remote workspace. +echo "Hello from my workspace" | ti fs copy-file --file-system-id "" --from-stdin --to-remote /hello.txt +``` + +```shell +# Read the same file from the service. +ti fs read-file --file-system-id "" --path /hello.txt +``` + +The read returns `Hello from my workspace`. The file remains available after you close the terminal. There is no local mount to keep running for this workflow. + +## What's next + +- [Manage the Filesystem](/tidb-cloud-filesystem/manage-filesystem-resources.md) to inspect, check, select, and delete Filesystem resources. +- [Mount the Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) to use it through a local directory. +- [Share the workspace](/tidb-cloud-filesystem/filesystem-sharing.md) with another machine or agent. +- [Manage layers and checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md) to review changes before publishing them. + +## Clean up (optional) + +When you no longer need this tutorial Filesystem, delete it using your TiDB Cloud API credentials: + +```shell +# Permanently request deletion of only the tutorial resource. +ti fs delete-file-system --file-system-id "" +``` + +Deletion removes the remote resource and its data, not just a local registration. The command reports `deleting` when the asynchronous request is accepted. diff --git a/tidb-cloud-filesystem/filesystem-regions-and-limitations.md b/tidb-cloud-filesystem/filesystem-regions-and-limitations.md new file mode 100644 index 0000000000000..cd9c5c694a75d --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-regions-and-limitations.md @@ -0,0 +1,56 @@ +--- +title: TiDB Cloud Filesystem Regions and Limitations +summary: Review supported TiDB Cloud Filesystem regions, mount platform requirements, durability boundaries, and current product limitations. +--- + +# TiDB Cloud Filesystem Regions and Limitations + +TiDB Cloud Filesystem is available only in the regions and environments listed below. Check these boundaries before creating a Filesystem or choosing how to access its data. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Supported regions + +| Provider | Location | Canonical region code | +| --- | --- | --- | +| AWS | N. Virginia | `aws-us-east-1` | +| AWS | Oregon | `aws-us-west-2` | +| AWS | Singapore | `aws-ap-southeast-1` | +| Alibaba Cloud | Singapore | `alicloud-ap-southeast-1` | + +TiDB Cloud Starter also supports some regions where Filesystem is not available. Filesystem commands in those regions fail with an `unsupported endpoint` error. Supported Filesystem regions are built into each `ti` release. If a region was added after your installed version was released, upgrade `ti`; specifying a service URL cannot enable it. + +Filesystem list and describe commands query only the selected region. They do not aggregate resources across regions. For CLI-wide region selection and Starter availability, see [TiDB Cloud CLI Regions, Security, and Limitations](/ai/ti/reference/ti-regions-security-and-limitations.md#supported-regions). + +## Mount platform support + +| Platform | Filesystem mount | Vault mount | Requirements and alternatives | +| --- | --- | --- | --- | +| macOS | WebDAV without macFUSE; FUSE when macFUSE is installed (automatic or explicit) | FUSE | The built-in WebDAV helper supports Filesystem mounts. Install macFUSE and approve its system extension for FUSE or Vault mounts. | +| Linux | FUSE | FUSE | Install the `fuse3` package and provide access to `/dev/fuse`. WebDAV mounting is not supported. | +| Windows | Not supported | Not supported | Use `ti fs` data-plane commands and non-mount Vault commands instead. | + +FUSE and WebDAV are implemented by the bundled [Drive9](https://github.com/mem9-ai/drive9) companion. The CLI does not fall back to a separate native mount implementation. A running mount keeps the companion version loaded when it started; unmount and remount after updating `ti`. + +Ubuntu 26.04 also confines `fusermount3` with AppArmor. Use a mount path under `$HOME` or `/mnt`; `/workspace` requires an explicit local AppArmor rule even when `ti` runs as root. For setup and workarounds, see [Mount TiDB Cloud Filesystem on Linux](/tidb-cloud-filesystem/filesystem-mount-linux.md#ubuntu-2604-mount-path-restrictions). + +## Durability boundaries + +- Default FUSE behavior uses local buffering and asynchronous remote work where the companion permits it. Abruptly killing the mount process or deleting a machine can lose uncommitted memory or write-back state. +- The default coding-agent mount profile stores dependency trees, generated output, caches, and Git internals locally. Local-only data disappears when its disk disappears unless it is packed or otherwise preserved. +- Remote-committed Filesystem data survives client or sandbox deletion; deleting the machine does not delete the remote Filesystem. + +For the supported drain and graceful unmount workflow, see [Finish safely](/tidb-cloud-filesystem/filesystem-mount.md#finish-safely). `drain-file-system` is a FUSE-only online durability barrier; WebDAV does not support it. + +## Current limitations + +- Journals are append-only, and the current public command surface has no journal delete command. +- The local credential store keeps one selected token per profile and Filesystem. It does not mirror all remote tokens. Older create or import credentials without a known token ID remain usable, but cannot be correlated with remote token metadata. +- Filesystem extraction and embedding provider configuration is optional. Leaving it unconfigured does not block resource administration, file access, search, layers, Git, journal, vault, or mount workflows. +- OpenAI provider interfaces are supported for embedding and image, audio, and video extraction. Alibaba Cloud Model Studio Qwen ASR is supported only for audio extraction. Other vendors are conditionally compatible only through the exact OpenAI-compatible contract; native Anthropic, Gemini, Vertex AI, Bedrock, and Azure OpenAI interfaces are not supported. +- App-managed embedding requires a provider model that returns exactly 1024 dimensions. Filesystems that report `source=database_auto` use database-managed embedding and reject app-managed configuration. For configuration steps, see [Configure TiDB Cloud Filesystem AI Providers](/tidb-cloud-filesystem/configure-filesystem-ai-providers.md). +- Filesystem runtime operations, including direct file access, layers, mounts, Git workspaces, journals, and Vault operations, depend on the installed `ti-drive9` companion. + +For token or mount failures, see [Troubleshoot TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-troubleshooting.md). For command syntax and options, see the [Filesystem CLI command reference](/ai/ti/reference/ti-filesystem.md). diff --git a/tidb-cloud-filesystem/filesystem-sharing.md b/tidb-cloud-filesystem/filesystem-sharing.md new file mode 100644 index 0000000000000..297b03fd93fb5 --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-sharing.md @@ -0,0 +1,194 @@ +--- +title: Share a TiDB Cloud Filesystem Across Machines +summary: Learn how to share part of an existing TiDB Cloud Filesystem with another user, machine, CI job, or agent, and remove that access when it is no longer needed. +aliases: ['/ai/ti-share-filesystem-across-machines-example'] +--- + +# Share a TiDB Cloud Filesystem Across Machines + +You can share files in a TiDB Cloud Filesystem with another user, machine, CI job, or agent without copying the files between environments. + +Create a separate scoped token for each user or environment you want to share with. Each token can limit access to specific paths and actions, so you can give only the access that is needed and revoke it later without affecting anyone else. + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Prerequisites + +Before you begin: + +- Have access to an existing TiDB Cloud Filesystem. +- On a machine you trust, have an owner token for the Filesystem. You need an owner token to create scoped tokens. +- [Install TiDB Cloud CLI (`ti`)](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli) on the machine or environment that needs access to the shared Filesystem. +- Have a secure way, such as a secret manager, to transfer Filesystem tokens. + +For information about owner and scoped tokens, see [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md). + +## Give access to specific files + +To share part of a Filesystem with another user or environment: + +1. Decide which paths they need to access and what they need to do with those paths. + +2. If the path you want to share does not already exist, create it. The following example creates `/reports`: + + ```bash + ti fs create-directory \ + --file-system-id "" \ + --path /reports + ``` + +3. Create a scoped token for the user or environment. + + The following example gives a reviewer read-only access to `/reports` for 24 hours: + + ```bash + REVIEW_TOKEN="$(ti fs generate-file-system-scoped-token \ + --file-system-id "" \ + --subject reviewer \ + --ttl 24h \ + --allow /reports:read,list \ + --query fs_token \ + --output text)" + ``` + + This token lets the reviewer read and list files under `/reports`, but does not give access to other paths in the Filesystem. + + If the reviewer also needs to add or update files, include `write` for that path. For example, `--allow /reports:read,list,write` lets the reviewer read, list, and write files under `/reports` without giving them owner access to the Filesystem. + +4. Send the token and the Filesystem region code through a secure channel or secret manager. + + The other user or environment does not need your TiDB Cloud API credentials or a copy of your local `~/.ti/` directory. + + The token remains valid until it expires or you revoke it. Saving the token in an environment variable does not extend its lifetime. + +> **Note:** +> +> The token value is shown only when the token is created. Treat it as a secret and do not expose it in logs, issues, chat messages, or source control. + +For more information about token permissions and expiration, see [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md). + +## Access the shared files from another machine + +On the machine or environment that needs access: + +1. Set the scoped token and Filesystem region: + + ```bash + export TI_FS_TOKEN="" + export TI_REGION_CODE="" + ``` + + The token identifies the Filesystem, so you do not need to provide the Filesystem ID. + +2. Verify that you can access the shared path: + + ```bash + ti fs list-files --path /reports + ti fs read-file --path /reports/summary.txt + ``` + + The token can be used only for the paths and actions included in its scope. Other access is rejected by the Filesystem service. + +For other ways to access an existing Filesystem, see [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). + +### Mount the shared directory (optional) + +On a supported platform, you can also mount the shared directory and access its files through a local path: + +```bash +mkdir -p "$HOME/reports" + +ti fs mount-file-system \ + --remote-path /reports \ + --mount-path "$HOME/reports" \ + --read-only +``` + +The remote `/reports` directory becomes the root of the local mount. For example, `/reports/summary.txt` is available locally at: + +```text +$HOME/reports/summary.txt +``` + +The scoped token limits what you can do in the Filesystem. The `--read-only` option also prevents writes through this local mount. + +Direct `ti fs` commands and mounts access the same files in the Filesystem. For example, a file uploaded with `ti fs copy-file` is also available through a mount. Changes made through a mount become available to direct commands and other users after the writes reach the service. + +For mount requirements and platform-specific setup, see [Mount TiDB Cloud Filesystem Locally](/tidb-cloud-filesystem/filesystem-mount.md). + +## Make sure updates are ready to share + +When multiple users or environments access the same Filesystem, they work with the same files rather than separate copies. + +If files are written through a mount, make sure the latest changes have reached the Filesystem before telling someone else that they are ready. + +For a FUSE mount: + +1. Stop applications from writing to the files and close any files that are still open. + +2. Make sure pending writes reach the Filesystem: + + - To keep the mount running, drain it. + - If you are finished with the mount, unmount it successfully. + +3. If you need to verify the handoff, read the updated file directly from the Filesystem: + + ```bash + ti fs read-file --path /reports/summary.txt + ``` + +For a WebDAV mount, close open files and unmount normally. WebDAV does not support drain. See [Finish safely](/tidb-cloud-filesystem/filesystem-mount.md#finish-safely). + +If multiple users or environments have write access, avoid writing to the same files at the same time. TiDB Cloud Filesystem does not automatically merge conflicting changes. + +If different users or workflows need to make changes independently before applying them to the base Filesystem, see [Layers and Checkpoints](/tidb-cloud-filesystem/filesystem-layers-checkpoints.md). + +## Stop sharing access + +### On the machine using the shared Filesystem + +1. Stop applications that use the shared files. + +2. If the Filesystem is mounted, unmount it: + + ```bash + ti fs unmount-file-system --mount-path "$HOME/reports" + ``` + +3. Remove the token from the local environment: + + ```bash + unset TI_FS_TOKEN TI_REGION_CODE + ``` + +Removing the token from the local environment prevents that environment from using the saved value, but does not revoke the token itself. Anyone who still has the token can continue using it until it expires or is revoked. + +### On the machine where you manage the Filesystem + +1. Find the token you want to revoke: + + ```bash + ti fs list-file-system-tokens \ + --file-system-id "" \ + --output text + ``` + +2. Revoke that token: + + ```bash + ti fs delete-file-system-token \ + --file-system-id "" \ + --token-id "" + ``` + +Revoking one token removes that user's or environment's access without affecting other tokens or deleting the Filesystem. + +Do not delete the Filesystem just to stop sharing it with one user or environment. Deleting the Filesystem removes the shared Filesystem and its data for everyone. + +## What's next + +- [Manage TiDB Cloud Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) to create, rotate, disable, or revoke tokens. +- [TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/filesystem-layers-checkpoints.md) to make changes independently before applying them to the base Filesystem. +- [Automation and AI Agent Workflows](/tidb-cloud-filesystem/use-filesystem-for-automation-and-ai-agents.md) for workflows that use shared Filesystem data. diff --git a/tidb-cloud-filesystem/filesystem-troubleshooting.md b/tidb-cloud-filesystem/filesystem-troubleshooting.md new file mode 100644 index 0000000000000..d73b6f03aaf21 --- /dev/null +++ b/tidb-cloud-filesystem/filesystem-troubleshooting.md @@ -0,0 +1,186 @@ +--- +title: Troubleshoot TiDB Cloud Filesystem +summary: Learn how to diagnose TiDB Cloud Filesystem token, region, companion, mount, and access failures and choose a safe recovery path. +--- + +# Troubleshoot TiDB Cloud Filesystem + +Use the symptoms below to diagnose Filesystem access and mount failures. Add `--debug` only when needed, and review redacted output before sharing it. For CLI installation, API key authentication, Starter, or SQL failures, see [Troubleshoot TiDB Cloud CLI](/ai/ti/reference/ti-troubleshooting.md). + +> **Note:** +> +> TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. + +## Filesystem token is missing + +For a clean sandbox, provide the token and region. `ti` derives the Filesystem ID from the token: + +```bash +export TI_FS_TOKEN="" +export TI_REGION_CODE="" +ti fs check-file-system +``` + +The FS token is not the TiDB Cloud API private key. `TI_FS_FILE_SYSTEM_ID` is optional when a token is supplied; set it only when you want `ti` to verify that a separately distributed ID matches the token. + +If the token is known but is not stored on the current machine, import it and then select the derived ID: + +```bash +# Store a known token without requiring TiDB Cloud API keys. +chmod 600 ./fs-token +ti fs import-file-system-token --from-file ./fs-token --region +ti fs list-files --file-system-id --path / +``` + +If every known token is lost or revoked, use TiDB Cloud API keys to generate another owner token: + +```bash +ti fs generate-file-system-token \ + --file-system-id "" \ + --token-name recovery \ + --ttl 24h +``` + +The new plaintext appears once in the response. Store it securely or add `--store-locally` to select it on the current machine. + +## Filesystem token is rejected + +A data-plane HTTP 401 cannot distinguish a token that was disabled, expired, refreshed on another machine, or revoked. Inspect remote metadata with TiDB Cloud API keys: + +```bash +ti fs list-file-system-tokens \ + --file-system-id "" \ + --include-expired \ + --output text +``` + +Token names are not unique. Use the immutable `token_id` from this output for enable, disable, or delete operations. Old credentials created or imported without token lifecycle metadata can remain valid, but `ti` cannot safely identify their list row and never guesses a match. + +After enable, disable, delete, or refresh, allow approximately 10 seconds for authentication caches to converge. If refresh reports `fs.token_refresh_ambiguous`, the server might have rotated the token even though the response was lost. The outcome is unknown: the old token might still work if the refresh did not commit, or it might already be invalid. The replacement token from a committed refresh cannot be recovered because its response was lost. Do not retry the refresh with the old token. Instead, use TiDB Cloud credentials to generate an independent owner token. + +If token mutation reports `fs.token_mount_active`, use the exact mount path in the error: + +```bash +ti fs drain-file-system --mount-path /path/to/workspace +ti fs unmount-file-system --mount-path /path/to/workspace +``` + +Then retry the token operation. A mount on another machine is not visible locally; coordinate rotation with that machine separately. + +## Filesystem selection is missing + +List remote resources in the configured region with TiDB Cloud API keys and select one explicitly: + +```bash +ti fs list-file-systems --output text +ti fs list-files --file-system-id --path / +``` + +Or select the Filesystem for subsequent commands in the current shell: + +```bash +export TI_FS_FILE_SYSTEM_ID="" +``` + +The CLI intentionally does not infer a Filesystem from local credential count, including when only one credential exists. Supply its ID or an FS token whose embedded ID can be derived. + +## Filesystem region is unsupported + +The configured TiDB Cloud region might not be one of the Filesystem endpoints built into the installed `ti` release. Compare it with [supported Filesystem regions](/tidb-cloud-filesystem/filesystem-regions-and-limitations.md#supported-regions). Change placement with a valid profile or command-scoped `--region`; do not configure a raw server URL. + +## Companion is missing or incompatible + +The release installer places `ti-drive9`, the companion runtime for Filesystem commands, next to `ti`. You do not invoke `ti-drive9` directly. Re-run the current installer when the CLI reports a missing companion: + +```bash +curl -fsSL https://github.com/tidbcloud/ti-cli/releases/latest/download/install.sh | sh -s -- --yes +``` + +Verify that `PATH` resolves the expected `ti`: + +```bash +command -v ti +ti --version +``` + +Do not copy an arbitrary standalone Drive9 binary into place. + +## Filesystem creation reaches quota + +If creation returns a quota or capacity error, list existing Filesystems in the configured region before trying again: + +```bash +ti fs list-file-systems --output text +``` + +Do not delete an unrelated Filesystem to make automation pass. If the error links to TiDB Cloud billing because a payment method is required, follow that guidance before retrying. + +## Mount does not become ready + +Background mount success prints the CLI result without Drive9 startup messages. If startup fails or times out, inspect the companion log path in the error. Confirm: + +- the mount path exists and is writable; +- no existing mount covers the path; +- the FS token and region are valid; +- FUSE prerequisites or the WebDAV helper are installed; +- the remote region is reachable. + +On macOS without macFUSE, `ti` uses WebDAV. If macFUSE is installed, automatic driver selection prefers FUSE. To explicitly request FUSE: + +```bash +ti fs mount-file-system \ + --mount-path /path/to/workspace \ + --driver fuse +``` + +Linux needs FUSE support, the `fuse3` package, and access to `/dev/fuse`. Filesystem and Vault mounts are not supported on Windows; use `ti fs` data-plane commands or non-mount Vault commands instead. + +## Ubuntu 26.04 rejects a FUSE mount under `/workspace` + +Ubuntu 26.04 applies an AppArmor profile to `fusermount3`. Its default mount-path allowlist does not include `/workspace`, so root and non-root users can both receive: + +```text +/usr/bin/fusermount3: mount failed: Permission denied +``` + +Confirm the denial: + +```bash +sudo journalctl -k --since "10 minutes ago" | + grep 'profile="fusermount3"' +``` + +An entry with `operation="mount"`, `name="/workspace/"`, and `info="failed mntpnt match"` identifies this restriction. Mount under `$HOME` or `/mnt` instead: + +```bash +mkdir -p "$HOME/workspace" +ti fs mount-file-system --mount-path "$HOME/workspace" +``` + +Changing the owner or mode of `/workspace` does not bypass AppArmor. If the path cannot change, add explicit `/workspace` mount and unmount rules to `/etc/apparmor.d/local/fusermount3` as described in [Ubuntu 26.04 mount-path restrictions](/tidb-cloud-filesystem/filesystem-mount-linux.md#ubuntu-2604-mount-path-restrictions). + +## Mount becomes stale after a process crash + +If the companion is killed without graceful unmount, FUSE access can return `EIO` or `Transport endpoint is not connected`. Stop processes with open files, then try: + +```bash +ti fs unmount-file-system \ + --mount-path /path/to/workspace \ + --force +``` + +Use `--ignore-absent` when cleanup should succeed if no locator remains. Abrupt cleanup cannot guarantee recovery of pending writes from a deleted local disk. + +## Unmount reports busy + +Close editors, shells whose working directory is inside the mount, and other open file handles, and then retry: + +```bash +ti fs unmount-file-system --mount-path /path/to/workspace +``` + +Unmount performs the graceful FUSE drain automatically. Running `drain-file-system` separately does not close file descriptors or resolve a busy mount; use it only when you need to flush pending work while leaving the mount online. Drain is not supported for WebDAV. + +## Report a problem + +Include the `ti` version, OS and architecture, command name, stable error code, and redacted logs. Never include API keys, FS or vault tokens, DB passwords, SQL containing private data, or file contents. Report issues at [github.com/tidbcloud/ti-cli/issues](https://github.com/tidbcloud/ti-cli/issues). diff --git a/tidb-cloud-filesystem/manage-filesystem-layers.md b/tidb-cloud-filesystem/manage-filesystem-layers.md new file mode 100644 index 0000000000000..79cee28df0e69 --- /dev/null +++ b/tidb-cloud-filesystem/manage-filesystem-layers.md @@ -0,0 +1,172 @@ +--- +title: Manage TiDB Cloud Filesystem Layers and Checkpoints +summary: Learn how to create, inspect, checkpoint, fork, commit, roll back, and delete TiDB Cloud Filesystem layers. +aliases: ['/ai/manage-filesystem-layers'] +--- + +# Manage TiDB Cloud Filesystem Layers and Checkpoints + +A layer gives you a separate workspace for changing files without immediately affecting the base Filesystem. You can make and review changes in the layer, then decide whether to apply them to the base Filesystem or discard them. + +You can also create a checkpoint to preserve a point in the layer's history, or fork a new layer from the current layer or a checkpoint to continue working independently. + +For an overview of how layers, checkpoints, forks, and the base Filesystem relate to each other, see [Layers and Checkpoints](/tidb-cloud-filesystem/filesystem-layers-checkpoints.md). + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- Have access to an existing TiDB Cloud Filesystem with a token that provides the required read or write permissions. +- Select the Filesystem and make its token available to `ti`. For available access options, see [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). +- Choose the base path whose data the layer overlays. + +## Create a layer + +Choose the base path that you want the layer to overlay, and create a layer: + +```shell +ti fs create-layer \ + --base-root-path /workspace \ + --layer-name agent-task \ + --durability-mode restore-safe \ + --tag task=review +``` + +`--base-root-path` determines which part of the base Filesystem the layer overlays. + +`restore-safe` is the only `--durability-mode` value accepted by the current CLI. For all available options, see the [`create-layer` command reference](/ai/ti/reference/ti-fs-create-layer.md). + +The command returns a layer ID. Use the layer ID for subsequent operations, especially in automation, because layer names are not guaranteed to be unique. + +## Work with and inspect layer changes + +Write a file to the layer by specifying its layer ID: + +```shell +ti fs copy-file \ + --from-local ./proposal.md \ + --to-remote /workspace/proposal.md \ + --layer-id "" +``` + +Inspect the layer and its changes: + +```shell +ti fs describe-layer --layer-id "" +ti fs diff-layer --layer-id "" +``` + +List all layers in the selected Filesystem: + +```shell +ti fs list-layers --output text +``` + +Changes that have not been committed remain in the layer. File operations that do not select the layer access the base Filesystem and do not show its uncommitted changes. + +> **Note:** +> +> `copy-file` with `--layer-id` does not support recursive copy. To copy a directory tree into a layer, mount the layer as a writable FUSE mount and copy files through the mount path. +> +> Do not mount the same writable layer at multiple local paths concurrently. Reuse its existing mount, or unmount it before mounting the layer elsewhere. + +## Create a checkpoint + +A checkpoint preserves a point in the layer's durable history. + +If the layer has an active writable FUSE mount, drain pending writes before creating the checkpoint so that the checkpoint includes the changes that have reached the service: + +```shell +ti fs drain-file-system \ + --mount-path "/path/to/workspace" +``` + +Then create the checkpoint: + +```shell +ti fs create-layer-checkpoint \ + --layer-id "" \ + --checkpoint-id seed \ + --label "before review" +``` + +A checkpoint mount is read-only. To continue making changes from a checkpoint, fork a new writable layer. + +## Fork a layer + +Fork a new writable layer from the current layer or one of its checkpoints: + +```shell +ti fs fork-layer \ + --parent-layer-ref "" \ + --layer-name experiment \ + --checkpoint-id seed +``` + +Use the layer ID returned for the fork when you perform subsequent operations on it. + +To inspect the fork's pinned ancestry: + +```shell +ti fs list-layer-chain --layer-ref "" +``` + +After a fork is created, changes made to the parent and child layers are independent. + +## Commit or discard layer changes + +Before committing or rolling back a layer with an active writable FUSE mount, stop applications that are writing to the mount, drain pending writes, and unmount it: + +```shell +ti fs drain-file-system \ + --mount-path "/path/to/workspace" + +ti fs unmount-file-system \ + --mount-path "/path/to/workspace" +``` + +For more information about safely finishing mount activity, see [Finish safely](/tidb-cloud-filesystem/filesystem-mount.md#finish-safely). + +To apply the layer's changes to the base Filesystem: + +```shell +ti fs commit-layer --layer-id "" +``` + +A commit applies the layer's effective changes to the base Filesystem. If the layer was created by forking another layer, committing it does not merge the changes back into its parent layer. + +If the base Filesystem contains conflicting changes, the commit can fail instead of automatically merging them. Keep the layer and inspect its changes and the base Filesystem before deciding how to proceed. + +To discard the layer's uncommitted changes instead: + +```shell +ti fs rollback-layer --layer-id "" +``` + +Rollback discards the current layer changes. It does not reset the layer to an earlier checkpoint. To continue from a checkpoint, fork a new layer from that checkpoint. + +## Delete a layer + +When you no longer need a layer, delete it by its layer ID: + +```shell +ti fs delete-layer --layer-ref "" +``` + +Deleting a layer abandons it without immediately erasing all of its history. If the layer has live descendants, the command fails by default. + +To abandon the layer and all of its live descendants, use `--cascade`: + +```shell +ti fs delete-layer \ + --layer-ref "" \ + --cascade +``` + +Use `--cascade` only when you intend to abandon the descendant layers as well. + +## What's next + +- [Mount TiDB Cloud Filesystem Locally](/tidb-cloud-filesystem/filesystem-mount.md) +- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/tidb-cloud-filesystem/manage-filesystem-resources.md b/tidb-cloud-filesystem/manage-filesystem-resources.md new file mode 100644 index 0000000000000..4d1881a11410b --- /dev/null +++ b/tidb-cloud-filesystem/manage-filesystem-resources.md @@ -0,0 +1,103 @@ +--- +title: Manage TiDB Cloud Filesystems +summary: Learn how to create, inspect, check, select, and delete TiDB Cloud Filesystems by using TiDB Cloud CLI. +aliases: ['/ai/manage-filesystem-resources'] +--- + +# Manage TiDB Cloud Filesystems + +You can use [TiDB Cloud CLI (`ti`)](/ai/ti/ti-overview.md) to create, inspect, check, select, and delete TiDB Cloud Filesystems. + +For command syntax, flags, and output fields, see the [`ti fs` command reference](/ai/ti/reference/ti-filesystem.md). + +## Prerequisites + +Before you begin, follow [Get Started with TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-quick-start.md) to install TiDB Cloud CLI (`ti`) and configure the access. + +## Create a Filesystem + +Create a Filesystem and wait until it is ready: + +```shell +ti fs create-file-system \ + --display-name agent-workspace \ + --label environment=development \ + --wait +``` + +The response includes the Filesystem ID and an owner FS token in the `fs_token` field. The CLI automatically stores the owner FS token locally. + +Copy the returned `file_system_id` and select the Filesystem for subsequent commands in the current shell: + +```shell +export TI_FS_FILE_SYSTEM_ID="" +``` + +Setting `TI_FS_FILE_SYSTEM_ID` lets subsequent commands identify the target Filesystem without requiring `--file-system-id` on every command. For data-access commands, the CLI uses the locally stored FS token for the selected Filesystem. + +> **Warning:** +> +> The owner FS token plaintext in `fs_token` is returned only when the token is issued. Treat it as a secret and do not expose it in logs, issues, or source control. If you need to use the token on another machine or store a backup, save it securely in a secret manager. For more information, see [Manage Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md). + +> **Note:** +> +> Do not put credentials, connection strings, private paths, or personal data in Filesystem labels. + +## List and inspect Filesystems + +List the Filesystems available in the current region: + +```shell +ti fs list-file-systems --output text +``` + +View metadata for a Filesystem: + +```shell +ti fs describe-file-system --file-system-id "" +``` + +If you work with more than one Filesystem, specify the target Filesystem in one of the following ways: + +- Pass `--file-system-id ""` to an individual command. +- Set `TI_FS_FILE_SYSTEM_ID` to select a Filesystem for subsequent commands in the current shell. + +The CLI does not automatically select a Filesystem based on the number of Filesystems or locally stored credentials. + +The current CLI does not provide a command to change a Filesystem's display name or labels after creation. Choose these values when you create the Filesystem. + +## Check access + +Check whether the CLI can access a Filesystem: + +```shell +ti fs check-file-system --file-system-id "" +``` + +The result includes an overall `status` and checks local credentials, endpoint selection, the bundled Filesystem runtime, and remote connectivity. + +- `passed` means all checks succeeded. +- `warning` or `failed` identifies a check that needs attention. + +For common access and connectivity issues, see [Troubleshoot TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-troubleshooting.md). + +## Delete a Filesystem + +> **Warning:** +> +> Deleting a Filesystem permanently removes its remote data. Before deletion, stop applications that are using the Filesystem and successfully unmount any active local mounts. For information about finishing pending writes safely, see [Finish safely](/tidb-cloud-filesystem/filesystem-mount.md#finish-safely). + +Delete a Filesystem by its ID: + +```shell +ti fs delete-file-system --file-system-id "" +``` + +Filesystem deletion is asynchronous. After the service accepts the request, the CLI reports the Filesystem status as `deleting` and removes the matching locally stored credential. This status means that deletion has started, not that the remote Filesystem and its data have already been removed. + +## What's next + +- [Manage Filesystem Tokens](/tidb-cloud-filesystem/manage-filesystem-tokens.md) to generate, delegate, rotate, or revoke Filesystem access. +- [Work with Files and Directories](/tidb-cloud-filesystem/work-with-filesystem-data.md) to copy, read, organize, and search Filesystem data. +- [Mount a Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) to access remote files through a local directory. +- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) for command syntax, flags, and output fields. \ No newline at end of file diff --git a/tidb-cloud-filesystem/manage-filesystem-tokens.md b/tidb-cloud-filesystem/manage-filesystem-tokens.md new file mode 100644 index 0000000000000..b24059efba10b --- /dev/null +++ b/tidb-cloud-filesystem/manage-filesystem-tokens.md @@ -0,0 +1,126 @@ +--- +title: Manage TiDB Cloud Filesystem Tokens +summary: Learn how to import, generate, scope, inspect, disable, refresh, and revoke access tokens for a TiDB Cloud Filesystem. +aliases: ['/ai/manage-filesystem-tokens'] +--- + +# Manage TiDB Cloud Filesystem Tokens + +Filesystem tokens let you give users, applications, and automation access to a TiDB Cloud Filesystem without sharing your TiDB Cloud API credentials. + +You can use an owner token for full access to a Filesystem, or create scoped tokens that limit access to specific paths and operations. For more information about token types and permissions, see [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md). + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- Have access to an existing TiDB Cloud Filesystem. If you do not have one, follow [Get Started with TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-quick-start.md) to create one. + +Some token management operations require TiDB Cloud API credentials or an existing owner token. The relevant requirements are described in each section of this guide. + +> **Note:** +> +> Treat Filesystem tokens as secrets. When a command creates or refreshes a token, the token plaintext is returned only once and cannot be retrieved later. + +## Import an existing token + +If you already have a Filesystem token, import it to the local CLI credential store: + +```shell +ti fs import-file-system-token --from-file ./fs-token --region aws-us-east-1 +``` + +The CLI validates the token, extracts the Filesystem ID from it, verifies connectivity, and stores the token locally. + +## Generate an owner token + +To generate another owner token, configure TiDB Cloud API credentials and obtain the Filesystem ID. + +Generate the token and save its one-time plaintext response securely: + +```shell +umask 077 +ti fs generate-file-system-token \ + --file-system-id "" \ + --token-name ci \ + --ttl 24h > ./ci-token.json +``` + +The CLI does not store the generated token locally by default. To store it locally, add `--store-locally`. If a different token is already stored for this Filesystem, also add `--replace`. + +## Generate and delegate a scoped token + +To generate a scoped token, use an existing owner token on a trusted machine. Provide the owner token through `--fs-token` or `TI_FS_TOKEN`, or use the token stored locally for the selected Filesystem. + +The following example uses the locally stored owner token and creates a scoped token that allows an agent to read, list, and write files under `/workspace`: + +```shell +SCOPED_TOKEN="$(ti fs generate-file-system-scoped-token \ + --file-system-id "" \ + --subject report-agent \ + --ttl 24h \ + --allow /workspace:read,list,write \ + --query fs_token --output text)" +``` + +Transfer the token through a secret manager. In the receiving environment, provide the scoped token and the Filesystem region: + +```shell +export TI_FS_TOKEN="" +export TI_REGION_CODE="" + +ti fs list-files --path /workspace +``` + +The `--allow` value uses the format `:`. Supported operations are `read`, `list`, `search`, `write`, and `delete`; `search` requires `read`. In this example, the token permits `read`, `list`, and `write` operations under `/workspace`. + +The remote `/workspace` directory must already exist. To use this token for a mount, specify `--remote-path /workspace`. A token restricted to `/workspace` cannot mount the Filesystem root `/`. + +For more information about scoped permissions and credential selection, see [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md). + +## Inspect and change token status + +List non-secret metadata for Filesystem tokens: + +```shell +ti fs list-file-system-tokens \ + --file-system-id "" \ + --output text +``` + +The output does not include token plaintext. If you lose an owner token, generate a replacement using TiDB Cloud API credentials. You cannot recover the original token by listing tokens. + +Use [`disable-file-system-token`](/ai/ti/reference/ti-fs-disable-file-system-token.md) to temporarily suspend a token, and [`enable-file-system-token`](/ai/ti/reference/ti-fs-enable-file-system-token.md) to restore it. + +## Rotate or revoke a token + +Use [`refresh-file-system-token`](/ai/ti/reference/ti-fs-refresh-file-system-token.md) to rotate a Filesystem token. + +When you refresh a locally stored token, the CLI automatically updates the local credential. When you refresh a token provided through `--fs-token` or `TI_FS_TOKEN`, the CLI returns the new token in the command output without storing it locally. + +> **Note:** +> +> Refresh is non-idempotent. For example, after a network timeout, the service might have rotated the token even though you did not receive the new value. Do not retry the refresh with the old token. Instead, generate a new owner token using TiDB Cloud API credentials. + +> **Warning:** +> +> Before rotating, disabling, or deleting a token used by an active mount, stop applications that are writing to the mount and successfully unmount it. The CLI can detect known local mounts but cannot discover mounts on other machines. Coordinate with those machines before changing the token. For more information, see [Finish safely](/tidb-cloud-filesystem/filesystem-mount.md#finish-safely). + +Before retiring a token, distribute and validate its replacement. Then revoke the old token by its token ID: + +```shell +ti fs delete-file-system-token \ + --file-system-id "" \ + --token-id "" +``` + +If the deleted token matches the locally stored token, the CLI automatically removes the local credential. Token changes can take time to propagate through authorization caches. + +Disabling or revoking an owner token does not automatically revoke scoped tokens generated from it. Review and revoke those scoped tokens separately when necessary. + +## What's next + +- [Share a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-sharing.md) +- [Explore automation and AI agent workflows](/tidb-cloud-filesystem/use-filesystem-for-automation-and-ai-agents.md) +- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) diff --git a/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md b/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md new file mode 100644 index 0000000000000..feb93c72dba66 --- /dev/null +++ b/tidb-cloud-filesystem/manage-filesystem-vault-secrets.md @@ -0,0 +1,180 @@ +--- +title: Manage TiDB Cloud Filesystem Vault Secrets +summary: Learn how to store and rotate secrets, delegate temporary access, inject secrets into processes, audit and revoke access, and optionally mount secrets as read-only files. +aliases: ['/ai/manage-filesystem-vault-secrets'] +--- + +# Manage TiDB Cloud Filesystem Vault Secrets + +Use TiDB Cloud Filesystem Vault when an application, automation, or agent needs credentials or other sensitive values, but you do not want to store those values in regular Filesystem files or give the workflow broad access to the Filesystem. + +With Vault, a trusted owner can store a secret once and grant access to only the secret or field that a user, application, or agent needs, for a limited time. The delegated workflow can then read the permitted value, inject it into a process, or access it through a read-only mount. The owner can audit the access and revoke the grant when it is no longer needed. + +This guide shows you how to store and rotate secrets, delegate limited access, use delegated secrets, audit and revoke access, and optionally mount secrets as files. + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- Have access to an existing TiDB Cloud Filesystem. +- Make the Filesystem and its owner token available to `ti`. See [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). + +An owner token is used to create and replace secrets, create and revoke grants, and view audit events. A delegated Vault token provides only the secret access allowed by its grant. + +Treat both owner tokens and delegated Vault tokens as credentials. Do not expose them in logs, source control, shared terminal output, or command-line arguments. + +## Create a secret + +A Vault secret can contain multiple named fields. For example, a database secret might contain a connection URL and a password. + +Create a secret named `db-prod`: + +```shell +ti fs-vault create-secret \ + --secret-name db-prod \ + --field DB_URL=mysql://example \ + --field PASSWORD=@./password.txt +``` + +In `PASSWORD=@./password.txt`, the `@` prefix tells `ti` to read the field value from the local file instead of treating the file path as the value. + +Some Vault commands identify a secret by name, such as `db-prod`. Commands that operate on a specific secret path, such as `replace-secret` and `run-with-secret`, use its full Vault path instead. For example, the Vault path of `db-prod` is `/n/vault/db-prod`. + +### Read a secret value + +`read-secret` returns plaintext secret values. Use it only when you need the value directly, and make sure its output is not written to logs or other unintended destinations. + +For example, to read only the `DB_URL` field: + +```shell +ti fs-vault read-secret \ + --secret-name db-prod \ + --field DB_URL \ + --format raw +``` + +When an application needs the secret, prefer [injecting it into the process](#inject-a-secret-into-a-process) instead of reading and handling the plaintext value yourself. + +## Rotate a secret + +`replace-secret` replaces all fields in the secret, not just the field whose value changed. + +To rotate `DB_URL`, create a local directory containing the new `DB_URL` value and the current `PASSWORD` value that you want to keep: + +```text +./secret-fields/ +├── DB_URL +└── PASSWORD +``` + +Then replace the secret: + +```shell +ti fs-vault replace-secret \ + --secret-path /n/vault/db-prod \ + --from-directory ./secret-fields +``` + +Each file in the directory becomes a field in the replacement secret. Any existing field that is not included in the directory is not retained. + +Keep these local files out of source control and remove them when they are no longer needed. For details, see the [`replace-secret` reference](/ai/ti/reference/ti-fs-vault-replace-secret.md). + +## Delegate limited access + +Instead of sharing the Filesystem owner token, create a short-lived grant for only the secret fields that another user, application, or agent needs. + +For example, the following grant allows `deploy-agent` to read only the `DB_URL` field for 10 minutes: + +```shell +ti fs-vault create-grant \ + --agent-id deploy-agent \ + --scope db-prod/DB_URL \ + --permission read \ + --ttl 10m +``` + +The command returns a delegated Vault token and a grant ID. Give the delegated token only to the workflow that needs the secret, and retain the grant ID so that you can revoke the grant before it expires if necessary. + +In the environment that uses the delegated secret, make the token available as `TI_VAULT_TOKEN`. Avoid putting the token directly in a command-line argument because command arguments can appear in shell history or process listings. + +## Inject a secret into a process + +If an application can receive credentials through environment variables, use `run-with-secret` to make the permitted secret fields available only to the child process: + +```shell +ti fs-vault run-with-secret \ + --secret-path /n/vault/db-prod \ + -- +``` + +Each permitted secret field becomes an environment variable with the same name. For example, the `DB_URL` and `PASSWORD` fields become the `DB_URL` and `PASSWORD` environment variables. + +The Vault credential used by `ti` is not passed to the child process. This lets the application use the secret without writing its plaintext value to a file. + +Field names used with `run-with-secret` must match `[A-Z_][A-Z0-9_]*`. Use uppercase environment-variable-style field names for secrets that you plan to inject into a process. + +## Audit and revoke access + +To review recent access to `db-prod` by `deploy-agent`, run: + +```shell +ti fs-vault list-audit-events \ + --secret-name db-prod \ + --agent-id deploy-agent \ + --since 24h \ + --limit 20 +``` + +When the delegated access is no longer needed, revoke the grant using the grant ID returned by `create-grant`: + +```shell +ti fs-vault delete-grant \ + --grant-id "" \ + --revoked-by operator \ + --reason task-complete +``` + +Revoking a grant prevents the delegated token from authorizing new operations. It cannot remove a secret value that a process has already read. + +## Mount secrets as read-only files + +If an application expects credentials as files instead of environment variables, you can optionally expose permitted Vault fields through a read-only FUSE mount on Linux or macOS. + +For delegated access, first make the delegated Vault token available as `TI_VAULT_TOKEN`. Then create a local mount directory and mount the Vault: + +```shell +mkdir -p /path/to/vault + +ti fs-vault mount-vault \ + --mount-path /path/to/vault +``` + +The permitted secret fields are available as files under the mount path. For example: + +```text +/path/to/vault/db-prod/DB_URL +``` + +Processes that can access the mount can read the permitted secret values, so keep access to the mount limited to the intended workload. + +Before unmounting, stop processes that are using the mounted secrets: + +```shell +ti fs-vault unmount-vault \ + --mount-path /path/to/vault +``` + +Vault mounts require FUSE and are not available on Windows. Direct secret reads and `run-with-secret` do not require a mount. + +## Security recommendations + +- Grant access only to the secret fields required by the workflow and use the shortest practical TTL. +- Prefer `run-with-secret` when an application can receive credentials through environment variables. +- Do not expose owner or delegated tokens in logs, source control, or command-line arguments. +- Revoke grants when their tasks finish or access is no longer needed. + +## What's next + +- [Delegate TiDB Cloud Filesystem Vault Secrets to an Agent](/ai/ti/guides/ti-vault-agent-secrets-example.md) +- [TiDB Cloud Filesystem Vault CLI Command Reference](/ai/ti/reference/ti-filesystem-vault.md) diff --git a/tidb-cloud-filesystem/manage-git-workspaces.md b/tidb-cloud-filesystem/manage-git-workspaces.md new file mode 100644 index 0000000000000..21998035cab76 --- /dev/null +++ b/tidb-cloud-filesystem/manage-git-workspaces.md @@ -0,0 +1,122 @@ +--- +title: Manage Git Workspaces on TiDB Cloud Filesystem +summary: Learn how to clone Git repositories into a mounted TiDB Cloud Filesystem, speed up large repository setup, and manage linked worktrees. +aliases: ['/ai/manage-git-workspaces'] +--- + +# Manage Git Workspaces on TiDB Cloud Filesystem + +If you want to work with a Git repository directly on a mounted TiDB Cloud Filesystem, you can follow this guide to set up and manage the Git workspace. + +This is especially useful for large repositories, where you can start working before all file contents finish downloading, or when you need separate working directories for different branches without cloning the repository multiple times. This guide covers cloning a repository, completing background downloads when needed, and creating and removing linked Git worktrees. + +After the workspace is set up, you can continue to use ordinary Git commands such as `git status`, `git add`, `git commit`, `git fetch`, and `git push` for your daily Git work. + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- [Mount a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) through FUSE. Git workspaces are not supported on WebDAV mounts. +- Make the mounted Filesystem and a token with the required permissions available to `ti`. See [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). +- Install Git and configure authentication for the repository you want to use. + +All Git workspaces in this guide are created inside the mounted Filesystem path. + +## Clone a Git repository + +To clone a Git repository into your mounted Filesystem directory, use the `ti fs-git clone-git-workspace` command and specify the repository URL and target path as follows: + +```shell +ti fs-git clone-git-workspace \ + --repo-url https://github.com/pingcap/tidb.git \ + --target-path /path/to/workspace/tidb +``` + +The repository is cloned to `/path/to/workspace/tidb`. + +For a large repository, you can start working before all file contents finish downloading by specifying `--blobless`: + +```shell +ti fs-git clone-git-workspace \ + --repo-url https://github.com/pingcap/tidb.git \ + --target-path /path/to/workspace/tidb \ + --blobless +``` + +With `--blobless`, `ti` downloads the repository structure and Git metadata first. When `--hydrate` is `auto` (the default), `ti` then continues downloading the remaining file contents in the background. You can start working with the repository while this download continues. To disable background hydration, specify `--hydrate off`. + +If you need all file contents to finish downloading before the clone command returns, also specify `--hydrate sync`: + +```shell +ti fs-git clone-git-workspace \ + --repo-url https://github.com/pingcap/tidb.git \ + --target-path /path/to/workspace/tidb \ + --blobless \ + --hydrate sync +``` + +## Finish downloading a blobless workspace + +If you created a workspace with `--blobless`, you can later wait for any remaining file contents to finish downloading: + +```shell +ti fs-git hydrate-git-workspace \ + --target-path /path/to/workspace/tidb \ + --timeout 30m +``` + +Run this command when you want to make sure the remaining Git file data has finished downloading before you continue. For example, you might do this before a task that needs repository file contents to be available. + +This process, called hydration, downloads the Git data that is still missing from the workspace without discarding changes you have already made to files. + +If cloning or hydration fails, check the CLI error and diagnostic log before retrying. See [Troubleshoot TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-troubleshooting.md). + +## Create a linked worktree + +Use a linked worktree when you want a separate working directory for another branch without cloning the repository again. The new worktree shares Git data with the base workspace. + +For example, create a worktree for a new `feature-x` branch: + +```shell +ti fs-git add-git-worktree \ + --base-path /path/to/workspace/tidb \ + --worktree-path /path/to/workspace/tidb-feature \ + --branch-name feature-x +``` + +After the worktree is created, use ordinary Git commands in `/path/to/workspace/tidb-feature`: + +```shell +git -C /path/to/workspace/tidb-feature status +``` + +For other options, such as creating a detached worktree at a specific commit, see the [`add-git-worktree` command reference](/ai/ti/reference/ti-fs-git-add-git-worktree.md). + +## Remove a linked worktree + +When you no longer need a linked worktree, remove it: + +```shell +ti fs-git remove-git-worktree \ + --worktree-path /path/to/workspace/tidb-feature +``` + +Removing a linked worktree does not remove the base workspace or the Git data shared by other worktrees. + +If the worktree contains uncommitted changes, the command refuses to remove it. Commit or preserve any changes you need before removing the worktree. + +Use `--force` only when you are sure that the uncommitted changes can be discarded: + +```shell +ti fs-git remove-git-worktree \ + --worktree-path /path/to/workspace/tidb-feature \ + --force +``` + +When you are finished using the mounted Filesystem, commit or push any Git changes you want to preserve and [unmount the Filesystem safely](/tidb-cloud-filesystem/filesystem-mount.md#unmount-when-you-are-finished). + +## What's next + +- [Prepare a Git Workspace for Agents on TiDB Cloud Filesystem](/ai/ti/guides/ti-git-workspace-for-agents-example.md) for an agent workflow that uses blobless cloning and background hydration. +- [TiDB Cloud Filesystem Git CLI Command Reference](/ai/ti/reference/ti-filesystem-git.md) for all `ti fs-git` commands and options. diff --git a/tidb-cloud-filesystem/use-filesystem-for-automation-and-ai-agents.md b/tidb-cloud-filesystem/use-filesystem-for-automation-and-ai-agents.md new file mode 100644 index 0000000000000..503462d75e97f --- /dev/null +++ b/tidb-cloud-filesystem/use-filesystem-for-automation-and-ai-agents.md @@ -0,0 +1,27 @@ +--- +title: Explore Automation and AI Agent Workflows +summary: Find TiDB for AI examples for sharing Filesystem workspaces, handing off CI artifacts, and supporting agents with data, Git, journals, and secrets. +--- + +# Explore Automation and AI Agent Workflows + +TiDB Cloud Filesystem keeps files available when a machine, CI job, or agent sandbox is temporary. Choose a workflow below based on what you need to share or preserve. + +If you are new to Filesystem, start with the [Quick Start](/tidb-cloud-filesystem/filesystem-quick-start.md) to create a Filesystem and work with your first file. + +To share work across machines and CI jobs, see the following guides in the **TiDB for AI** documentation. + +| What you want to do | Guide | +| --- | --- | +| Pass build artifacts between isolated CI jobs | [Hand Off CI Artifacts Between Isolated Jobs with TiDB Cloud Filesystem](/ai/ti/guides/ti-ci-artifact-handoff-example.md) | + +To support AI agent workflows, see the following guides in the **TiDB for AI** documentation. If you are evaluating Filesystem for agents, start with the sandbox example before moving to longer-running or parallel workflows. + +| What you want to do | Guide | +| --- | --- | +| Give an agent sandbox access to a shared workspace | [Use TiDB Cloud Filesystem in an Agent Sandbox](/ai/ti/guides/ti-agent-sandbox-example.md) | +| Keep agent state across disposable sandboxes | [Persist Agent State Across Disposable Sandboxes with TiDB Cloud Filesystem](/ai/ti/guides/ti-persistent-agent-state-example.md) | +| Share a read-only dataset with parallel agents | [Share a Read-Only Dataset Across Parallel Agents with TiDB Cloud Filesystem](/ai/ti/guides/ti-parallel-agent-dataset-example.md) | +| Prepare a large Git workspace for an agent | [Prepare a Git Workspace for Agents on TiDB Cloud Filesystem](/ai/ti/guides/ti-git-workspace-for-agents-example.md) | +| Record and verify an agent workflow | [Record an Agent Workflow in a TiDB Cloud Filesystem Journal](/ai/ti/guides/ti-journal-agent-workflow-example.md) | +| Give an agent limited access to a Vault secret | [Delegate TiDB Cloud Filesystem Vault Secrets to an Agent](/ai/ti/guides/ti-vault-agent-secrets-example.md) | diff --git a/tidb-cloud-filesystem/use-filesystem-journals.md b/tidb-cloud-filesystem/use-filesystem-journals.md new file mode 100644 index 0000000000000..feecdc019c183 --- /dev/null +++ b/tidb-cloud-filesystem/use-filesystem-journals.md @@ -0,0 +1,104 @@ +--- +title: Use TiDB Cloud Filesystem Journals +summary: Learn how to record, read, search, and verify ordered events from agent and automation workflows in a TiDB Cloud Filesystem. +aliases: ['/ai/use-filesystem-journals'] +--- + +# Use TiDB Cloud Filesystem Journals + +Use a journal when you need an ordered, persistent record of events from an agent or automation workflow. For example, a journal can record when a task starts or finishes, which agent performed an action, and when work is handed off between agents or processes. You can later read or search these events to understand what happened during the workflow. + +Journal entries are append-only: new events are added as new entries, and existing entries cannot be modified. The entries are also linked through a hash chain, which lets you verify that the recorded history remains intact and in order. + +This guide shows you how to create a journal, record events, read and search recorded events, and verify the journal history. + +Journals are intended for workflow events and history. Store artifacts, working files, and other workflow outputs as regular files in the Filesystem. A journal records what happened; it does not replay workflow actions or replace the files produced by the workflow. + +> **Note:** +> +> The current `ti` CLI does not provide a command to delete individual journals. Do not record secrets or other data that you might need to remove later. + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- Have access to an existing TiDB Cloud Filesystem. +- Make the Filesystem and its token available to `ti`. See [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). + +## Create a journal + +Create a journal for the workflow you want to record: + +```shell +ti fs-journal create-journal \ + --journal-kind agent \ + --title "review task" \ + --actor agent:reviewer +``` + +Because no journal ID is specified, the service generates one. Save the returned journal ID—you will use it to append, read, and verify entries in this journal. + +The journal kind, title, and actor provide context that can also help you find related workflow records later. + +## Append entries + +Append an event to the journal: + +```shell +ti fs-journal append-journal-entries \ + --journal-id "" \ + --entry-json '{"type":"review_started"}' +``` + +Each entry needs a `type`, unless you provide one with `--entry-type`. You can also include fields such as `summary`, `actor`, and `occurred_at`. + +The example above omits `--idempotency-key` for brevity. If your workflow might retry the same append operation, provide an idempotency key and reuse the same key for every retry. This prevents the retry from recording the same event more than once: + +```shell +ti fs-journal append-journal-entries \ + --journal-id "" \ + --idempotency-key review-started \ + --entry-json '{"type":"review_started"}' +``` + +For all supported fields and input formats, see the [`append-journal-entries` reference](/ai/ti/reference/ti-fs-journal-append-journal-entries.md). + +## Read and search entries + +To review the history of one journal, read its entries: + +```shell +ti fs-journal read-journal-entries \ + --journal-id "" +``` + +Entries are returned in sequence order, so you can follow the workflow in the order it was recorded. + +To find events across journals in the selected Filesystem, use `search-journal-entries`. For example, the following command finds `review_started` events and returns their entry contents: + +```shell +ti fs-journal search-journal-entries \ + --entry-type review_started \ + --include-entries +``` + +Unlike `read-journal-entries`, the search command is not limited to one journal. Use filters such as journal kind, actor, entry type, labels, or time range to narrow the results. + +## Verify a journal + +To check that the stored journal history is internally consistent, verify its hash chain: + +```shell +ti fs-journal verify-journal \ + --journal-id "" +``` + +A successful verification confirms that the stored sequence and hash chain are consistent. + +Hash-chain verification checks the integrity of the recorded journal history. It does not prove that the original event information recorded by an agent or application was accurate. + +## What's next + +- [Record an Agent Workflow in a TiDB Cloud Filesystem Journal](/ai/ti/guides/ti-journal-agent-workflow-example.md) for an end-to-end agent workflow example. +- [TiDB Cloud Filesystem Journal CLI Command Reference](/ai/ti/reference/ti-filesystem-journal.md) for all journal commands and options. diff --git a/tidb-cloud-filesystem/work-with-filesystem-data.md b/tidb-cloud-filesystem/work-with-filesystem-data.md new file mode 100644 index 0000000000000..f7aed56fac2a6 --- /dev/null +++ b/tidb-cloud-filesystem/work-with-filesystem-data.md @@ -0,0 +1,135 @@ +--- +title: Work with Files and Directories in TiDB Cloud Filesystem +summary: Learn how to upload, download, read, organize, inspect, and search files and directories in TiDB Cloud Filesystem. +aliases: ['/ai/work-with-filesystem-data'] +--- + +# Work with Files and Directories in TiDB Cloud Filesystem + +You can work with files and directories in a TiDB Cloud Filesystem directly from TiDB Cloud CLI (`ti`) without mounting the Filesystem. + +Use `ti fs` commands to upload and download files, read file contents, list and inspect paths, organize files and directories, and search for data. + +For the complete command reference, see [`ti fs`](/ai/ti/reference/ti-filesystem.md). + +## Prerequisites + +Before you begin: + +- [Install TiDB Cloud CLI](/tidb-cloud-filesystem/filesystem-quick-start.md#step-1-install-the-cli). +- [Create a Filesystem](/tidb-cloud-filesystem/manage-filesystem-resources.md) or obtain access to an existing one. +- Select the Filesystem and make its token available to `ti`. For available access options, see [Access an Existing TiDB Cloud Filesystem](/tidb-cloud-filesystem/access-filesystem.md). + +## Upload and download files + +Upload a local file to the Filesystem: + +```shell +ti fs copy-file \ + --from-local ./report.md \ + --to-remote /reports/report.md +``` + +Download a file from the Filesystem: + +```shell +ti fs copy-file \ + --from-remote /reports/report.md \ + --to-local ./downloads/report.md \ + --create-parents +``` + +You can also use `copy-file` to copy files or directories within the Filesystem, stream data through standard input or output, append to a file, or resume an interrupted transfer. + +To copy a directory recursively, use `--recursive`. For all supported copy operations and options, see the [`copy-file` reference](/ai/ti/reference/ti-fs-copy-file.md). + +## Read and inspect files and directories + +Read the complete contents of a file: + +```shell +ti fs read-file --path /reports/report.md +``` + +To read only part of a file, use `--offset` and `--length`. For example, the following command reads the first 1024 bytes: + +```shell +ti fs read-file \ + --path /reports/report.md \ + --offset 0 \ + --length 1024 +``` + +List the contents of a directory: + +```shell +ti fs list-files --path /reports --output text +``` + +Inspect metadata for a file or directory: + +```shell +ti fs describe-file --path /reports/report.md +``` + +## Organize files and directories + +Create a directory: + +```shell +ti fs create-directory --path /reports/archive +``` + +Move a file to another path: + +```shell +ti fs move-file \ + --from-remote /draft.md \ + --to-remote /reports/final.md +``` + +Delete a file or directory: + +```shell +ti fs delete-file --path /scratch --recursive +``` + +> **Warning:** +> +> `delete-file --recursive` permanently deletes the specified directory and its contents. Verify the path before deleting it. You can use `--dry-run` to validate the request without deleting data. + +For workflows that need POSIX-style metadata or links, you can also use [`chmod-file`](/ai/ti/reference/ti-fs-chmod-file.md), [`create-symlink`](/ai/ti/reference/ti-fs-create-symlink.md), and [`create-hardlink`](/ai/ti/reference/ti-fs-create-hardlink.md). + +## Search files and content + +Use `search-file-content` when you want to find files based on their content: + +```shell +ti fs search-file-content \ + --path /reports \ + --pattern "TODO" +``` + +`--pattern` is a text query, not a regular expression or glob. Use `--limit` to control the maximum number of results. + +For details, see the [`search-file-content` reference](/ai/ti/reference/ti-fs-search-file-content.md). + +If you know something about the file itself rather than its contents, use `find-files`. You can filter by file name, type, tags, size, or modification time. + +For example, find Markdown files tagged `stage=review`: + +```shell +ti fs find-files \ + --path /reports \ + --file-name-pattern "*.md" \ + --tag stage=review +``` + +For all available filters, see the [`find-files` reference](/ai/ti/reference/ti-fs-find-files.md). + +## What's next + +- [Mount a TiDB Cloud Filesystem](/tidb-cloud-filesystem/filesystem-mount.md) to work with Filesystem data through a local directory and existing local tools. +- [Manage TiDB Cloud Filesystem Layers and Checkpoints](/tidb-cloud-filesystem/manage-filesystem-layers.md) to make and review isolated changes before applying them to the base Filesystem. +- [Share a TiDB Cloud Filesystem Across Machines](/tidb-cloud-filesystem/filesystem-sharing.md) to give another machine or environment access to the Filesystem. +- [TiDB Cloud Filesystem CLI Command Reference](/ai/ti/reference/ti-filesystem.md) for complete command syntax and options.