Skip to content

[metrics] Add KV snapshot upload throughput metrics #4347

Description

@platinumhamburg

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Fluss exposes remoteLogCopyBytesPerSecond to measure remote log upload throughput, but there is no equivalent metric for KV snapshot uploads.

KV snapshots upload newly created SST files and other snapshot files to remote storage. Without upload throughput metrics, users cannot observe the remote write traffic generated by KV snapshots or assess its contribution to the overall remote storage write workload.

Solution

Expose the following metrics in the existing table-level KV metric group:

Metric Type Description
remoteKvCopyBytes Counter Cumulative bytes of successfully uploaded KV snapshot files.
remoteKvCopyBytesPerSecond Meter KV snapshot upload throughput in bytes per second, derived from the same counter.

Record the uploaded file size in KvSnapshotDataUploader after each file upload completes successfully.

The metrics should follow these semantics:

  • Include newly uploaded SST files and other snapshot files handled by the uploader.
  • Exclude SST files reused from previous snapshots without uploading.
  • Count successful file uploads even if another file upload or the subsequent snapshot commit fails.
  • Count successful reuploads again, since they generate additional write traffic.
  • Support concurrent file uploads with a thread-safe counter.

Anything else?

No response

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions