[ORT] onnxruntime_perf_test: Add multi-shape profiling support to perftest tool (--data_shape) by ankitm3k · Pull Request #29571 · microsoft/onnxruntime

ankitm3k · 2026-07-06T06:19:34Z

Description

This PR adds a --data_shape flag to onnxruntime_perf_test app that enables profiling multiple input shapes within a single session. The model is compiled once and then run with each specified shape group in round-robin order, providing per-shape latency statistics (Avg, Min, Max, P50, P90, P95, P99). This is useful for evaluating dynamic-shape model performance across different input dimensions without needing to recompile or restart the session for each shape.

Usage:

#With -I (generate random default inputs of specified shapes):
onnxruntime_perf_test.exe -v -e cpu -m times -r 10 -I --data_shape "data:[1,3,60,10][1,3,60,20]" "model.onnx"

#Without -I (select matching test data folders):
onnxruntime_perf_test.exe -v -e cpu -m times -r 10 --data_shape "data:[1,3,60,10][1,3,60,20]" "model.onnx"

Motivation and Context

Addresses the feature request: #28628

…ta_shape) (#1132) Usage: .\onnxruntime_perf_test.exe -v -e openvino -m times -r 10 -I --data_shape "data:[1,3,60,40][1,3,60,60]" -i "device_type|NPU" "PSO2_ctx.onnx" or, using testdata- .\onnxruntime_perf_test.exe -v -e openvino -m times -r 10 --data_shape "data:[1,3,60,40][1,3,60,60]" -i "device_type|NPU" "PSO2_ctx.onnx" Addresses the feature request: microsoft#28628

Copilot

Pull request overview

This PR extends the onnxruntime_perf_test tool to support multi-shape profiling via a new --data_shape flag, enabling a single session to run multiple input shape groups (round-robin) and report per-shape latency statistics.

Changes:

Added --data_shape CLI flag parsing and storage in RunConfig.
Implemented multi-shape input generation/selection and round-robin execution, with per-shape latency tracking and reporting.
Added a small ONNX model generator script for exercising multi-shape behavior.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
onnxruntime/test/testdata/dynamic_shape_add.py	Adds a small dynamic-shape ONNX model generator for testing `--data_shape`.
onnxruntime/test/perftest/test_session.h	Extends `RunTiming` to carry the shape group index used by an iteration.
onnxruntime/test/perftest/test_configuration.h	Adds `RunConfig::data_shape_groups` to hold per-input shape groups.
onnxruntime/test/perftest/strings_helper.h	Declares `ParseDataShapeGroups` for parsing the `--data_shape` argument.
onnxruntime/test/perftest/strings_helper.cc	Implements `ParseDataShapeGroups` string parsing + validation.
onnxruntime/test/perftest/performance_runner.h	Adds per-shape timing storage to `PerformanceResult` and a printing helper.
onnxruntime/test/perftest/performance_runner.cc	Implements warmup per shape group, per-shape stats reporting, and multi-shape init paths.
onnxruntime/test/perftest/ort_test_session.h	Adds round-robin controls and helper APIs for multi-shape test data selection.
onnxruntime/test/perftest/ort_test_session.cc	Implements round-robin selection and multi-shape input generation; tags timing with shape index.
onnxruntime/test/perftest/command_args_parser.cc	Adds the `--data_shape` flag and wires parsing into `RunConfig`.

n1harika · 2026-07-07T04:08:34Z

 RunTiming OnnxRuntimeTestSession::Run() {
-  // Randomly pick one OrtValueArray from test_inputs_. (NOT ThreadSafe)
-  const std::uniform_int_distribution<int>::param_type p(0, static_cast<int>(test_inputs_.size() - 1));
-  const size_t id = static_cast<size_t>(dist_(rand_engine_, p));
+  // Select input set: round-robin for multi-shape mode, random otherwise.
+  size_t id;
+  if (use_round_robin_ && test_inputs_.size() > 1) {
+    id = shape_group_counter_.fetch_add(1, std::memory_order_relaxed) % test_inputs_.size();
+  } else {
+    const std::uniform_int_distribution<int>::param_type p(0, static_cast<int>(test_inputs_.size() - 1));
+    id = static_cast<size_t>(dist_(rand_engine_, p));
+  }


This is pre-existing behavior, the round-robin method introduced by this PR is thread safe.

can we keep the old warning comment about the original random selection not being thread-safe?

+std::vector<int64_t> OnnxRuntimeTestSession::GetLoadedInputShape(size_t test_data_id, size_t input_id) const {
+  return test_inputs_.at(test_data_id).at(input_id).GetTensorTypeAndShapeInfo().GetShape();
+}


+graph_proto = helper.make_graph(
+    nodes=[
+        helper.make_node(
+            "Add",
+            inputs=["A", "B"],
+            outputs=["output"],
+            name="add_0",
+        ),
+    ],
+    name="dynamic_shape_add",
+    inputs=[
+        helper.make_tensor_value_info("A", TensorProto.FLOAT, ["batch", "seq", 4]),
+        helper.make_tensor_value_info("B", TensorProto.FLOAT, ["batch", "seq", 4]),
+    ],
+    outputs=[
+        helper.make_tensor_value_info("output", TensorProto.FLOAT, ["batch", "seq", 4]),
+    ],
+)
+
+model = helper.make_model(graph_proto, opset_imports=[helper.make_operatorsetid("", 18)])
+checker.check_model(model, True)
+save(model, "dynamic_shape_add.onnx")
+print("Saved dynamic_shape_add.onnx")
+print(
+    f"  Inputs: {[(i.name, [d.dim_param or d.dim_value for d in i.type.tensor_type.shape.dim]) for i in model.graph.input]}"
+)
+print(
+    f"  Outputs: {[(o.name, [d.dim_param or d.dim_value for d in o.type.tensor_type.shape.dim]) for o in model.graph.output]}"
+)


edgchen1 · 2026-07-06T22:47:32Z

  std::cout << "\nSession creation time cost: " << session_create_duration.count() << " s\n";
 }

+void PerformanceRunner::PrintPerShapeStats() const {


there seems to be some overlap between what this function prints and what was added to PerformanceResult::DumpToFile(). both will go to stdout. can we avoid the redundant output?

Removed the duplicated output while dumping to file.

thanks for removing the duplicate output. in the interest of reducing code duplication, could we keep the print in PerformanceResult::DumpToFile() which was reusing the lambda instead of adding the separate PerformanceRunner::PrintPerShapeStats() for printing to stdout?

Removed the separate PrintPerShapeStats() function entirely but modified DumpToFile() by upgrading its lambda to use its better format (input labels via FormatShapeGroup, ms units, averages).

edgchen1 · 2026-07-06T22:48:43Z

  }

  bool PopulateGeneratedInputTestData(int32_t seed);
+  bool PopulateMultiShapeInputTestData(


it would be good to also include "generated" in this function's name, since it is populating generated input.

also, there seems to be substantial code duplication between this function and PopulateGeneratedInputTestData(). could they be consolidated or refactored to share more code?

edgchen1 · 2026-07-06T22:57:26Z

    timing.submit_timing = std::chrono::high_resolution_clock::now() - start;
    timing.total_timing = timing.submit_timing;
  }
+  timing.shape_group_index = id;


the name shape_group_index is a bit misleading if it is also being set when there are no shape groups specified. maybe test_input_index would be a more accurate name.

edgchen1 · 2026-07-06T22:58:18Z

+
+        ORT_TRY {
+          size_t chars_parsed = 0;
+          int64_t dim_val = std::stoll(dim_token, &chars_parsed);


nit: can we use std::from_chars? it is not locale-dependent and it doesn't throw.

edgchen1 · 2026-07-06T23:07:39Z

@@ -0,0 +1,44 @@
+"""Generate a simple ONNX model with two dynamic-dimension inputs for testing --data_shape.


is this used?

I think it would actually be good to have some tests for onnxruntime_perf_test. but if this script is not used by any existing tests, perhaps we don't need to add it yet.

Yes- they are for manual verification. Removed them from the current PR.

edgchen1 · 2026-07-06T23:54:04Z

+        }
+      }
+      if (!found) {
+        std::cerr << "No test data found matching shape [";


This error only reports the first input (data_shape_groups.begin()), but the match fails if any specified input's shape doesn't match. When multiple inputs are given and the mismatch is on a later one, this message points at the wrong input and can be confusing to debug. Could we print the whole shape group instead, e.g.:

No test data found matching shape group 2: input:[1,3,448,448], mask:[1,448].

If we add the FormatShapeGroup helper mentioned in the other comment, this becomes a one-liner.

🤖 Written by GitHub Copilot on behalf of @edgchen1.

Done - the error now prints all inputs in the shape group that failed. Also, added a FormatShapeGroup helper to avoid duplicating this formatting across the codebase.

edgchen1 · 2026-07-06T23:54:04Z

+          return false;
+        }
+      } else {
+        input_node_dim = tensor_info.GetShape();


For inputs that the user didn't specify in --data_shape, we silently fall back to the model's declared shape and coerce any dynamic dim (-1) to 1. If such an input has a dynamic dim that's meant to track a specified input (e.g. a shared batch/sequence dim), this can produce a shape mismatch at inference or misleading perf numbers, with no diagnostic. Could we at least emit a warning when a model input is not covered by --data_shape, so the user knows its shape was inferred rather than specified?

🤖 Written by GitHub Copilot on behalf of @edgchen1.

edgchen1 · 2026-07-06T23:54:04Z

+  std::cout << "\nLatency per shape group:" << std::endl;
+
+  for (size_t g = 0; g < num_groups; g++) {
+    // Build label: "input_name : [d0,d1,...]"


This name:[dims] formatting loop is duplicated in three places: here, the per-shape block in DumpToFile, and the error message in Initialize (the "No test data found" path). Could we factor it into a single helper, e.g. std::string FormatShapeGroup(const std::map<...>& groups, size_t g)? That would deduplicate the formatting and make the error-message fix straightforward. This is related to the consolidation already requested for PrintPerShapeStats/DumpToFile.

🤖 Written by GitHub Copilot on behalf of @edgchen1.

…larity

n1harika · 2026-07-07T04:52:29Z

Thanks @edgchen1 for the review! Comments are addressed.

edgchen1 · 2026-07-07T23:49:34Z

  std::cout << "\nSession creation time cost: " << session_create_duration.count() << " s\n";
 }

+void PerformanceRunner::PrintPerShapeStats() const {


thanks for removing the duplicate output. in the interest of reducing code duplication, could we keep the print in PerformanceResult::DumpToFile() which was reusing the lambda instead of adding the separate PerformanceRunner::PrintPerShapeStats() for printing to stdout?

edgchen1 · 2026-07-07T23:50:33Z

 RunTiming OnnxRuntimeTestSession::Run() {
-  // Randomly pick one OrtValueArray from test_inputs_. (NOT ThreadSafe)
-  const std::uniform_int_distribution<int>::param_type p(0, static_cast<int>(test_inputs_.size() - 1));
-  const size_t id = static_cast<size_t>(dist_(rand_engine_, p));
+  // Select input set: round-robin for multi-shape mode, random otherwise.
+  size_t id;
+  if (use_round_robin_ && test_inputs_.size() > 1) {
+    id = shape_group_counter_.fetch_add(1, std::memory_order_relaxed) % test_inputs_.size();
+  } else {
+    const std::uniform_int_distribution<int>::param_type p(0, static_cast<int>(test_inputs_.size() - 1));
+    id = static_cast<size_t>(dist_(rand_engine_, p));
+  }


can we keep the old warning comment about the original random selection not being thread-safe?

edgchen1 · 2026-07-07T23:55:43Z

+  for (const auto& [name, shapes] : groups) {
+    if (!result.empty()) result += ", ";
+    result += name + ":[";
+    const auto& dims = shapes[g];


nit: can we ensure that g is in range?

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

 // Copyright (c) Microsoft Corporation. All rights reserved.
 // Copyright (c) 2023 NVIDIA Corporation.
 // SPDX-FileCopyrightText: Copyright 2024 Arm Limited and/or its affiliates <open-source-office@arm.com>
 // Licensed under the MIT License.
+#include <cstdint>


+#include <charconv>
 #include <iostream>
 #include <sstream>


+  for (char c : input) {
+    if (c == '[') {
+      bracket_depth++;
+      current += c;
+    } else if (c == ']') {
+      bracket_depth--;
+      current += c;
+    } else if ((c == ' ' || c == '\t') && bracket_depth == 0) {


 RunTiming OnnxRuntimeTestSession::Run() {
-  // Randomly pick one OrtValueArray from test_inputs_. (NOT ThreadSafe)
-  const std::uniform_int_distribution<int>::param_type p(0, static_cast<int>(test_inputs_.size() - 1));
-  const size_t id = static_cast<size_t>(dist_(rand_engine_, p));
+  // Select input set: round-robin for multi-shape mode, random otherwise.
+  size_t id;


n1harika · 2026-07-08T07:21:51Z

+std::vector<int64_t> OnnxRuntimeTestSession::GetLoadedInputShape(size_t test_data_id, size_t input_id) const {
+  const auto& v = test_inputs_.at(test_data_id).at(input_id);
+  if (!v.IsTensor()) {
+    ORT_THROW("--data_shape only supports tensor inputs; input_id=", input_id, " in test_data_id=", test_data_id,
+              " is not a tensor.");
+  }
+  return v.GetTensorTypeAndShapeInfo().GetShape();


We have checks for this in init-time. We will not know test data contents at the time of parsing.

…add apt headers

edgchen1 requested a review from Copilot July 6, 2026 18:42

Copilot started reviewing on behalf of edgchen1 July 6, 2026 18:42 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

edgchen1 reviewed Jul 6, 2026

View reviewed changes

Address review feedback, remove manual test files, add warnings for c…

33172bc

…larity

edgchen1 reviewed Jul 7, 2026

View reviewed changes

edgchen1 requested a review from Copilot July 7, 2026 23:56

Copilot started reviewing on behalf of edgchen1 July 7, 2026 23:56 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Address review feedback 2, adjust output formatting per shape group, …

7507ed2

…add apt headers

		@@ -0,0 +1,44 @@
		"""Generate a simple ONNX model with two dynamic-dimension inputs for testing --data_shape.

Uh oh!

Conversation

ankitm3k commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n1harika commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ankitm3k commented Jul 6, 2026 •

edited

Loading