Skip to content

[Go] Add Go bindings for ONNX Runtime C API#29615

Open
dannyota wants to merge 3 commits into
microsoft:mainfrom
dannyota:dannyota/go-bindings
Open

[Go] Add Go bindings for ONNX Runtime C API#29615
dannyota wants to merge 3 commits into
microsoft:mainfrom
dannyota:dannyota/go-bindings

Conversation

@dannyota

@dannyota dannyota commented Jul 8, 2026

Copy link
Copy Markdown

Description

Official Go bindings for the ONNX Runtime C API via CGO, following
the pattern of existing language bindings (rust/, java/, csharp/).

Closes #9786

Motivation

Go is widely used for backend services that need ML inference
(embeddings, classification, NLP). There is strong community demand
(#9786, 19 comments) and no official Go support. This PR fills that gap.

What's included

  • Session management: create from file or byte buffer, introspect
    inputs/outputs with dynamic dimension support
  • Tensor I/O: generic CreateTensor[T] (Go 1.18+ generics),
    untyped NewTensorFromBytes, string tensors, zero-length dimensions
  • Session options: thread counts, graph optimization, execution mode,
    memory arena, profiling, free dimension overrides
  • Execution providers: CUDA (V2), TensorRT (V2), generic string
    key-value API for all other EPs
  • IO Binding: bind inputs/outputs to devices for GPU workflows
  • Model metadata: producer, graph name, domain, version, custom keys
  • Run options: log level, tag, config entries
  • Cancellation: context.Context support via RunOptionsSetTerminate
  • Concurrency: Session.Run safe for concurrent use (ORT guarantee)
  • Sequence/Map: value type inspection and sequence element access

Design

Wraps the C API vtable (OrtApi) through ~80 C shim functions in
cshim.h/cshim.c — CGO cannot call C function pointers directly.
Library loaded at runtime via dlopen (Linux/macOS) or LoadDLL
(Windows). Singleton OrtEnv per process via sync.Mutex.

Zero-copy tensor inputs via runtime.Pinner. Output tensors wrap
ORT-allocated memory with views valid until Close().

Module path: github.com/microsoft/onnxruntime/go
Package: onnxruntime (import as ort)
Go 1.26 minimum. stdlib + CGO only — no third-party dependencies.

Tests

46 tests passing with -race:

  • Lifecycle, inference correctness, dynamic shapes, zero-length dims
  • All tensor types (float32, int64, bool, string, bytes, scalars)
  • Error handling, type/shape mismatch, use-after-close
  • Concurrency: 16 goroutines × 50 runs on one session
  • Context cancellation, IO binding, model metadata, run options
  • Integration: Qwen3-Embedding-0.6B ONNX INT8 (59 inputs, 57 outputs)
  • Semgrep Pro: 0 findings (195 rules)
  • golangci-lint: 0 issues

Idiomatic Go wrapper for the ORT C API via CGO, matching the pattern
of existing language bindings (rust/, java/, csharp/).

Covers session management, typed/untyped tensor I/O, string tensors,
IO binding, model metadata, run options, session options with CUDA and
TensorRT execution providers, context cancellation, and concurrent
inference. Tested with real Qwen3-Embedding-0.6B ONNX INT8 model.
@dannyota

dannyota commented Jul 8, 2026

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Fix TOCTOU race in NewIOBinding (use-after-free on concurrent Close),
empty outputNames panic in Run/RunWithOptions, and metadata use-after-
close segfault. Add 15 tests covering all identified coverage gaps.
Remove dead code. Bump minimum ORT version to 1.27.0 for getter APIs.
@dannyota

dannyota commented Jul 8, 2026

Copy link
Copy Markdown
Author

Updated with a second commit:

  • Fixed TOCTOU race in NewIOBinding (use-after-free on concurrent Close)
  • Fixed empty outputNames panic in Run/RunWithOptions
  • Fixed metadata use-after-close segfault
  • Added 15 tests (63 total), covering all tensor types, error paths, IO binding, and edge cases
  • Bumped minimum ORT to 1.27.0 for GetMemPatternEnabled/GetSessionExecutionMode getters
  • Tested against ORT 1.28.0 built from rel-1.28.0 branch — all passing

Try API version 27 first, fall back to 17 for ORT 1.17-1.26.
GetExecutionMode and IsMemPatternEnabled return a clear error
on older libraries instead of crashing. Minimum ORT is now 1.17.
@dannyota

dannyota commented Jul 8, 2026

Copy link
Copy Markdown
Author

Added API version fallback: bindings now try API version 27 first, fall back to 17 for older ORT libraries. This means ORT 1.17+ is supported (previously required 1.27+). The two getter functions added in 1.27 (GetExecutionMode, IsMemPatternEnabled) return a clear error on older libraries instead of crashing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Do we have official Golang support for ONNXRuntime?

1 participant