Read Tool & Reasoning Tokens from GenAI API instead of Catalog Metadata

[sayanshaw24]

Migrate tool/reasoning metadata from catalog to GenAI + optimize decode loop

Summary

Two goals:

1. Catalog v2 migration: Tool-call and reasoning token metadata (toolCallStart, toolCallEnd, reasoningStart, reasoningEnd) is being removed from the catalog. This PR sources it from GenAI's genai_config.json instead, via the new OgaModel::GetTagId() API, and writes the decoded strings into ModelInfo so all existing downstream consumers continue working.

2. Decode loop optimization: The current per-token detection of special tokens (double-decode + string.find()) is replaced with a single integer comparison against cached token IDs — the same pattern used for EOS detection.

Performance

Before (per token): 2 tokenizer stream decodes + 2 string.find() + string comparison
After (per token): 1 integer comparison (fast path) or unchanged (slow path fallback)

The fast path also eliminates the special tokenizer stream entirely — when tag IDs are available, only the normal stream is maintained.

Storage (follows EOS pattern)

Data	Storage	Consumer
eos_token_ids_	GenAIModelInstance	Decode() — filters EOS tokens
tag_info_ (IDs + strings)	GenAIModelInstance	Decode() — fast-path detection
String copies	ModelInfo.string_properties	BuildToolCallContext() → ToolCallStreamAccumulator

Changes

GenAIModelInstance (genai_model_instance.h / .cc)

• New TagInfo struct: caches both IDs (for the decode loop) and decoded strings (for text pipeline)
• GetTagInfo() — lazy-cached via std::call_once (same pattern as GetEosTokenIds())
  • Calls OgaModel::GetTagId() for each tag (4 calls total, one-time)
  • Decodes each valid ID through tokenizer_with_special_->Decode(&id, 1) to get the string (4 decodes, one-time)

OnnxChatGenerator::Decode() (onnx_chat_generator.cc)

• Fast path (tag IDs available): integer compare → return cached string, skip special stream
• Slow path (no tag IDs, backward compat): existing double-decode + string.find() behavior preserved

Model::Load() (model.cc)

• After loading the GenAI model, enriches ModelInfo string properties from GetTagInfo() cached strings
• Only writes if the property isn't already set (catalog metadata takes precedence)
• Infers supports_tool_calling / supports_reasoning from tag ID presence (>= 0)

No changes to ToolCallContext, ToolCallStreamAccumulator, or BuildToolCallContext()

• They continue consuming strings from ModelInfo as before — single source of truth

Dependency

Requires updated onnxruntime-genai nuget with OgaModelGetTagId API.

• GenAI PR: Add tool calling and reasoning tag token IDs to genai_config.json
• deps_versions.json bump needed once GenAI nuget publishes

How it works

Before (per token, every single generated token):

Decode Loop:
  token_id = generator.GetNextTokens()[0]

  // Decode through BOTH streams (expensive)
  token_text    = stream_->Decode(token_id)              // normal stream decode
  special_text  = stream_with_special_->Decode(token_id) // special stream decode

  // String operations to detect markers
  if (special_text != token_text):
    is_tool  = special_text.find("tool_call")            // string search
    is_think = special_text.find("think")                // string search
    if (is_tool || is_think): return special_text
  return token_text

Cost per token: 2 tokenizer stream decodes + 2 string.find() + string comparison — on every token, even though markers appear maybe 2-4 times per generation.

After (per token):

Model Load (one-time, 4 GetTagId calls + 4 Decode calls):
  GenAI:  GetTagId("tool_call_start") → 151657   (config ID or fallback vocab lookup)
  FL:     Decode(151657) → "<tool_call>"           (one tokenizer decode, cached in TagInfo)
  FL:     ModelInfo["tool_call_start"] = "<tool_call>"

Decode Loop (fast path):
  token_id = generator.GetNextTokens()[0]

  // Integer comparison (virtually free)
  if (token_id == 151657) { stream_->Decode(token_id); return "<tool_call>"; }
  if (token_id == 151658) { stream_->Decode(token_id); return "</tool_call>"; }
  // ... reasoning tags ...

  // Normal token — single decode, no special stream needed
  return stream_->Decode(token_id)

Cost per token: 1 integer comparison + 1 normal decode. No special stream. No string search.

Estimated performance gain

For a typical 500-token generation:

• Before: 500 × (2 decodes + 2 string searches) = 1000 decode calls + 1000 string ops
• After: 500 × (1 decode + 1 int compare) = 500 decode calls + 500 trivial comparisons

~50% reduction in tokenizer decode calls and complete elimination of string search operations in the hot loop. The special tokenizer stream is never even created/maintained for models with tag IDs configured.

Tag names supported by the GenAI API

• "tool_call_start", "tool_call_end" — tool call delimiters
• "reasoning_start", "reasoning_end" — chain-of-thought reasoning delimiters

Testing

Will be validated end-to-end once the GenAI nuget is available. The slow path (no tag IDs) is unchanged, so existing behavior is preserved for models without the new config fields.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Jul 1, 2026 12:28am