Skip to content

grill-perf: allow thinking/reasoning control in portable-chat-v1 (chat_template_kwargs) #11

Description

@plotarmordev

Problem

portable-chat-v1 cannot send chat_template_kwargs, so on GLM-5.3-Flash (thinking on by default) a 64-token capped structured prompt produced only reasoning tokens: no answer-text timestamp in any of 18 requests, and the A/A receipt in docs/performance/2026-09-09-recipe-smoke.md is effectively a prose measurement. A hand probe with max_tokens: 1 and thinking on produced no content delta at all and failed.

Evidence

  • docs/performance/2026-09-09-recipe-smoke.json: first_answer_text_us null on all measured lanes.
  • Live probe fix that made the run work: adding "chat_template_kwargs": {"enable_thinking": false} to the request body.

Proposal

Workload-level request.thinking: "on" | "off" | null (null = don't send), serialized as chat_template_kwargs.enable_thinking for vLLM-style servers, recorded in plan.json. Keep portable-chat-v1 strict otherwise. Optional generic request.extra_body (closed map of scalars) recorded in the plan would cover reasoning_effort too.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions