Routing hygiene for agent harnesses, structured-output fidelity, OpenRouter hardening, shortfall semantic signals

[veerareddyvishal144]

Summary

Found while benchmarking Lynkr tier routing against openrouter/auto on Terminal-Bench-core 0.1.1 (Terminus agent via LiteLLM, 80 tasks, ~1,200 calls per run). Every change is gated or scoped so Claude Code, Cursor and OpenCode keep their current behaviour unless a flag is set. Final numbers with these changes: 39/80 for $1.28 vs auto 44/80 for $4.72 on the same VM and evening.

Request / response fidelity

• Forward the client's output_format / response_format to every provider (json_schema where the host supports it, json_object otherwise). Lynkr dropped both, so structured-output clients never got schema enforcement.
• Never run the XML tool-call extractor on a JSON-object reply. It mangled command text containing <...> and flipped stop_reason to tool_use.
• Structured-output requests bypass the response cache. Harnesses retry a failed parse with the identical conversation; a cache hit returned the same bad reply every time.
• Remove server-side STANDARD_TOOLS injection from all provider invokers. A tool-less request stays tool-less. Injecting 12 Claude Code tool schemas turned chat / structured clients into tool-calling agents (+~2.5k tokens per call, write access to the host workspace). The OpenAI-compat router's IDE_SAFE_TOOLS injection is left untouched (tied to CLIENT_TOOL_MAPPINGS for Codex) — follow-up.
• FMT_GUARD_ENABLED=false opt-out for the markdown format guard (its "use fenced code blocks" instruction contradicts raw-JSON clients).

Routing on the ask, not the wrapper

• harness-envelope: recognise instruction-schema harness prompts (fixed preamble + JSON schema + Instruction: block) and expose the instruction as the ask for every turn. Complexity analyser, intent scorer, agentic detector, kNN query and window scorer use it. Later user turns are terminal output and previously drove force / risk / text scoring (62 mid-session COMPLEX→MEDIUM demotions in one run).
• client-profiles: terminus profile (tool-less by design) with prompt-pattern detection; tool-less profiles excluded from side-request detection.
• agentic-detector: drop solve from the autonomous regex — "solving command-line tasks" boilerplate pinned every request to REASONING.
• FORCE_TIER_PATTERNS=false, RISK_TIER_ESCALATION=false opt-outs for the keyword escalators.
• kNN: LYNKR_KNN_ENABLED=false skips the query and its embedding call; picks are validated against configured TIER_* models (the index remembered a decommissioned model and sent 81 calls to it).
• embeddings: cap input at 5000 chars (nomic-embed-text's 2048-token context), surface Ollama's error body, retry once with halved text on input-too-long instead of degrading the provider for 60s.

Provider plumbing and verification

• [ModelCheck] warning when the served model differs from the tier's requested model (live incident: TIER_*=zai:glm-5.3-flash silently served by ZAI_MODEL=glm-5.2 for two full runs).
• Z.AI: unknown tier model ids pass through instead of falling back to the default model.
• OpenRouter: per-model provider pin, per-model reasoning effort, strict json_schema for hosts that honour it, usage.include for provider-reported cached / reasoning tokens and cost, per-model upstream timeout with failover down the pinned list before tier-fallback climbs, per-model output cap, [ProviderCheck] warning when an unpinned host served. Replies converted in the invoker with cache_read mapped from prompt_tokens_details.cached_tokens.
• Fireworks: per-model reasoning effort map, keep thinking enabled for models that reject thinking:disabled, response_format forwarding.
• OpenAI-compatible: reasoning_effort knob and response_format forwarding.
• Baidu Qianfan: clamp max_tokens to 12288 (hard provider limit).

Shortfall: semantic requirement + Jev telemetry (second commit)

• Shortfall's requirement vector came from the structural complexity dimensions only and was nearly constant on task-description prompts (0.11–0.34 across 77 Terminal-Bench tasks), so cheapest-covering always picked the cheapest model. shortfall.liftRequirement now lifts each head with the anchor intent score (interpolated between tier profiles) and the Jev tier probabilities (probability-weighted profile); never lowers. Replay: requirement median 0.15 → 0.61, agreement with the anchor+Jev pick 51/61.
• Jev verdicts were null on 100% of served telemetry rows while the judge re-tiered ~36% of fresh routes. Two gaps: the window wrapper stores the verdict as _jev (reader only knew jev/analysis.jev) and the forced-provider hop never carried it. jevFields reads _jev, api/router.js stamps req.body._jev, the reconstituted routingResult sets jev. Verified live: verdict, confidence, probabilities and judge model now land on every row.
• New test/shortfall-semantic.test.js.

Test plan

• npm run test:unit: 1562 pass / 6 fail on this branch and on main (the 6 are pre-existing, environment-dependent: Jev LRU, TencentDB launcher, downgrade gate, task-ledger).
• Live: Terminus harness end-to-end through Lynkr on OpenRouter (Friendli glm-5.3-flash cheap tier, DeepSeek official strong tier): two full 80-task runs, 0 wrong-model warnings, 0 harness parse failures, failover drill (forced timeout → next host → tier climb) verified.
• Requests that carry their own tools still forward them (probed: tool_use block returned).

New flags are documented in .env.example. Happy to split this into smaller PRs (tool-injection removal / structured output / harness awareness / OpenRouter hardening) if preferred.

🤖 Generated with Claude Code

[github-actions]

🔍 OpenCodeReview found 23 issue(s) in this PR.

• ✅ Successfully posted inline: 23 comment(s)

[github-actions]

The backup file write is not validated before overwriting the config. If copyFileSync fails (e.g., permission issues), the config could be partially overwritten or lost. Add a check after copyFileSync to ensure the backup exists before proceeding with the rewrite.

[github-actions]

The _INPUT_LENGTH_RE regex at line 184 is overly broad and could match false positives. The pattern context length|input length|too long|exceeds? (the )?(maximum|context)|maximum context would incorrectly match valid messages containing these words. For example, "too long" appears in many contexts unrelated to errors. Consider making the regex more specific by anchoring to error contexts (e.g., case sensitivity, surrounding keywords like "error", "rejected", or requiring specific error message patterns from known providers).

Suggestion:

Suggested change

	const _INPUT_LENGTH_RE = /context length|input length|too long|exceeds? (the )?(maximum|context)|maximum context/i;
	const _INPUT_LENGTH_RE = /\b(context|input)\s+length\s+\b(exceeds?|too long)\b|\bmaximum\s+(context|length)\b|\binput\s+too\s+long\b/i;

[github-actions]

At line 2266, body._tierModel is accessed without null checking. When body._tierModel is falsy, the expression (body._tierModel ? requestedModel : null) correctly returns null, but if _tierModel is an empty string or 0, it will use requestedModel (the original value). This could lead to unexpected behavior. Better to explicitly check typeof body._tierModel === 'string' && body._tierModel to ensure it's a valid non-empty string.

[github-actions]

In src/orchestrator/index.js, the JSON detection check startsWith("{") && endsWith("}") is fragile and can be bypassed by strings containing JSON anywhere within them. For example, a string like "Here is the JSON: {"key": "value"}" would incorrectly skip extraction. A more robust approach would be to try parsing with JSON.parse() in a try-catch block, or check for a more specific pattern.

Suggestion:

Suggested change

	const _bt = block.text.trim();
	if (_bt.startsWith("{") && _bt.endsWith("}")) continue;
	const _bt = block.text.trim();
	try {
	JSON.parse(_bt); continue; // Valid JSON object
	} catch {}
	// Only try to parse first/last char if trim didn't change the string
	if (_bt === block.text && _bt.startsWith("{") && _bt.endsWith("}")) continue;

[github-actions]

In src/orchestrator/index.js, there's duplicated logic for JSON detection on two separate code paths (lines 2318-2319 and 2346-2347). This could be extracted into a helper function like isJsonReply(content) to avoid duplication.

Suggestion:

Suggested change

	if (toolCalls.length === 0 && typeof message.content === "string" && message.content.trim()
	&& !(message.content.trim().startsWith("{") && message.content.trim().endsWith("}"))) {
	const isJsonReply = (text) => {
	const trimmed = text.trim();
	if (trimmed === text && trimmed.startsWith("{") && trimmed.endsWith("}")) return true;
	try { JSON.parse(trimmed); return true; } catch {}
	return false;
	};
	// Then use: !isJsonReply(message.content)

[github-actions]

In src/routing/complexity-analyzer.js, both shouldForceCloud and shouldForceReasoning extract content using extractContent(payload) which could return '' (empty string) if payload has no messages. The regex patterns might match unexpected empty content. Consider adding an explicit check for empty content before pattern matching.

Suggestion:

Suggested change

	if (process.env.FORCE_TIER_PATTERNS === "false") return false;
	const content = extractContent(payload);
	if (process.env.FORCE_TIER_PATTERNS === "false") return false;
	const content = extractContent(payload);
	if (!content || !content.trim()) return false;

[github-actions]

In src/routing/index.js, the check if (knnResult && process.env.LYNKR_KNN_ENABLED === 'false') is redundant when the outer code already checks config.routing?.knnEnabled !== false && process.env.LYNKR_KNN_ENABLED !== 'false'. The inner check will never be true unless LYNKR_KNN_ENABLED changes between checks (which shouldn't happen). This nested check is defensive but unnecessary.

Suggestion:

Suggested change

	if (knnResult && process.env.LYNKR_KNN_ENABLED === 'false') {
	knnResult = null;
	}
	// This check is redundant given the outer condition already checks LYNKR_KNN_ENABLED
	// Consider removing or adding a comment explaining why it's needed

[github-actions]

In src/routing/index.js, the kNN tier validation code has duplicated logic in two try-catch blocks (one for success, one for error handling). The same model set construction code appears twice. This could be extracted into a helper function to avoid duplication.

Suggestion:

Suggested change

	const _sel = getModelTierSelector();
	const _allowed = new Set();
	for (const _t of TIER_ORDER) for (const _m of (_sel.getModelsForTier(_t) || [])) _allowed.add(`${_m.provider}:${_m.model}`);
	const getModelSetFromTiers = () => {
	const _sel = getModelTierSelector();
	const _allowed = new Set();
	for (const _t of TIER_ORDER) for (const _m of (_sel.getModelsForTier(_t) || [])) _allowed.add(`${_m.provider}:${_m.model}`);
	return _allowed;
	};
	// Then use: const _allowed = getModelSetFromTiers();

[github-actions]

In src/routing/shortfall.js, the liftRequirement function uses default parameters (anchorScore = null, jev = null) which is good. However, it doesn't validate that the input req object actually has the expected properties (HEADS), which could cause issues if an unexpected object is passed.

Suggestion:

Suggested change

	function liftRequirement(req, { anchorScore = null, jev = null } = {}, tierProfiles = null) {
	function liftRequirement(req, { anchorScore = null, jev = null } = {}, tierProfiles = null) {
	if (!req || typeof req !== 'object') return { req: {}, lift: { anchor: null, jev: null, applied: [] } };

[github-actions]

The liftRequirement function uses Array.includes() to track which lifts were applied ('anchor', 'jev'). This is inefficient and could be simplified by just counting the number of successful lifts instead of maintaining an array.

Suggestion:

Suggested change

	for (const h of HEADS) {
	if (a && a[h] > out[h]) { out[h] = a[h]; if (!applied.includes('anchor')) applied.push('anchor'); }
	if (j && j[h] > out[h]) { out[h] = j[h]; if (!applied.includes('jev')) applied.push('jev'); }
	const appliedSet = new Set();
	for (const h of HEADS) {
	if (a && a[h] > out[h]) { out[h] = a[h]; appliedSet.add('anchor'); }
	if (j && j[h] > out[h]) { out[h] = j[h]; appliedSet.add('jev'); }
	out[h] = Math.round(Math.max(0, Math.min(1, out[h])) * 1000) / 1000;
	}
	const applied = [...appliedSet];

[github-actions]

package.json update: Added test/shortfall-semantic.test.js to test:unit. No dependency conflicts or duplicate declarations found. All tools referenced in scripts (eslint, pino-pretty) are declared in devDependencies.

[github-actions]

Unsanitized file paths: The script reads user-supplied paths (--model=run-dir, --out) and writes to them without validation.

Suggestion:

Suggested change

	const v = next(); const eq = v.indexOf('=');
	out.models.push(eq > 0 ? { spec: v.slice(0, eq), runDir: expand(v.slice(eq + 1)) } : { spec: v, runDir: null });
	// Validate runDir exists and is accessible before using
	if (m.runDir && !fs.existsSync(m.runDir)) die(`--model run-dir does not exist: ${m.runDir}`);

[github-actions]

Inconsistent error messages: --help prints usage before exit, but validation errors via die() don't. Users encountering errors may not know valid arguments.

Suggestion:

Suggested change

	else die(`unknown arg ${a}`);
	else { console.error(`unknown arg: ${a}`); console.error('Run with --help for usage'); die(''); }

[github-actions]

Hardcoded .env paths: The script assumes .env is in process.cwd() or os.homedir(). This may fail in containerized or different working directory contexts.

Suggestion:

Suggested change

	const srcEnv = fs.existsSync(path.join(process.cwd(), '.env')) ? path.join(process.cwd(), '.env') : path.join(os.homedir(), '.env');
	// Add --env-path flag or look for .env relative to ROOT, then cwd, then homedir
	const envPaths = [path.join(ROOT, '.env'), path.join(process.cwd(), '.env'), path.join(os.homedir(), '.env')];

[github-actions]

Repeated string pattern: fs.existsSync(path.join(process.cwd(), '.env')) ? path.join(process.cwd(), '.env') : path.join(os.homedir(), '.env') appears twice and should be extracted.

Suggestion:

Suggested change

	const srcEnv = fs.existsSync(path.join(process.cwd(), '.env')) ? path.join(process.cwd(), '.env') : path.join(os.homedir(), '.env');
	function findEnvPath() {
	const cwd = path.join(process.cwd(), '.env');
	if (fs.existsSync(cwd)) return cwd;
	return path.join(os.homedir(), '.env');
	}

[github-actions]

The terminus profile's prompt pattern matches the HARNESS_PREAMBLE_RES pattern in harness-envelope.js. This creates tight coupling between client detection and harness parsing logic. Consider unifying these patterns or moving to a shared configuration file.

[github-actions]

Same dynamic require pattern here - could cause circular dependency issues if the module loading order changes.

[github-actions]

The harness prompt detection pattern /You are an AI assistant tasked with solving command-line tasks/i is very broad and may match non-harness prompts from various providers. Consider making it more specific to your actual harness system.

Suggestion:

Suggested change

	const HARNESS_PREAMBLE_RES = [
	/You are an AI assistant tasked with solving command-line tasks/i,
	/"title":\s*"CommandBatchResponse"/,
	];
	const HARNESS_PREAMBLE_RES = [
	// Match more specific harness identifiers to avoid false positives
	/You are an AI assistant.*?command-line tasks.*?harness/i,
	/"title":\s*"CommandBatchResponse"/,
	];

[github-actions]

The kNN tier validation loop creates a Set and iterates all tiers every request. If TIER_ORDER has many tiers/models, this could add latency to every routed request.

[github-actions]

Same dynamic require pattern here - could cause circular dependency issues if the module loading order changes.