Skip to content

Add BFCL candidate selection guidance ablation - #41

Open
SonAIengine wants to merge 1 commit into
codex/bfcl-rank-hint-ablationfrom
codex/bfcl-selection-guidance-ablation
Open

Add BFCL candidate selection guidance ablation#41
SonAIengine wants to merge 1 commit into
codex/bfcl-rank-hint-ablationfrom
codex/bfcl-selection-guidance-ablation

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • add --candidate-selection-guidance to BFCL model loop and sweep runners
  • record the option in report/cache keys and expose it in sweep console output
  • document the qwen3.6-27B smoke result: guidance fixes the call-count mismatch class, but not parallel_multiple sibling ambiguity

qwen3.6-27B Smoke Evidence

Via go165 vLLM qwen3.6-27b, category limit 5, row/retrieved k=5.

Artifact: /tmp/gtc-bfcl-qwen027-guidance-smoke.json

  • row exact: 1.00
  • retrieved exact: 0.90 (0.85 -> 0.90 vs baseline smoke)
  • retrieval recall: 1.00
  • failed gates: row preservation 0.90 < 0.94, parallel_multiple 0.60 < 0.75
  • remaining failures: parallel_multiple_2, parallel_multiple_4 candidate ambiguity

Hard-case bundle: /tmp/gtc-bfcl-qwen027-guidance-hardcases

  • cases: 2
  • failure categories: candidate_ambiguity:2

Interpretation: prompt selection policy can remove call-count over-decomposition, but the remaining 0.27 blocker needs candidate equivalence/grouping rather than more rank wording.

Validation

  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q -> 951 passed, 3 skipped

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant