Skip to content

Tag BFCL argument mismatch causes - #51

Closed
SonAIengine wants to merge 1 commit into
codex/bfcl-nested-argument-matcherfrom
codex/bfcl-argument-failure-tags
Closed

Tag BFCL argument mismatch causes#51
SonAIengine wants to merge 1 commit into
codex/bfcl-nested-argument-matcherfrom
codex/bfcl-argument-failure-tags

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • add stable failure_tags for BFCL argument_value_mismatch cases
  • classify unexpected arguments, optional value mismatches, structured missing values, percentage scale errors, and data-reference substitutions
  • document the argument-tag subset loop for faster next-step experiments

Tags

  • unexpected_argument
  • optional_value_mismatch
  • structured_value_missing
  • percentage_scale_mismatch
  • data_reference_substitution

Validation

  • poetry run pytest tests/test_bfcl_tool_selection_llm_loop.py -q
  • poetry run pytest tests/test_bfcl_tool_selection_llm_loop.py tests/test_bfcl_tool_selection_hard_cases.py tests/test_bfcl_tool_selection_sweep.py -q
  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q
  • Qwen3.6-27B argument subset smoke:
    • command: poetry run python -m benchmarks.bfcl_tool_selection.llm_loop --case-ids-file /tmp/gtc-bfcl-qwen027-tag-subsets/failure_argument_value_mismatch.txt --top-k 5 --tool-source retrieved --model qwen3.6-27b --llm-url http://127.0.0.1:18000/v1 --disable-thinking --candidate-selection-guidance --cohesive-namespace-candidates --cache-dir /tmp/gtc-bfcl-argument-tags-cache --cache-namespace argument-tags --timeout 180 --progress --progress-every 1 --output /tmp/gtc-bfcl-argument-tags-subset.json --json
    • cases: 6
    • evaluator_exact_match: 0.166667
    • failure_tag_breakdown: data_reference_substitution=1, optional_value_mismatch=1, percentage_scale_mismatch=1, structured_value_missing=1, unexpected_argument=1
  • Hard-case bundle produced tag files under /tmp/gtc-bfcl-qwen027-argument-tags-hardcases/ for each tag.

@SonAIengine

Copy link
Copy Markdown
Owner Author

Closing this internal stacked PR as superseded by the current main branch. The branch is preserved for history.

@SonAIengine SonAIengine closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant