Skip to content

Guard BFCL cohesive namespace compression - #43

Closed
SonAIengine wants to merge 1 commit into
codex/bfcl-cohesive-namespace-ablationfrom
codex/bfcl-cohesive-namespace-guard
Closed

Guard BFCL cohesive namespace compression#43
SonAIengine wants to merge 1 commit into
codex/bfcl-cohesive-namespace-ablationfrom
codex/bfcl-cohesive-namespace-guard

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • guard opt-in BFCL cohesive namespace compression so it only keeps namespace families that are first-ranked or explicitly signaled by the query
  • preserve singleton dotted namespaces when the query names that namespace, including simple plural forms like restaurants -> restaurant
  • document the 100-case qwen3.6-27B middle sweep result and remaining 0.27 gate gaps

Model Evidence

Command:

poetry run python -m benchmarks.bfcl_tool_selection.sweep \
  --categories simple_python,multiple,parallel,parallel_multiple \
  --limit 25 \
  --top-ks 5 \
  --tool-sources row,retrieved \
  --model qwen3.6-27b \
  --llm-url http://127.0.0.1:18000/v1 \
  --disable-thinking \
  --candidate-selection-guidance \
  --cohesive-namespace-candidates \
  --cache-dir /tmp/gtc-bfcl-qwen027-cohesive-guard-limit25-cache \
  --concurrency 6 \
  --progress \
  --progress-every 5 \
  --output /tmp/gtc-bfcl-qwen027-cohesive-guard-limit25.json

Result:

  • row source: exact 0.94, strict 0.94
  • retrieved source: exact 0.83, retrieval@5 0.99, strict 0.83
  • milestone xgen-0.27: fail
  • row preservation: 0.883
  • parallel_multiple exact: 0.84
  • failure categories: candidate_ambiguity:8, argument_value_mismatch:6, call_count_mismatch:2, retrieval_miss:1

Compared with the previous 100-case cohesive run, this removes candidate_not_present failures (2 -> 0) and lifts parallel_multiple (0.76 -> 0.84), but aggregate exact is still below the 0.27 gate (0.83 < 0.85).

Validation

  • poetry run pytest tests/test_bfcl_tool_selection_llm_loop.py -q
  • poetry run ruff check benchmarks/bfcl_tool_selection/llm_loop.py tests/test_bfcl_tool_selection_llm_loop.py
  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q (954 passed, 3 skipped)

@SonAIengine

Copy link
Copy Markdown
Owner Author

Closing this internal stacked PR as superseded by the current main branch. The branch is preserved for history.

@SonAIengine SonAIengine closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant