Skip to content

Attribute BFCL row vs retrieved deltas - #44

Open
SonAIengine wants to merge 1 commit into
codex/bfcl-cohesive-namespace-guardfrom
codex/bfcl-near-duplicate-selection
Open

Attribute BFCL row vs retrieved deltas#44
SonAIengine wants to merge 1 commit into
codex/bfcl-cohesive-namespace-guardfrom
codex/bfcl-near-duplicate-selection

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • add summary.row_vs_retrieved_deltas to BFCL model sweep artifacts
  • pair row-source and retrieved-source runs by repeat/top-K/category/case ID
  • report row-pass/retrieved-fail counts, breakdowns, and case IDs so retrieval-layer losses are separated from row-source/model upper-bound failures
  • document the latest 100-case qwen3.6-27B attribution: row-pass retrieved losses are 11 cases, mostly candidate ambiguity

Why

The 0.27 gate is currently failing, but raw retrieved failures mix two different problems:

  • failures that also happen with row-source tool lists, which are model/tool-schema upper-bound issues
  • failures that only happen after graph retrieval/presentation, which are the actual graph-tool-call search/presentation losses

This change makes that split explicit in every sweep artifact so the next hard-case subset can focus on the cases graph-tool-call can actually improve.

Evidence

Cached rerun of the 100-case qwen3.6-27B middle sweep:

poetry run python -m benchmarks.bfcl_tool_selection.sweep \
  --categories simple_python,multiple,parallel,parallel_multiple \
  --limit 25 \
  --top-ks 5 \
  --tool-sources row,retrieved \
  --model qwen3.6-27b \
  --llm-url http://127.0.0.1:18000/v1 \
  --disable-thinking \
  --candidate-selection-guidance \
  --cohesive-namespace-candidates \
  --cache-dir /tmp/gtc-bfcl-qwen027-cohesive-guard-limit25-cache \
  --concurrency 6 \
  --progress \
  --progress-every 25 \
  --output /tmp/gtc-bfcl-qwen027-cohesive-guard-limit25-paired.json

Output includes:

row_vs_retrieved k=5 repeat=1 paired=100 row_pass_loss=11 retrieved_on_row_pass=0.883 retrieval_layer_failures=argument_value_mismatch:2,candidate_ambiguity:8,retrieval_miss:1

Validation

  • poetry run pytest tests/test_bfcl_tool_selection_sweep.py -q
  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q (955 passed, 3 skipped)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant