Skip to content

Expose graphify tool equivalence evidence - #46

Open
SonAIengine wants to merge 1 commit into
codex/bfcl-duplicate-surface-attributionfrom
codex/tool-equivalence-evidence
Open

Expose graphify tool equivalence evidence#46
SonAIengine wants to merge 1 commit into
codex/bfcl-duplicate-surface-attributionfrom
codex/tool-equivalence-evidence

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • add public graphify.build_tool_equivalence_groups(...)
  • include evidence-only target_equivalence_groups and target_equivalence_group_count in build_candidate_set(...)
  • refactor BFCL near-duplicate failure tagging to use the new graphify helper instead of benchmark-private duplicate-surface logic
  • document the contract in API/reference docs and the research loop

Why

The current 0.27 bottleneck includes row-pass/retrieved-fail cases where retrieved candidates contain semantically near-equivalent sibling tools. We need this as product-neutral evidence for XGEN target selectors, not only as a BFCL benchmark diagnostic.

This change does not merge, suppress, rerank, or score differently by default. It only emits deterministic equivalence evidence so adapters can inspect/display duplicate candidate surfaces and later decide how to use them.

Evidence

Cached 100-case qwen3.6-27B rerun, using existing model outputs:

poetry run python -m benchmarks.bfcl_tool_selection.sweep \
  --categories simple_python,multiple,parallel,parallel_multiple \
  --limit 25 \
  --top-ks 5 \
  --tool-sources row,retrieved \
  --model qwen3.6-27b \
  --llm-url http://127.0.0.1:18000/v1 \
  --disable-thinking \
  --candidate-selection-guidance \
  --cohesive-namespace-candidates \
  --cache-dir /tmp/gtc-bfcl-qwen027-cohesive-guard-limit25-cache \
  --concurrency 6 \
  --progress \
  --progress-every 25 \
  --output /tmp/gtc-bfcl-qwen027-cohesive-guard-limit25-equivalence.json

Output still reports the same row/retrieved gap and near-duplicate tag count:

row_vs_retrieved k=5 repeat=1 paired=100 row_pass_loss=11 retrieved_on_row_pass=0.883 retrieval_layer_failures=argument_value_mismatch:2,candidate_ambiguity:8,retrieval_miss:1 retrieval_layer_tags=near_duplicate_tool_surface:4

Validation

  • poetry run pytest tests/test_graphify_contract_025.py tests/test_bfcl_tool_selection_llm_loop.py tests/test_bfcl_tool_selection_sweep.py -q
  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q (957 passed, 3 skipped)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant