Skip to content

Recover BFCL route planning retrieval - #56

Closed
SonAIengine wants to merge 1 commit into
codex/bfcl-equivalent-sibling-pruningfrom
codex/bfcl-route-retrieval-hardening
Closed

Recover BFCL route planning retrieval#56
SonAIengine wants to merge 1 commit into
codex/bfcl-equivalent-sibling-pruningfrom
codex/bfcl-route-retrieval-hardening

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • Add deterministic route-planning query expansion for fastest/best/optimal/shortest route intents.
  • Boost route-planning tools when the user intent is split across operation text and parameter enum values, e.g. description says best route while method enum carries fastest.
  • Add a regression test for the BFCL multiple_24 failure shape where chess/event context previously displaced the route planner.
  • Update research docs to mark multiple_24 route retrieval miss as recovered and narrow the next bottleneck to sequence+3D over-decomposition.

Validation

  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/test_retrieval.py -q -> 14 passed
  • poetry run pytest tests/ -q -> 968 passed, 3 skipped, 1 warning
  • Deterministic BFCL retrieval all four categories: /tmp/gtc-bfcl-route-hardening-deterministic.json -> 1000 cases, recall@5 0.953, status pass
  • BFCL multiple_24 deterministic: route_planner.calculate_route moved from rank 8 to rank 1; recall@1/3/5 1.00
  • Fresh Qwen single-case smoke: /tmp/gtc-bfcl-route-hardening-multiple24-qwen.json -> evaluator exact 1.00, retrieved exact 1.00

Remaining Bottleneck

  • parallel_3: model/tool selection over-decomposes by calling lower-level sequence-only plus sequence+3D tools together.

@SonAIengine

Copy link
Copy Markdown
Owner Author

Closing this internal stacked PR as superseded by the current main branch. The branch is preserved for history.

@SonAIengine SonAIengine closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant