Skip to content

Add BFCL retrieval rank hint ablation - #40

Open
SonAIengine wants to merge 1 commit into
codex/bfcl-sweep-milestone-gatesfrom
codex/bfcl-rank-hint-ablation
Open

Add BFCL retrieval rank hint ablation#40
SonAIengine wants to merge 1 commit into
codex/bfcl-sweep-milestone-gatesfrom
codex/bfcl-rank-hint-ablation

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • add an opt-in --retrieval-rank-hints ablation for BFCL model loops and sweeps
  • record the option in reports/cache keys and prefix retrieved-source tool descriptions with graph rank hints only when enabled
  • document the qwen3.6-27B smoke evidence and the next bottleneck: candidate ambiguity / call-count mismatch, not retrieval miss

qwen3.6-27B Smoke Evidence

Via go165 vLLM qwen3.6-27b, category limit 5, row/retrieved k=5:

Baseline artifact: /tmp/gtc-bfcl-qwen027-smoke-live.json

  • row exact: 1.00
  • retrieved exact: 0.85
  • retrieval recall: 1.00
  • milestone status: fail
  • failed gates: row preservation 0.85 < 0.94, parallel_multiple 0.60 < 0.75

Rank-hint artifact: /tmp/gtc-bfcl-qwen027-rankhint-smoke.json

  • retrieved exact: 0.85
  • retrieval recall: 1.00
  • parallel_multiple: 0.60
  • aggregate improvement: none on this small smoke

Interpretation: simple rank wording is not enough; next work should focus on candidate equivalence/grouping and multi-call set construction.

Validation

  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q -> 950 passed, 3 skipped

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant