benchmarks: confirm --model against the server before running (#10) - #11
Merged
Merged
Conversation
llama.cpp accepts any model name and answers with whatever is loaded, so 'node benchmarks/run-semif.mjs' without --model produced rows labelled qwen3.8-27b-text-64k on a server holding something else. The llama.cpp benchmark now lists GET /v1/models first and stops unless the requested model is served, naming what the server does hold. The report's runtime block records resolvedModel and modelChecked. --skip-model-check runs anyway and records the run as unchecked.
Owner
|
Thanks for the contribution! I updated the branch to master, aligned baseURL validation, and tightened the diagnostics and docs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10.
What changed
--modelagainst the exact IDs returned by llama.cpp's/v1/modelsbefore inference.--skip-model-checkand record unchecked runs withruntime.modelChecked: false.baseURLbefore model discovery so credentials, query parameters, and fragments cannot reach the request or error output.Why
llama.cpp may answer with its loaded model even when the request contains another model name. Without the preflight check, a benchmark report can be labelled with a model that never ran.
Checks
npm run check— 123 testsnpm pack --dry-runcd packages/choosekit-mcp && npm run check— 17 testscd packages/choosekit-mcp && npm pack --dry-runqwen3.8-27b-text-64k— 144/144 rows completed, 0 runtime errors,modelChecked: true