Add serving-llms-on-instinct walkthrough - #155
Conversation
Adds a step-by-step walkthrough for the existing serving-llms-on-instinct skill, mirroring the structure of serving-llms-on-epyc, and lists it in the walkthroughs index.
|
@Mahdi-CV Can you please take a look? |
Mahdi-CV
left a comment
Mahdi-CV
left a comment
There was a problem hiding this comment.
Looks great. Thanks @satyam-jani just a few minor changes will make it more user friendly and complete IMO
|
|
||
| ## Step 3 - Running the skill | ||
|
|
||
| Run `claude --model sonnet` on your Instinct host with this prompt: |
There was a problem hiding this comment.
let's not enforce any model
| Run `claude --model sonnet` on your Instinct host with this prompt: | ||
|
|
||
| ``` | ||
| Serve Qwen/Qwen3.5-9B on this AMD Instinct GPU with vLLM. |
There was a problem hiding this comment.
we should mention the user either can run this skill if claude is installed on the server like the way you have or if the server is reachable with ssh then they can just give the server address.
| Once Claude reports the endpoint is healthy, use the **base URL, served-model name, and | ||
| port from Claude's connection table** (it uses port `8000` by default): | ||
|
|
||
| ```bash |
There was a problem hiding this comment.
let's remove this and just ask them to follow the terminal instructions. The curl command might look different for some models.
| Prefer Python? Point the OpenAI SDK at the local server (`base_url` ends in `/v1`; | ||
| the SDK needs a non-empty key, so use any placeholder when there is no auth): | ||
|
|
||
| ```python |
There was a problem hiding this comment.
I'd also remove this python part just to make things future proof for different model and just keep the instructions as follow agent's output which should include a curl command on how to reach the model and prompt it.
3 participants
Adds a step-by-step walkthrough for the existing serving-llms-on-instinct skill, mirroring the structure of serving-llms-on-epyc, and lists it in the walkthroughs index.