Repository navigation
How to use it when there is only one model locally? #202
Description
My current issue is that my graphics memory is only sufficient to deploy one model locally, but the config requires two. How should I configure it to use one locally and one online? Because I see that only one api_base can be set in the config. Or how can I use two models with different api_bases?
Activity
codelion commentedon Aug 13, 2025
You can use a single model by just putting the weight to 1.0. It is currently not possible to use different api_base but you can use an existing proxy or router like OptiLLM - https://github.com/codelion/optillm to router between models as needed with the same url.