Allow selecting the embedding model (and other utiliy models) from different providers?

[h8ghbzc9bbr8c7]

Currently when I select the embedding model (and other utiliy models) I can only select models which the currently selected AI provider offers. This works well when the selected provider offers more than one (main and utility) model.
This works well for i.e. ollama which can serve more than one model through the same provider. But i.e. vllm can only serve one model in one instance. Also the capabilities of different providers may be differently suited for different kinds od models (a embedding model may only need a CPU where a "big" model needs a gpu).
I understand that I can use a proxy (LiteLLM for example) to combine different providers (not tested yet but should work).
But would it not be better when Askimo would allow to select utility models from different providers? This would give much more flexibility for "Combined scenarios".

[haiphucnguyen]

Hi @h8ghbzc9bbr8c7, thank you for raising many valuable ideas - this is something I have also been thinking about a lot, and it relates directly to why we want to support models from different providers. I also have use cases beyond the ones you mentioned:

Anthropic does not have an embedding model yet, so RAG is not supported when a user is on Anthropic. Similarly, I want to support audio models, but we don't have good alternatives comparable to OpenAI's audio models. So not all providers provide all models satisfy users' needs

So I do see the need to use different models from different providers at the same time, or in general how we let users select the best models from multiple providers (cheap model from local ai, reasoning model from Claude/OpenAI, image model from X/Gemini, embedding model from OpenRouter) etc. The challenge for Askimo is not really a technical one - it's about how to design a UX that remains easy for our users. Even though Askimo is open source, the majority of our users are non-technical - not developers or advanced users. For that reason, I am hesitant to add more complexity to the UI.
Server-side solutions like OpenWebUI and LiteLLM have their own advantages - for example, they allow admins to aggregate multiple providers through a proxy, as you suggested - but they also come with their own trade-offs. To keep the user experience simple, the community edition will only support one provider instance at a time.

I am currently developing a Team Edition that helps users manage providers and more on the server side. I have enclosed the screenshot for reference. With the Team Edition, users can configure multiple providers easily, and the Askimo desktop app avoids the UX complexity - because the app only connects to one provider endpoint, which can then route to different underlying models: for example, a chat model from OpenAI, an image model from Gemini, and a utility model from Ollama.

If you are interested, then I can send you when the software is ready to review (Note: it is closed source).

[h8ghbzc9bbr8c7]

Thank you for the detailed response and for sharing your thoughts! I'm very interested in new features, and I'm also excited to see what the Team Edition will bring. I hope you have success with it, and I wish you all the best!
But the main reason I like askimo so much is that it’s an application that runs entirely locally and doesn’t require any central configuration. We’re a team of tech-savvy scientists, and that fits our way of working perfectly.
I’d also like to stick strictly to open-source modules when it comes to AI. In that regard, askimo in its current form is the best option I’ve found so far—especially since it’s not a Node.js application :-). I’ll continue to try to suggest improvements, but I understand if you want to—and need to—keep the local and team versions separate.

[haiphucnguyen]

especially since it’s not a Node.js application :-)

Yes, it is not the Electron wrapper application :). Before writing Askimo, I tried to use some available tools both web and node.js but not satisfied about the performance, and application responsive when I had many messages in a session, so I created another chat client.

If you use Askimo in a group of users, then the solution is using LiteLLM like your suggestion is the best. This solution can help users bring their key in each askimo instance. I have not try it yet, but it is a big favor to us if you can try and let us know if it works, then we can update our document to let Askimo users know they can use askimo in team without manage the keys by their own. That's very helpful for us and the community users as well.

[haiphucnguyen]

I tested with LiteLLM, and with the LiteLLM gateway it works well with Askimo and lets Askimo use different AI providers for utility, embedding, etc. I wrote a blog post http://askimo.chat/blog/share-ai-models-team-litellm-proxy/ to share how Askimo works with LiteLLM. The combination of Askimo and LiteLLM fits your use case well.

[h8ghbzc9bbr8c7]

Great! I tested in the meantime with nginx which also works. But I did not try to share API keys, we currently have none. I will use you solution when we have to manage API keys (which may happen soon). Thanks!