Flexible LLM providers + manage Ollama with 4CAT

[dale-wahl]

That's right, boys and girls, now you can spin up an Ollama container right beside your 4CAT containers. Lil admin UI action to pull and delete models on it (should work with other Ollama servers as well) as well as enable and disable models (should work with tags as well, but did not test that).

The gist: docker compose -f docker-compose.yml -f docker-compose_ollama.yml up -d

It's that simple! (Or almost that simple; you do need to un-comment some lines if you want it to use GPU, but a) works without GPU--albeit slowly--and b) doesn't crash for those GPU-less users.)

You're welcome.

Fixes #564, fixes #563

[Copilot]

Pull request overview

This PR adds Ollama LLM container support to 4CAT's Docker stack, along with an admin UI to manage LLM models (pull, delete, enable/disable). The LLM model refresh logic is extracted from the old refresh_items worker into a dedicated OllamaManager worker. A new llm.enabled_models configuration setting allows admins to control which available models are exposed to users.

Changes:

• New OllamaManager backend worker for refreshing, pulling, and deleting Ollama models via the Ollama HTTP API
• New /admin/llm/ admin panel (views_llm.py + llm-server.html) for managing LLM models, gated by both admin privileges and llm.access
• New docker-compose_ollama.yml override for running Ollama as a Docker sidecar, with auto-configuration in docker_setup.py

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
backend/workers/ollama_manager.py	New worker for Ollama model refresh/pull/delete operations
backend/workers/refresh_items.py	LLM refresh logic removed; worker now does nothing
webtool/views/views_llm.py	New Flask blueprint for the admin LLM management panel
webtool/templates/controlpanel/llm-server.html	New admin panel template for model listing and actions
webtool/templates/controlpanel/layout.html	Adds "LLM Server" nav link when llm.access is enabled
webtool/__init__.py	Registers the new views_llm blueprint
processors/machine_learning/llm_prompter.py	Filters available models by enabled list before showing to users
common/lib/config_definition.py	Adds llm.enabled_models config definition
docker/docker_setup.py	Auto-configures LLM settings when Ollama is detected on Docker network
docker-compose_ollama.yml	New Docker Compose override for the Ollama sidecar service
docker/README.md	Documents the Ollama Docker setup and usage

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

When a model is successfully deleted from the Ollama server, refresh_models() updates llm.available_models to remove the deleted model, but llm.enabled_models is never cleaned up. This means deleted models accumulate as stale entries in llm.enabled_models. While this doesn't cause an immediate runtime error (since llm_prompter.py intersects the two lists), it's misleading: after a delete-and-refresh cycle, the model would disappear from the available models table in the UI, but it remains in the enabled list. If the model is later re-pulled, it would reappear as already enabled, which could be surprising.

The delete_model() method (or the work() method after a successful delete) should remove the model from llm.enabled_models, or at minimum refresh_models() should reconcile llm.enabled_models to remove entries no longer present in llm.available_models.

[Copilot]

The docker/README.md section "Configuring 4CAT to use Ollama" (steps 1–3) instructs users to manually set the LLM Provider Type, LLM Server URL, and LLM Access fields in the Control Panel Settings. However, docker/docker_setup.py now automatically detects the Ollama sidecar on first startup and configures these settings without user intervention. The README should mention this auto-configuration so users know they can skip steps 1–3 on a fresh install with the Ollama override.

[dale-wahl]

@copilot open a new pull request to apply changes based on the comments in this thread:

• remove enabled models via refresh_models, if they have been deleted.
• update docs to reflex docker auto setting those variables, but ensure it is also clear how to manually set them to for another Ollama server
• keep the refresh_items worker for later use, but comment out the ensure job or have it prematurely return since it is at the moment doing nothing.

[Copilot]

@dale-wahl I've opened a new pull request, #581, to work on those changes. Once the pull request is ready, I'll request review from you.

[stijn-uva]

• Add support for multiple LLM providers, via config setting llm.providers
• LLM manager worker checks all providers and loads list of available models
• Providers may or may not implement pull_model and delete_model methods, if implemented models can be added/removed via control panel
• Third-party APIs (via llms.json) are also available via such a 'provider' (a special case)
• LLM prompter processor simply lists all enabled models, no need to choose from local/api/etc
• LLM classes now in common/lib/llm

Todo:

• further testing (probably broke a few things refactoring this all)
• llm.access config setting doesn't really work at the moment
• Investigate if other providers can also get a pull/delete_model method (not really)
• Migrate script
• Ensure Docker Ollama is added as a provider on install when Dockering

[stijn-uva]

@dale-wahl this is now mostly ready to merge, I think - need to test the migrate script (which I plan to do after merge, when we can test it on our own 4CAT) and the Docker setup. Can you do the latter? I tried it but it pulled the stable image, which doesn't contain the relevant updates to the LLM code yet, so that would fail. Maybe that also needs to be tested after merging?

[sal-uva]

@stijn-uva running migrate from 1.53 results in the following error

Checking if llm.providers setting exists...
    ...does not exist, filling with currently configured proviers

  Unexpected error while running migrate-1.54-1.55.py. Migration halted.
  The following exception occurred:

Traceback (most recent call last):
  File "C:\Users\shagen\repos\4cat\helper-scripts\migrate\migrate-1.54-1.55.py", line 51, in <module>
    provider_type = {"ollama": "ollama"}.get(provider_type, "openai-like")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'RealDictRow'

[stijn-uva]

@sal-uva should be fixed now, thanks!

[dale-wahl]

startswith("api-") / provider type == "api" is doing a lot of work.

So far, it seems to mean:

1. models come from the static llms.json menu,
2. the user supplies the API key,
3. data leaves the instance to a commercial service. (for our comments/tooltips and such)

I am still reviewing, but I need to drop this someplace because it feels off. I think there are multiple axis here (e.g. the wrapper we use (Ollama, Anthropic, OpenAI etc.), where it is hosted, who's credentials are used).

Possibly related (?), third party (the static file llms) are not showing as available to actually select. I think it is semi related because they have the api- type, but the processor itself needs to know what looks like it was moved to provider_key. Also, I haven't tested, but a new build might fail as I think _id only appears from the migrate script.

Reminder for me: look at docker_setup.py; it did not detect Ollama or at least it does not appear as a provider, also might overwrite existing providers...

[dale-wahl]

@stijn-uva this looks weird to me. maybe options = settings.get("options", {}) if it really is supposed to be a dict. list.values() is going to fail.

Also the if choice not in match_options uses the chain iterator to check for choice so you'll only be left with the remaining items in match_options.

[dale-wahl]

This one took me a bit. I do think some renaming would help as "providers" collides with OpenAI, Anthorpic, etc. and our "providers" is more connections to LLMs... services 😅 like Ollama vs LiteLLM etc. I also think renaming api to thirdparty may be beneficial.

Once I understood the providers/clients were connections, I think we could expand on them to help with the other "axis" I mentioned in my earlier comment. I added wrapper to fix the Third Party class and allow you to connect which LangChain wrapper to use. We could also add an egress key so you could denote which connections are external vs internal (e.g. do we warn if a user is sending data to UvA via LiteLLM or whatever setup others come up with). You could also add key_source and allow users to provide the key. That axis is perhaps less important, but we are conflating it now with the api- means thirdparty "provider". And I wouldn't mind adding my own keys to providers for my own instance. (Plus we could then have keys available to groups of users by making providers/connections available by tag...).

All that said, I tested out some configurations and the docker setup and think we are pretty good.