[Bug]: `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` is not enforced — image selection relies entirely on an unverified LLM response, with no deterministic fallback

[sh4nx0r]

Affected Component

External Integrations (LLM/Search APIs)

Describe the bug

Summary

DOCKER_DEFAULT_IMAGE_FOR_PENTEST (and the equivalent DefaultImageForPentest template variable) is documented and configured as if it constrains which Docker image PentAGI uses for a flow. In practice, it is only ever injected as text into an LLM prompt (image_chooser prompt type). The LLM's raw text response is used directly as the image name, with no validation, no allow-list check, and no deterministic fallback if the model ignores the guidance. With a weaker/smaller model (tested with a local, self-hosted qwen2.5:7b-instruct via Ollama), the model reliably ignores the "prefer the pentest image" instruction and returns node:latest instead of the intended pentest image, even when the env var is correctly set and even after explicitly strengthening the prompt's wording.

This means DOCKER_DEFAULT_IMAGE_FOR_PENTEST currently functions as a hint, not a default or constraint, contrary to what its name and documentation imply.

Environment

• PentAGI commit: 879e87c (pinned, built from clean upstream source — see reproduction steps below)
• Deployment: self-hosted Docker Compose, Ubuntu 26.04 host
• LLM provider: local Ollama (qwen2.5:7b-instruct, ~5.1GB, CPU-only inference)
• DOCKER_DEFAULT_IMAGE_FOR_PENTEST=vxcontrol/kali-linux (also reproduces with the unset/default value, which per config.go defaults to the same image)

Steps to reproduce

1. Configure a local/self-hosted LLM provider (in our case, Ollama running qwen2.5:7b-instruct) as the active provider for a flow.
2. Submit a flow whose task description could plausibly read as either a general/web task or a security-testing task — e.g. "Inspect Juice-Shop API CORS configuration."
3. Observe the resulting flow's terminal container image via docker ps or the Terminals column in the Flows UI.

Expected: the terminal container uses the configured pentest image (vxcontrol/kali-linux), consistent with DOCKER_DEFAULT_IMAGE_FOR_PENTEST.

Actual: the terminal container uses node:latest.

This was reproduced identically across two separate flows, submitted hours apart, on two different Docker images (see root-cause investigation below) — ruling out a one-off model hallucination.

Root cause (traced in source, not assumed)

In backend/pkg/providers/providers.go, function NewFlowProvider:

imageTmpl, err := prompter.RenderTemplate(templates.PromptTypeImageChooser, map[string]any{
    "DefaultImage":           pc.docker.GetDefaultImage(),
    "DefaultImageForPentest": pc.defaultDockerImageForPentest,
    "Input":                  input,
})
if err != nil {
    return nil, fmt.Errorf("failed to get primary docker image template: %w", err)
}
image, err := prv.Call(ctx, pconfig.OptionsTypeSimple, imageTmpl)
if err != nil {
    return nil, fmt.Errorf("failed to select primary docker image via llm call: %w", err)
}
image = strings.ToLower(strings.TrimSpace(image))

DOCKER_DEFAULT_IMAGE_FOR_PENTEST (surfaced here as pc.defaultDockerImageForPentest, sourced from config.go's DockerDefaultImageForPentest field) is used only as a template substitution value inside the image_chooser prompt (backend/pkg/templates/prompts/image_chooser.tmpl). The LLM's raw text response becomes the literal image string used for the flow's terminal container, with:

• No check that the returned string is one of the two configured images (DefaultImage / DefaultImageForPentest) or any known-good value at all.
• No fallback if the LLM's answer doesn't match anything sensible.
• No way for a deployment to force deterministic behavior short of forking this function.

We initially suspected this was caused by a locally-built, unsourced/custom Docker image running our deployment (built from an unknown commit, with undocumented modifications). To rule this out, we:

1. Cloned upstream vxcontrol/pentagi fresh, checked out pinned commit 879e87c.
2. Applied a documented, minimal prompt-strengthening edit to image_chooser.tmpl (changing "for ambiguous or uncertain cases, use {{.DefaultImage}}" to "for ambiguous or uncertain cases, prefer {{.DefaultImageForPentest}} over {{.DefaultImage}}, since tasks on this platform are security-testing in nature").
3. Built a clean image from this source (docker build, no errors, verified the compiled binary contains the exact patched prompt string via strings /opt/pentagi/bin/pentagi | grep ...).
4. Swapped the deployment onto this clean, verified image.
5. Re-ran an equivalent flow.

Result: identical failure. node:latest was selected again, even with a stronger, verified, correctly-compiled prompt nudge in place. This confirms the bug is not caused by any custom/unsourced build — it's inherent to relying on unconstrained LLM output for a decision that has real operational consequences (a security-testing task running in a general-purpose image without the intended toolset).

Why this matters

For a platform whose stated purpose is security/pentest automation, the terminal container's toolset is not cosmetic — an agent that lands in node:latest instead of a pentest-tooling image cannot perform the intended work correctly, and the failure is silent: no error is raised, no warning is logged, the flow simply proceeds in the wrong environment. This is especially significant for deployments using smaller/local/self-hosted models (increasingly common given cost and data-locality motivations), where prompt-following reliability is measurably weaker than with frontier hosted models — exactly the population most likely to hit this bug in practice.

Suggested fix

DOCKER_DEFAULT_IMAGE_FOR_PENTEST should be enforced deterministically, not merely suggested. Two options, in order of preference:

1. Skip the LLM call when the deployment's use case is unambiguous. Many deployments (like ours) are exclusively for security testing — for these, add a config option (e.g. DOCKER_IMAGE_SELECTION_MODE=fixed vs. llm) that, when set, bypasses prv.Call(...) entirely and uses DefaultImageForPentest directly.
2. At minimum, validate the LLM's response before trusting it. After receiving image back from the model, check it against a small allow-list (DefaultImage, DefaultImageForPentest, or any explicitly configured additional images) and fall back to DefaultImageForPentest if the response doesn't match a known-good value. This preserves the "let the LLM decide when it's genuinely ambiguous" intent while eliminating the silent-wrong-environment failure mode.

Option 2 is likely the smaller, safer change and would meaningfully improve robustness without removing the flexibility the current design intends.

Additional notes

• We can provide full logs, the exact prompt template diff, and the flow IDs/timestamps from our reproduction if helpful.
• Happy to test a patch against our own deployment if one is proposed.

Steps to Reproduce

1. Access PentAGI Web UI at https://:8443/ (self-hosted deployment)
2. Configure an LLM provider using a local/self-hosted model (reproduced with Ollama running qwen2.5:7b-instruct, CPU-only). Confirm DOCKER_DEFAULT_IMAGE_FOR_PENTEST is set (we used the default, vxcontrol/kali-linux) in the deployment's .env.
3. Start a new flow with a prompt describing a security-testing task that could also read as a general/web task, e.g. "Inspect Juice-Shop API CORS configuration."
4. Observe the flow's assigned terminal container image via the Flows UI "Terminals" column, or docker ps --filter name=pentagi-terminal-
5. Error occurs when the flow's terminal container is created: instead of the configured pentest image (vxcontrol/kali-linux), the container uses node:latest. No error, warning, or log entry is raised — the flow proceeds silently in the wrong environment. Reproduced identically across two separate flow submissions, hours apart.

System Configuration

PentAGI Version: custom build from pinned upstream commit 879e87c (not Docker Hub — built via docker build from a clean vxcontrol/pentagi checkout, with one documented prompt-template patch applied to image_chooser.tmpl — see reproduction steps)

Deployment Type:

• Docker Compose
• Custom Deployment

Environment:

• Docker Version: 29.5.3 (build d1c06ef)
• Docker Compose Version: v5.1.4
• Host OS: Ubuntu 26.04
• Available Resources:
  • RAM: 7.2GB
  • CPU: 4 cores
  • Disk Space: 56GB free

Enabled Features:

• Langfuse Analytics
• Grafana Monitoring
• Custom LLM Server

Active Integrations:

• LLM Provider: Custom (self-hosted Ollama — qwen2.5:7b-instruct)
• Search Systems: none configured (DuckDuckGo/Google/Traversaal/Tavily/Perplexity all unset)

Logs and Artifacts

No response

Screenshots or Recordings

No response

Verification

• I have checked that this issue hasn't been already reported
• I have provided all relevant configuration files (with sensitive data removed)
• I have included relevant logs and error messages
• I am running the latest version of PentAGI