diff --git a/changelog.d/20260902_131734_doug.goldstein.md b/changelog.d/20260902_131734_doug.goldstein.md new file mode 100644 index 000000000..96993a173 --- /dev/null +++ b/changelog.d/20260902_131734_doug.goldstein.md @@ -0,0 +1,50 @@ +### Deploy repo changes + +`envoy-configs` gained knobs to scale the Envoy proxy pods behind a Gateway. +Both `gateways.external` and `gateways.internal` now accept `envoyDeployment` +and `externalTrafficPolicy`, in `envoy-configs/values.yaml`: + +```yaml title="envoy-configs/values.yaml" +gateways: + external: + name: external-gateway + namespace: envoy-gateway + className: eg + envoyDeployment: + replicas: 3 + container: + resources: + requests: + cpu: "250m" + memory: "256Mi" + limits: + cpu: "1" + memory: "512Mi" + externalTrafficPolicy: Local +``` + +This was added to give the shared external Gateway enough proxy replicas and +headroom to absorb bursty traffic to nova's osapi, which routes through it +alongside every other HTTP/TLS route. Both fields are optional; leaving them +unset keeps Envoy Gateway's own defaults (a single replica, no resource +requests or limits). See the +[Gateway Schema reference](../operator-guide/gateway-api.md#gateway-schema) +for the full set of fields. + +### Notes + +nova osapi (`nova_api_uwsgi`) was re-tuned to survive bursts of concurrent +requests, which were filling the default 2-worker / 100-deep listen queue and +causing connection refusals and liveness-probe restarts: + +- uWSGI `processes` raised from 2 to 8, and a `listen` backlog of 1024 was + added. Values above the node's `net.core.somaxconn` are silently capped, so + confirm that sysctl is at least 1024 on osapi nodes. +- The osapi liveness probe was loosened (`initialDelaySeconds: 30`, + `periodSeconds: 15`, `timeoutSeconds: 10`, `failureThreshold: 6`) so a slow + response under load no longer triggers a destructive restart. Readiness + keeps a tighter probe so busy pods still drop out of rotation briefly. +- Default osapi pod resources increased: requests from `256Mi`/none to + `512Mi`/`500m`, with new limits of `2Gi`/`4` CPU. This applies automatically + on the next resync -- confirm your cluster has the headroom before + upgrading, particularly if `pod.replicas.osapi` is scaled up. diff --git a/docs/deploy-guide/components/nova.md b/docs/deploy-guide/components/nova.md index 85a82a62f..b2766a619 100644 --- a/docs/deploy-guide/components/nova.md +++ b/docs/deploy-guide/components/nova.md @@ -55,3 +55,4 @@ Optional additions: ## Notes - This service is rendered by `application-openstack-helm.yaml`, which also reads the shared site-level `secret-openstack.yaml` and optional `images-openstack.yaml` files before it reads `nova/values.yaml`. +- The osapi pod's default `uwsgi.processes` (8), `uwsgi.listen` backlog (1024), liveness/readiness probe timings, and `pod.resources.api` requests/limits are tuned in `components/nova/values.yaml` to absorb bursts of concurrent API requests. Override `pod.resources.api` in your deploy repo's `nova/values.yaml` if your cluster needs a different resource footprint, and confirm node `net.core.somaxconn` is at least 1024 so the listen backlog isn't silently capped. diff --git a/docs/operator-guide/gateway-api.md b/docs/operator-guide/gateway-api.md index a7dfa66d3..c5e5e3b46 100644 --- a/docs/operator-guide/gateway-api.md +++ b/docs/operator-guide/gateway-api.md @@ -489,6 +489,55 @@ The `envoy-configs` chart is deployed via ArgoCD and generates Gateway and Route **Site-specific values**: `deploy//envoy-configs/values.yaml` +### Gateway Schema + +```yaml +gateways: + external: # or internal + name: string # Required: name of the Gateway resource + namespace: string # Required: namespace where the Gateway is deployed + className: string # Required: GatewayClass name (e.g., "eg") + serviceAnnotations: {} # Optional: annotations for the generated Service + externalTrafficPolicy: string # Optional: "Cluster" or "Local" + envoyDeployment: # Optional: tune the Envoy proxy pods for this Gateway + replicas: integer # Number of Envoy proxy pod replicas (minimum: 1) + container: + resources: + requests: + cpu: string + memory: string + limits: + cpu: string + memory: string +``` + +`envoyDeployment` is unset by default, which keeps Envoy Gateway's own defaults +(a single replica with no resource requests or limits). Raise `replicas` and +set `container.resources` when a Gateway is fronting a service that needs to +absorb request bursts, for example: + +```yaml +gateways: + external: + name: external-gateway + namespace: envoy-gateway + className: eg + envoyDeployment: + replicas: 3 + container: + resources: + requests: + cpu: "250m" + memory: "256Mi" + limits: + cpu: "1" + memory: "512Mi" +``` + +Scale the Gateway's Envoy proxies, not the backend service's own pods, when +the bottleneck is connection handling or TLS termination in front of the +service rather than the service itself. + ### Route Schema ```yaml