Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/styles/base/Dictionary.txt
Original file line number Diff line number Diff line change
Expand Up @@ -994,3 +994,25 @@ semconv
rollup
rollups
llm_format
Autodiscovery
autostart
CVEs
Dockerized
endscript
HAProxy
hardcodes
Idletime
IDPs
Kyverno
logrotate
loopbacking
multiline
OOMKiller
OOMKilled
postrotate
recrawl
reenable
stunnel
supercookie
systemctl
tmp
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
title: After upgrading to Kong 3.10.0.0 I can not log into Kong Manager after making documented changes
content_type: support
description: Kong 3.10.x moved Kong Manager session handling to the openid-connect plugin, which can produce a session cookie header larger than Nginx's default proxy buffer, causing login to fail with an upstream error.
products:
- gateway
works_on:
- on-prem
- konnect
tldr:
q: Why can't I log into Kong Manager after upgrading to Kong 3.10.0.0 and configuring openid-connect authentication?
a: |
Kong 3.10.x moved Kong Manager session handling to the openid-connect plugin, which returns a session cookie header that can be larger than in previous versions — often larger than Nginx's default header buffer when proxied through Kong, causing an "invalid response from upstream" error. Increase `nginx_proxy_proxy_buffer_size` (try 8k, 10k, 12k, or 16k) to fix it; note this is a global Nginx setting that affects all traffic through the same data plane.
related_resources: []
---

## Problem

While upgrading to Kong 3.10.0.0 we changed the Kong Manager configuration according to the document requirements when using openid-connect authentication for the Manager.

We are proxying all traffic to the Kong Manager service through Kong, and after the changes made, we can still not log into Kong Manager but get the following error in the Browser:

```
Error

An invalid response was received from the upstream server.

request_id: 8150ea003c03816da5ecc51920993417
```

Are we missing a configuration option?

## Cause

The changes regarding openid-connect authentication in Kong Manager in version 3.10.X.X of Kong mean that the session handling is now done via the openid-connect plugins. This means the plugin will initiate the creation of a session cookie as documented in the openid-connect plugin documentation.

With this new configuration a session cookie header will be sent back to the Browser client which depending on what information is returned by the IdP in the tokens it generates during the login process can be much larger than the session data that was generated in previous versions when using the default session object.

If you are proxying your traffic through Kong, this may breach the default header limit, and you should see an error log entry similar to this:

```
2025/03/12 12:34:49 [error] 2385#0: *4019 upstream sent too big header while reading response header from upstream, client: 192.168.1.4, server: kong, request: "GET /auth?code=<code>&state=<state>&session_state=:session_state>", host: "api.kong.lan", referrer: "https://login.live.com/", request_id: "8150ea003c03816da5ecc51920993417"
```

## Solution

To address this issue, you will have to increase the `nginx_proxy_proxy_buffer_size` setting.

Since the size of the cookie headers depends on what information the IdP returns in the tokens, you may have to try different values for the `nginx_proxy_proxy_buffer_size` starting with 8k, 10k, 12k or 16k.

Please note that this setting is a global setting which means all requests that are proxied through the same Kong data plane as the Kong Manager traffic will be affected by this, i.e. traffic with larger headers will be allowed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
title: OIDC authenticated group mapping fails when using the `groups` claim with Azure AD
content_type: support
description: "With Azure AD as the IdP, use the `roles` claim (not `groups`) as the `authenticated_groups_claim` so RBAC role mapping resolves correctly, since the `groups` claim only contains UUIDs."
products:
- gateway
works_on:
- on-prem
- konnect
related_resources: []
tldr:
q: Why doesn't OIDC authenticated group mapping work with the `groups` claim in Azure AD?
a: |
Azure AD's `groups` claim returns UUIDs, so mapping `authenticated_groups_claim` to `groups` won't match your RBAC role names. Use the `roles` claim instead, and assign string values in the format `<workspace_name>:<role_name>` (e.g. `default:super-admin`) to the Azure AD app roles. If mapping still doesn't work, enable `debug` log level on Kong and check log entries prefixed with `[openid-connect]`.
---

## Problem

We are trying to set up openid-connect authentication in Kong Manager with OIDC Authenticated Group Mapping with Azure AD using `authenticated_groups_claim` set to `groups`, but since the `groups` claim only contains UUIDs, this does not lead to the user logging in getting the correct RBAC role assigned. How can we fix this?

## Solution

With Azure AD as the IdP, you should not use `groups` as the `authenticated_groups_claim`, but use `roles` instead. When setting up roles in Azure AD, you can assign string values to the `roles` claim that match the requirement necessary for the group mapping to work. As per documentation, the claim needs to have string values of the format `<workspace_name>:<role_name>`, e.g. `default:super-admin`, for mappings to be successful.

If you have tried setting roles in the way that should work as per the Kong documentation but still do not see the correct role assignment when logging in, enable `debug` log level on Kong, and check the log entries prefixed with `[openid-connect]` to see what is not working correctly during the login process.
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Azure Load Balancer health probe fails for Kong proxy service
content_type: support
description: When running Kong on Kubernetes v1.24 or higher with Azure Load Balancer, the health probe created by the Load Balancer fails and causes the Azure Kubernetes service to restart the Kong proxy service repeatedly, typically when HTTP is enabled for the proxy service.
products:
- kic
works_on:
- on-prem
- konnect
related_resources: []
tldr:
q: Why does the Azure Load Balancer health probe fail for the Kong proxy service when http is enabled?
a: |
On Kubernetes v1.24 and higher, the Azure Load Balancer health probe reads the protocol from `spec.ports.AppProtocol` instead of `spec.ports.Protocol`, so an HTTP-enabled Kong proxy service gets an HTTP health probe that fails and triggers repeated pod restarts. Set `proxy.http.appProtocol` (and/or `proxy.tls.appProtocol`) to `tcp` in your Helm `values.yaml` to force a TCP health probe, or point the Load Balancer's HTTP probe at Kong's status port (8100) instead.
---

## Problem

When running Kong on Kubernetes v1.24 or higher and using Azure Load Balancer, you find the health probe created by the Load Balancer fails and makes the Azure Kubernetes service restart the Kong proxy service constantly.

## Cause

The cause of the problem is a change in the health probe behavior for clusters v1.24 or higher. For clusters <=1.23, the health probe uses the protocol configured in `spec.ports.Protocol` (which is tcp) and for clusters >1.24 the health probe uses the protocol configured in `spec.ports.AppProtocol` (which is http/https). You can find more details here: Custom Load Balancer health probe

Then, when http is enabled for the Kong proxy service and Kubernetes version is >1.24, the Azure Load Balancer health probe uses HTTP and fails.

## Solution

The current Kong Helm chart exposes `appProtocol` as a first-class, conditional field directly in `values.yaml` (`proxy.http.appProtocol` / `proxy.tls.appProtocol`), so you no longer need to hand-patch `_helpers.tpl` to set it. The default value is unset (`""`), so unmodified recent installs may not trigger this bug at all; if you do hit it, set `proxy.http.appProtocol` (and/or `proxy.tls.appProtocol`) to `tcp` in your `values.yaml`. Using pure TCP fixes any problems with HTTP health probes as it removes HTTP health probes from the equation entirely.

Other alternatives are:

- Applying a Kubernetes manifest yaml instead of using Helm
- Adding a `/healthz` endpoint in the Kong proxy (file `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua`) but it requires creating a custom image.
- A custom image is not actually required to get a working HTTP health check: Kong already ships a dedicated, unauthenticated status endpoint (`status_listen`, default port 8100) that returns a plain `200` unconditionally — live-confirmed (`GET /status` on the status port returns `200` even when the plain proxy port returns `404 {"message":"no Route matched with those values"}` for the same path, since the proxy 404s on any path with no matching Route). Azure's cloud-provider-azure supports redirecting a Service port's health probe to a different port on the backend pods via the per-port annotation `service.beta.kubernetes.io/port_{port}_health-probe_port` (paired with `..._health-probe_protocol: http` and `..._health-probe_request-path: /status`), so the Load Balancer's HTTP probe can be pointed at Kong's existing status port (8100) instead of the proxy port — giving a real HTTP health check without needing `appProtocol: tcp` or a custom `/healthz` image at all.
24 changes: 24 additions & 0 deletions app/_support/can-a-global-plugin-have-exclusions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
title: Excluding specific services or routes from a global plugin
content_type: support
published: false
description: No, a globally enabled plugin applies to all traffic.
products:
- gateway
works_on:
- on-prem
- konnect
tldr:
q: Can a global plugin have exclusions?
a: |
No. A globally enabled plugin always applies to all traffic; it has no built-in exclusion mechanism. However, a plugin configured at a lower level (service or route) overrides the global plugin's configuration for that entity, so you can achieve a similar effect by adding a differently configured instance of the plugin at the service or route level.
related_resources: []
---

## Can a global plugin have exclusions?

Can a plugin enabled globally have exclusions? Scenario: I have 10 services and want a plugin to apply to 9 of them. It would be easier to enable a global plugin and add one exception / exclusion instead of applying individually to 9 services. Is this possible?

No, a globally enabled plugin applies to all traffic. However, its configuration depends on the entities it is associated with. Values from global plugins can be overridden by plugins configured at a lower level, such as on a service or route.

For more details, please review the documentation on plugin precedence .
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
title: Configuring {{site.base_gateway}} to output logs as JSON objects
content_type: support
description: "Kong doesn't have a built-in JSON `error_log` format, but JSON access logs can be produced using Kong logging plugins (`tcp-log`, `http-log`) or a custom Nginx `log_format`."
products:
- gateway
works_on:
- on-prem
- konnect
tldr:
q: Can Kong Gateway expose the logs as JSON objects?

Check failure on line 11 in app/_support/can-kong-gateway-expose-the-logs-as-json-objects.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [base.Kongterms] Use '{{site.base_gateway}}' instead of 'Kong Gateway'. Raw Output: {"message": "[base.Kongterms] Use '{{site.base_gateway}}' instead of 'Kong Gateway'.", "location": {"path": "app/_support/can-kong-gateway-expose-the-logs-as-json-objects.md", "range": {"start": {"line": 11, "column": 10}}}, "severity": "ERROR"}
a: |
Kong doesn't have a setting to change the Nginx `error_log` format, but you can get JSON access logs using the `tcp-log` or `http-log` plugins, or by defining a custom Nginx `log_format` (e.g. `json_logs`) in a custom Nginx template and setting `proxy_access_log` to use it.
related_resources:
- text: Nginx logging documentation
url: https://docs.nginx.com/nginx/admin-guide/monitoring/logging/
- text: Kong Logging plugins (tcp-log, http-log)
url: /plugins/?category=logging
- text: Customize what Kong Gateway logs

Check failure on line 19 in app/_support/can-kong-gateway-expose-the-logs-as-json-objects.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [base.Kongterms] Use '{{site.base_gateway}}' instead of 'Kong Gateway'. Raw Output: {"message": "[base.Kongterms] Use '{{site.base_gateway}}' instead of 'Kong Gateway'.", "location": {"path": "app/_support/can-kong-gateway-expose-the-logs-as-json-objects.md", "range": {"start": {"line": 19, "column": 26}}}, "severity": "ERROR"}
url: /gateway/logs/#customize-what-kong-gateway-logs
---

## Problem

{{site.base_gateway}} does not provide a built-in option to output its logs as JSON objects.

## Solution

Kong logs are based on the Nginx logging functionality.

There is no option to change the Nginx `error_log` format.

For `access_log`, there are options to use Kong Logging plugins to send JSON events using the `tcp-log` or `http-log` plugins.

Or using a custom Nginx template (`/usr/local/share/lua/5.1/kong/templates/nginx.lua`).

And using `log_format json_logs` similar to:

```nginx
log_format json_logs escape=json
'{'
'"time_local":"$time_local",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"status": "$status",'
'"body_bytes_sent":"$body_bytes_sent",'
'"request_time":"$request_time",'
'"http_referrer":"$http_referer",'
'"http_user_agent":"$http_user_agent"'
'}';
```

And then configure:

```nginx
proxy_access_log=logs/access.log json_logs
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
title: Configure Kong to handle public key rotation for consumer credentials without hardcoding the public key
content_type: support
description: The JWT plugin in Kong requires the public key to be uploaded for JWT validation.
products:
- gateway
works_on:
- on-prem
- konnect
tldr:
q: How do I configure Kong to handle public key rotation for consumer credentials without hardcoding the public key?
a: |
The JWT plugin requires the public key to be uploaded and does not support automatic key rotation. Use the OpenID Connect plugin instead: set `config.issuer` to your IdP's discovery endpoint so the plugin auto-configures from the JWKS URI and rotates keys automatically. Tune `config.rediscovery_lifetime` to control how often the JWKS is re-fetched.
related_resources: []
---

## Problem

The JWT plugin in Kong requires the public key to be uploaded for JWT validation, but this approach does not support automatic key rotation for consumer credentials.

## Solution

To handle key rotation without hardcoding the public key, you can use the OpenID Connect plugin, which supports JWKS URIs and can automatically handle key rotation.

Here are the steps to configure the OpenID Connect plugin to use JWKS URIs:

1. Configure the OpenID Connect plugin with the `config.issuer` parameter set to your Identity Provider's (IdP) discovery endpoint. This allows the plugin to auto-configure most settings and handle key rotation.

```json
{
"config.issuer": "https://<your-idp>/path/to/.well-known/openid-configuration"
}
```

2. Set the `config.rediscovery_lifetime` to specify how often the plugin should re-discover the JWKS (in seconds). This is useful when the plugin cannot find a key for verifying the signature.

```json
{
"config.rediscovery_lifetime": 30
}
```

3. Use `config.extra_jwks_uris` if you have additional JWKS URIs to specify.

```json
{
"config.extra_jwks_uris": ["https://<your-idp>/path/to/jwks.json"]
}
```

4. Test the JWKS retrieval using the OpenID Connect plugin's JWKS retrieval API call to ensure that the JWKS URI is correctly set up.

5. Check for caching behavior. When using Kong with a database, the discovery information and the JWKS are cached in the Kong configuration database. The plugin will re-discover upon failure to find the key.

6. Monitor for any authentication issues such as 401 Unauthorized errors, which may indicate a problem with the JWKS URI or the token validation process.

7. Review the debug logs if you encounter issues. The logs can provide additional information about the failure, such as signature verification problems.

8. Ensure that the `config.issuers_allowed` parameter is correctly configured if you are using it to restrict which issuers are allowed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
title: "\"Consumer was not found with access token\" error when the OIDC `consumer_claim` is missing from the token"
content_type: support
description: "This occurs when the claim defined in the OIDC `consumer_claim` field is not found in the token provided."
products:
- gateway
- kic
works_on:
- on-prem
- konnect
related_resources:
- text: OpenID Connect plugin documentation
url: /plugins/openid-connect/
tldr:
q: Why does Kong show "Consumer was not found with access token" for OIDC consumer mapping?
a: |
The OIDC plugin's `consumer_claim` config value must exactly match (case-sensitively) a claim present in the token. If the token doesn't include that claim, Kong can't map the token to a consumer and logs this error. Fix it by including the claim in the token, or by changing `consumer_claim` to a claim that already exists in the token (for example, `azp`).
---

## Problem

When consuming an OIDC protected route or service you receive an error similar to the below in the Kong error log.

The name of the claim, in this example `email`, may be different in your environment. What is causing this issue?

```
Consumer was not found with access token (claim (email) was not found for consumer mapping)
```

## Solution

This occurs when the claim defined in the OIDC `consumer_claim` field is not found in the token provided.

For example:

The OIDC config has `consumer_claim = email`

If the token provided has the below payload the error will occur as it is missing the `email` claim.

```json
{
"iss": "https://accounts.google.com",
"azp": "329599xxxxxx.apps.googleusercontent.com",
"aud": "329599xxxxxx.apps.googleusercontent.com",
"sub": "20376685496334",
"at_hash": "rN2G5jhTWjw8JEC",
"iat": 1624462383,
"exp": 1624465983
}
```

To address this you will need to either:

1. Include the claim as part of the token payload.
2. Change the `consumer_claim` to a value that exists in the token. For the above example, you could use `azp` for instance.

Note: The `consumer_claim` value is case sensitive. `Email` will not be treated the same as `email`.

More details on the OIDC plugin can be found in the OpenID Connect plugin documentation.
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: Controlling the hybrid configuration update frequency
content_type: support
description: How the `db_update_frequency` parameter controls how often {{site.base_gateway}} pushes configuration updates from the Control Plane to Data Planes in Hybrid mode.
products:
- gateway
works_on:
- on-prem
- konnect
related_resources:
- text: "`db_update_frequency` configuration reference"
url: /gateway/configuration/#db-update-frequency
tldr:
q: How can I control the frequency of configuration updates between the Control Plane and Data Plane in Kong Hybrid mode?
a: |
Set the `db_update_frequency` parameter, which controls how often Kong checks the datastore for updated entities. In Hybrid mode, the same parameter controls how frequently the Control Plane pushes configuration changes to the Data Plane, for example `db_update_frequency=30` to sync every 30 seconds.
---

## Problem

When using a Hybrid installation of Kong, how can the frequency of configuration updates between the Control Plane and Data Plane be controlled?

## Solution

When using a Classic deployment of Kong where all nodes have access to the datastore, the frequency at which the database is checked for updated entities is controlled by the `db_update_frequency` parameter.

The same parameter is used in a Hybrid deployment to control how frequently the Control Plane will push configuration changes to the Data Plane. As an example, if you only wish to synchronize the configuration to the Data Plane every 30 seconds, then set the Kong configuration like this:

```
db_update_frequency=30
```

This setting will make the Data Plane configuration push sleep for the configured `db_update_frequency` time.
Loading
Loading