diff --git a/.github/styles/base/Dictionary.txt b/.github/styles/base/Dictionary.txt index 96ad488106..e7b543c184 100644 --- a/.github/styles/base/Dictionary.txt +++ b/.github/styles/base/Dictionary.txt @@ -1008,3 +1008,37 @@ semconv rollup rollups llm_format +Autodiscovery +autostart +CVEs +Dockerized +endscript +HAProxy +hardcodes +Idletime +IDPs +Kyverno +logrotate +loopbacking +multiline +OOMKiller +OOMKilled +postrotate +recrawl +reenable +stunnel +supercookie +systemctl +tmp +iat +kong_rate_limiting_counters +log_disconnections +lokey +mem_cache_size +nginx_worker_processes +podAnnotations +prereading +server_version_num +unredact +userinfo +xyz diff --git a/app/_support/after-upgrading-to-kong-3-6-x-x-i-can-not-log-into-kong-manager-after-making.md b/app/_support/after-upgrading-to-kong-3-6-x-x-i-can-not-log-into-kong-manager-after-making.md new file mode 100644 index 0000000000..724b039e32 --- /dev/null +++ b/app/_support/after-upgrading-to-kong-3-6-x-x-i-can-not-log-into-kong-manager-after-making.md @@ -0,0 +1,51 @@ +--- +title: After upgrading to Kong 3.10.0.0 I can not log into Kong Manager after making documented changes +content_type: support +description: Kong 3.10.x moved Kong Manager session handling to the openid-connect plugin, which can produce a session cookie header larger than Nginx's default proxy buffer, causing login to fail with an upstream error. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why can't I log into Kong Manager after upgrading to Kong 3.10.0.0 and configuring openid-connect authentication? + a: | + Kong 3.10.x moved Kong Manager session handling to the openid-connect plugin, which returns a session cookie header that can be larger than in previous versions — often larger than Nginx's default header buffer when proxied through Kong, causing an "invalid response from upstream" error. Increase `nginx_proxy_proxy_buffer_size` (try 8k, 10k, 12k, or 16k) to fix it; note this is a global Nginx setting that affects all traffic through the same data plane. +related_resources: [] +--- + +## Problem + +While upgrading to Kong 3.10.0.0 we changed the Kong Manager configuration according to the document requirements when using openid-connect authentication for the Manager. + +We are proxying all traffic to the Kong Manager service through Kong, and after the changes made, we can still not log into Kong Manager but get the following error in the Browser: + +``` +Error + +An invalid response was received from the upstream server. + +request_id: 8150ea003c03816da5ecc51920993417 +``` + +Are we missing a configuration option? + +## Cause + +The changes regarding openid-connect authentication in Kong Manager in version 3.10.X.X of Kong mean that the session handling is now done via the openid-connect plugins. This means the plugin will initiate the creation of a session cookie as documented in the openid-connect plugin documentation. + +With this new configuration a session cookie header will be sent back to the Browser client which depending on what information is returned by the IdP in the tokens it generates during the login process can be much larger than the session data that was generated in previous versions when using the default session object. + +If you are proxying your traffic through Kong, this may breach the default header limit, and you should see an error log entry similar to this: + +``` +2025/03/12 12:34:49 [error] 2385#0: *4019 upstream sent too big header while reading response header from upstream, client: 192.168.1.4, server: kong, request: "GET /auth?code=&state=&session_state=:session_state>", host: "api.kong.lan", referrer: "https://login.live.com/", request_id: "8150ea003c03816da5ecc51920993417" +``` + +## Solution + +To address this issue, you will have to increase the `nginx_proxy_proxy_buffer_size` setting. + +Since the size of the cookie headers depends on what information the IdP returns in the tokens, you may have to try different values for the `nginx_proxy_proxy_buffer_size` starting with 8k, 10k, 12k or 16k. + +Please note that this setting is a global setting which means all requests that are proxied through the same Kong data plane as the Kong Manager traffic will be affected by this, i.e. traffic with larger headers will be allowed. diff --git a/app/_support/authenticated-group-mapping-using-groups-claim-using-does-not-work-with-azure-id.md b/app/_support/authenticated-group-mapping-using-groups-claim-using-does-not-work-with-azure-id.md new file mode 100644 index 0000000000..3dc432894c --- /dev/null +++ b/app/_support/authenticated-group-mapping-using-groups-claim-using-does-not-work-with-azure-id.md @@ -0,0 +1,25 @@ +--- +title: OIDC authenticated group mapping fails when using the `groups` claim with Azure AD +content_type: support +description: "With Azure AD as the IdP, use the `roles` claim (not `groups`) as the `authenticated_groups_claim` so RBAC role mapping resolves correctly, since the `groups` claim only contains UUIDs." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why doesn't OIDC authenticated group mapping work with the `groups` claim in Azure AD? + a: | + Azure AD's `groups` claim returns UUIDs, so mapping `authenticated_groups_claim` to `groups` won't match your RBAC role names. Use the `roles` claim instead, and assign string values in the format `:` (e.g. `default:super-admin`) to the Azure AD app roles. If mapping still doesn't work, enable `debug` log level on Kong and check log entries prefixed with `[openid-connect]`. +--- + +## Problem + +We are trying to set up openid-connect authentication in Kong Manager with OIDC Authenticated Group Mapping with Azure AD using `authenticated_groups_claim` set to `groups`, but since the `groups` claim only contains UUIDs, this does not lead to the user logging in getting the correct RBAC role assigned. How can we fix this? + +## Solution + +With Azure AD as the IdP, you should not use `groups` as the `authenticated_groups_claim`, but use `roles` instead. When setting up roles in Azure AD, you can assign string values to the `roles` claim that match the requirement necessary for the group mapping to work. As per documentation, the claim needs to have string values of the format `:`, e.g. `default:super-admin`, for mappings to be successful. + +If you have tried setting roles in the way that should work as per the Kong documentation but still do not see the correct role assignment when logging in, enable `debug` log level on Kong, and check the log entries prefixed with `[openid-connect]` to see what is not working correctly during the login process. diff --git a/app/_support/azure-load-balancer-health-probe-fails-for-kong-proxy-service.md b/app/_support/azure-load-balancer-health-probe-fails-for-kong-proxy-service.md new file mode 100644 index 0000000000..79df5f660e --- /dev/null +++ b/app/_support/azure-load-balancer-health-probe-fails-for-kong-proxy-service.md @@ -0,0 +1,35 @@ +--- +title: Azure Load Balancer health probe fails for Kong proxy service +content_type: support +description: When running Kong on Kubernetes v1.24 or higher with Azure Load Balancer, the health probe created by the Load Balancer fails and causes the Azure Kubernetes service to restart the Kong proxy service repeatedly, typically when HTTP is enabled for the proxy service. +products: + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does the Azure Load Balancer health probe fail for the Kong proxy service when http is enabled? + a: | + On Kubernetes v1.24 and higher, the Azure Load Balancer health probe reads the protocol from `spec.ports.AppProtocol` instead of `spec.ports.Protocol`, so an HTTP-enabled Kong proxy service gets an HTTP health probe that fails and triggers repeated pod restarts. Set `proxy.http.appProtocol` (and/or `proxy.tls.appProtocol`) to `tcp` in your Helm `values.yaml` to force a TCP health probe, or point the Load Balancer's HTTP probe at Kong's status port (8100) instead. +--- + +## Problem + +When running Kong on Kubernetes v1.24 or higher and using Azure Load Balancer, you find the health probe created by the Load Balancer fails and makes the Azure Kubernetes service restart the Kong proxy service constantly. + +## Cause + +The cause of the problem is a change in the health probe behavior for clusters v1.24 or higher. For clusters <=1.23, the health probe uses the protocol configured in `spec.ports.Protocol` (which is tcp) and for clusters >1.24 the health probe uses the protocol configured in `spec.ports.AppProtocol` (which is http/https). You can find more details here: Custom Load Balancer health probe + +Then, when http is enabled for the Kong proxy service and Kubernetes version is >1.24, the Azure Load Balancer health probe uses HTTP and fails. + +## Solution + +The current Kong Helm chart exposes `appProtocol` as a first-class, conditional field directly in `values.yaml` (`proxy.http.appProtocol` / `proxy.tls.appProtocol`), so you no longer need to hand-patch `_helpers.tpl` to set it. The default value is unset (`""`), so unmodified recent installs may not trigger this bug at all; if you do hit it, set `proxy.http.appProtocol` (and/or `proxy.tls.appProtocol`) to `tcp` in your `values.yaml`. Using pure TCP fixes any problems with HTTP health probes as it removes HTTP health probes from the equation entirely. + +Other alternatives are: + +- Applying a Kubernetes manifest yaml instead of using Helm +- Adding a `/healthz` endpoint in the Kong proxy (file `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua`) but it requires creating a custom image. +- A custom image is not actually required to get a working HTTP health check: Kong already ships a dedicated, unauthenticated status endpoint (`status_listen`, default port 8100) that returns a plain `200` unconditionally — live-confirmed (`GET /status` on the status port returns `200` even when the plain proxy port returns `404 {"message":"no Route matched with those values"}` for the same path, since the proxy 404s on any path with no matching Route). Azure's cloud-provider-azure supports redirecting a Service port's health probe to a different port on the backend pods via the per-port annotation `service.beta.kubernetes.io/port_{port}_health-probe_port` (paired with `..._health-probe_protocol: http` and `..._health-probe_request-path: /status`), so the Load Balancer's HTTP probe can be pointed at Kong's existing status port (8100) instead of the proxy port — giving a real HTTP health check without needing `appProtocol: tcp` or a custom `/healthz` image at all. diff --git a/app/_support/can-a-global-plugin-have-exclusions.md b/app/_support/can-a-global-plugin-have-exclusions.md new file mode 100644 index 0000000000..1fed6b2570 --- /dev/null +++ b/app/_support/can-a-global-plugin-have-exclusions.md @@ -0,0 +1,24 @@ +--- +title: Excluding specific services or routes from a global plugin +content_type: support +published: false +description: No, a globally enabled plugin applies to all traffic. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Can a global plugin have exclusions? + a: | + No. A globally enabled plugin always applies to all traffic; it has no built-in exclusion mechanism. However, a plugin configured at a lower level (service or route) overrides the global plugin's configuration for that entity, so you can achieve a similar effect by adding a differently configured instance of the plugin at the service or route level. +related_resources: [] +--- + +## Can a global plugin have exclusions? + +Can a plugin enabled globally have exclusions? Scenario: I have 10 services and want a plugin to apply to 9 of them. It would be easier to enable a global plugin and add one exception / exclusion instead of applying individually to 9 services. Is this possible? + +No, a globally enabled plugin applies to all traffic. However, its configuration depends on the entities it is associated with. Values from global plugins can be overridden by plugins configured at a lower level, such as on a service or route. + +For more details, please review the documentation on plugin precedence . diff --git a/app/_support/can-kong-gateway-expose-the-logs-as-json-objects.md b/app/_support/can-kong-gateway-expose-the-logs-as-json-objects.md new file mode 100644 index 0000000000..fbb69969d5 --- /dev/null +++ b/app/_support/can-kong-gateway-expose-the-logs-as-json-objects.md @@ -0,0 +1,58 @@ +--- +title: Configuring {{site.base_gateway}} to output logs as JSON objects +content_type: support +description: "Kong doesn't have a built-in JSON `error_log` format, but JSON access logs can be produced using Kong logging plugins (`tcp-log`, `http-log`) or a custom Nginx `log_format`." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Can {{site.base_gateway}} expose the logs as JSON objects? + a: | + Kong doesn't have a setting to change the Nginx `error_log` format, but you can get JSON access logs using the `tcp-log` or `http-log` plugins, or by defining a custom Nginx `log_format` (e.g. `json_logs`) in a custom Nginx template and setting `proxy_access_log` to use it. +related_resources: + - text: Nginx logging documentation + url: https://docs.nginx.com/nginx/admin-guide/monitoring/logging/ + - text: Kong Logging plugins (tcp-log, http-log) + url: /plugins/?category=logging + - text: Customize what {{site.base_gateway}} logs + url: /gateway/logs/#customize-what-kong-gateway-logs +--- + +## Problem + +{{site.base_gateway}} does not provide a built-in option to output its logs as JSON objects. + +## Solution + +Kong logs are based on the Nginx logging functionality. + +There is no option to change the Nginx `error_log` format. + +For `access_log`, there are options to use Kong Logging plugins to send JSON events using the `tcp-log` or `http-log` plugins. + +Or using a custom Nginx template (`/usr/local/share/lua/5.1/kong/templates/nginx.lua`). + +And using `log_format json_logs` similar to: + +```nginx +log_format json_logs escape=json +'{' +'"time_local":"$time_local",' +'"remote_addr":"$remote_addr",' +'"remote_user":"$remote_user",' +'"request":"$request",' +'"status": "$status",' +'"body_bytes_sent":"$body_bytes_sent",' +'"request_time":"$request_time",' +'"http_referrer":"$http_referer",' +'"http_user_agent":"$http_user_agent"' +'}'; +``` + +And then configure: + +```nginx +proxy_access_log=logs/access.log json_logs +``` diff --git a/app/_support/configure-kong-to-handle-public-key-rotation-for-consumer-credentials-without-hardcoding-the-public-key.md b/app/_support/configure-kong-to-handle-public-key-rotation-for-consumer-credentials-without-hardcoding-the-public-key.md new file mode 100644 index 0000000000..83bcabf041 --- /dev/null +++ b/app/_support/configure-kong-to-handle-public-key-rotation-for-consumer-credentials-without-hardcoding-the-public-key.md @@ -0,0 +1,59 @@ +--- +title: Configure Kong to handle public key rotation for consumer credentials without hardcoding the public key +content_type: support +description: The JWT plugin in Kong requires the public key to be uploaded for JWT validation. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I configure Kong to handle public key rotation for consumer credentials without hardcoding the public key? + a: | + The JWT plugin requires the public key to be uploaded and does not support automatic key rotation. Use the OpenID Connect plugin instead: set `config.issuer` to your IdP's discovery endpoint so the plugin auto-configures from the JWKS URI and rotates keys automatically. Tune `config.rediscovery_lifetime` to control how often the JWKS is re-fetched. +related_resources: [] +--- + +## Problem + +The JWT plugin in Kong requires the public key to be uploaded for JWT validation, but this approach does not support automatic key rotation for consumer credentials. + +## Solution + +To handle key rotation without hardcoding the public key, you can use the OpenID Connect plugin, which supports JWKS URIs and can automatically handle key rotation. + +Here are the steps to configure the OpenID Connect plugin to use JWKS URIs: + +1. Configure the OpenID Connect plugin with the `config.issuer` parameter set to your Identity Provider's (IdP) discovery endpoint. This allows the plugin to auto-configure most settings and handle key rotation. + + ```json + { + "config.issuer": "https:///path/to/.well-known/openid-configuration" + } + ``` + +2. Set the `config.rediscovery_lifetime` to specify how often the plugin should re-discover the JWKS (in seconds). This is useful when the plugin cannot find a key for verifying the signature. + + ```json + { + "config.rediscovery_lifetime": 30 + } + ``` + +3. Use `config.extra_jwks_uris` if you have additional JWKS URIs to specify. + + ```json + { + "config.extra_jwks_uris": ["https:///path/to/jwks.json"] + } + ``` + +4. Test the JWKS retrieval using the OpenID Connect plugin's JWKS retrieval API call to ensure that the JWKS URI is correctly set up. + +5. Check for caching behavior. When using Kong with a database, the discovery information and the JWKS are cached in the Kong configuration database. The plugin will re-discover upon failure to find the key. + +6. Monitor for any authentication issues such as 401 Unauthorized errors, which may indicate a problem with the JWKS URI or the token validation process. + +7. Review the debug logs if you encounter issues. The logs can provide additional information about the failure, such as signature verification problems. + +8. Ensure that the `config.issuers_allowed` parameter is correctly configured if you are using it to restrict which issuers are allowed. diff --git a/app/_support/consumer-was-not-found-with-access-token-claim-email-was-not-found-for-consumer-mapping.md b/app/_support/consumer-was-not-found-with-access-token-claim-email-was-not-found-for-consumer-mapping.md new file mode 100644 index 0000000000..b1a0876c21 --- /dev/null +++ b/app/_support/consumer-was-not-found-with-access-token-claim-email-was-not-found-for-consumer-mapping.md @@ -0,0 +1,59 @@ +--- +title: "\"Consumer was not found with access token\" error when the OIDC `consumer_claim` is missing from the token" +content_type: support +description: "This occurs when the claim defined in the OIDC `consumer_claim` field is not found in the token provided." +products: + - gateway + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: OpenID Connect plugin documentation + url: /plugins/openid-connect/ +tldr: + q: Why does Kong show "Consumer was not found with access token" for OIDC consumer mapping? + a: | + The OIDC plugin's `consumer_claim` config value must exactly match (case-sensitively) a claim present in the token. If the token doesn't include that claim, Kong can't map the token to a consumer and logs this error. Fix it by including the claim in the token, or by changing `consumer_claim` to a claim that already exists in the token (for example, `azp`). +--- + +## Problem + +When consuming an OIDC protected route or service you receive an error similar to the below in the Kong error log. + +The name of the claim, in this example `email`, may be different in your environment. What is causing this issue? + +``` +Consumer was not found with access token (claim (email) was not found for consumer mapping) +``` + +## Solution + +This occurs when the claim defined in the OIDC `consumer_claim` field is not found in the token provided. + +For example: + +The OIDC config has `consumer_claim = email` + +If the token provided has the below payload the error will occur as it is missing the `email` claim. + +```json +{ + "iss": "https://accounts.google.com", + "azp": "329599xxxxxx.apps.googleusercontent.com", + "aud": "329599xxxxxx.apps.googleusercontent.com", + "sub": "20376685496334", + "at_hash": "rN2G5jhTWjw8JEC", + "iat": 1624462383, + "exp": 1624465983 +} +``` + +To address this you will need to either: + +1. Include the claim as part of the token payload. +2. Change the `consumer_claim` to a value that exists in the token. For the above example, you could use `azp` for instance. + +Note: The `consumer_claim` value is case sensitive. `Email` will not be treated the same as `email`. + +More details on the OIDC plugin can be found in the OpenID Connect plugin documentation. diff --git a/app/_support/controlling-the-hybrid-configuration-update-frequency.md b/app/_support/controlling-the-hybrid-configuration-update-frequency.md new file mode 100644 index 0000000000..87087f9669 --- /dev/null +++ b/app/_support/controlling-the-hybrid-configuration-update-frequency.md @@ -0,0 +1,33 @@ +--- +title: Controlling the hybrid configuration update frequency +content_type: support +description: How the `db_update_frequency` parameter controls how often {{site.base_gateway}} pushes configuration updates from the Control Plane to Data Planes in Hybrid mode. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "`db_update_frequency` configuration reference" + url: /gateway/configuration/#db-update-frequency +tldr: + q: How can I control the frequency of configuration updates between the Control Plane and Data Plane in Kong Hybrid mode? + a: | + Set the `db_update_frequency` parameter, which controls how often Kong checks the datastore for updated entities. In Hybrid mode, the same parameter controls how frequently the Control Plane pushes configuration changes to the Data Plane, for example `db_update_frequency=30` to sync every 30 seconds. +--- + +## Problem + +When using a Hybrid installation of Kong, how can the frequency of configuration updates between the Control Plane and Data Plane be controlled? + +## Solution + +When using a Classic deployment of Kong where all nodes have access to the datastore, the frequency at which the database is checked for updated entities is controlled by the `db_update_frequency` parameter. + +The same parameter is used in a Hybrid deployment to control how frequently the Control Plane will push configuration changes to the Data Plane. As an example, if you only wish to synchronize the configuration to the Data Plane every 30 seconds, then set the Kong configuration like this: + +``` +db_update_frequency=30 +``` + +This setting will make the Data Plane configuration push sleep for the configured `db_update_frequency` time. diff --git a/app/_support/database-already-bootstrapped-messages-when-upgrading-kong-with-helm.md b/app/_support/database-already-bootstrapped-messages-when-upgrading-kong-with-helm.md new file mode 100644 index 0000000000..60eea6e46d --- /dev/null +++ b/app/_support/database-already-bootstrapped-messages-when-upgrading-kong-with-helm.md @@ -0,0 +1,63 @@ +--- +title: "\"Database already bootstrapped\" messages when upgrading Kong with Helm" +content_type: support +description: Reinstalling (rather than upgrading) the Kong Helm chart causes the init-migrations job to report "Database already bootstrapped" because the database has already been bootstrapped by a prior install; follow the Helm upgrade procedure instead. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does Kong's Helm upgrade report "Database already bootstrapped"? + a: | + This happens when the Kong Helm chart was uninstalled and reinstalled instead of upgraded — `helm install` always runs `kong migrations bootstrap`, which fails against a database that a previous install already bootstrapped. Use `helm upgrade` instead, which runs `kong migrations up` and `kong migrations finish`, after bumping the image tag in `values.yaml`. +related_resources: [] +--- + +## Problem + +When attempting to upgrade Kong using Helm, the `kong-init-migrations` pod reports: + +``` +Database already bootstrapped +``` + +Additionally, reviewing the Kong pod logs will show: + +``` +Run with --v (verbose) or --vv (debug) for more details +waiting for db +Error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:30: New migrations available; run 'kong migrations up' to proceed +``` + +## Cause + +This issue occurs when the chart is uninstalled and reinstalled rather than upgraded. + +## Solution + +When upgrading via Helm you should follow the below procedure. + +1. Change the version of Kong in the `values.yaml` to the desired target. For example, if you are upgrading from 3.10.0.0 to 3.14.0.0 you should change + + ```yaml + image: + repository: kong-docker-kong-enterprise-edition-docker.bintray.io/kong-enterprise-edition + tag: 3.10.0.0 + ``` + + to + + ```yaml + image: + repository: kong-docker-kong-enterprise-edition-docker.bintray.io/kong-enterprise-edition + tag: 3.14.0.0 + ``` + +2. Upgrade the chart + + ```bash + helm upgrade my-kong kong/kong -n kong --values ./values.yaml + ``` + +When `helm install` is run this causes the installation to run `kong migrations bootstrap`, whereas running `helm upgrade` will run `kong migrations up` and `kong migrations finish`. diff --git a/app/_support/deck-session-cookies-are-not-valid-for-the-configured-lifetime.md b/app/_support/deck-session-cookies-are-not-valid-for-the-configured-lifetime.md new file mode 100644 index 0000000000..d015a5575a --- /dev/null +++ b/app/_support/deck-session-cookies-are-not-valid-for-the-configured-lifetime.md @@ -0,0 +1,58 @@ +--- +title: Deck session cookies are not valid for the configured lifetime +content_type: support +description: The session cookie used by `deck ping` expires before the configured `cookie_lifetime`; increasing `cookie_discard` keeps the original cookie valid for its full lifetime. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does the session cookie expire before the configured `cookie_lifetime`? + a: | + Each `deck ping` call generates a new session cookie on the Kong server, so two valid cookies briefly exist. The original cookie is discarded after `cookie_discard` (default 10 seconds), which is shorter than `cookie_lifetime`, and `deck` has no way to retrieve the updated cookie. Increase `cookie_discard` in `admin_gui_session_conf` to match `cookie_lifetime` so the original cookie stays valid for its full lifetime. +related_resources: [] +--- + +## Problem + +The `admin_gui_session_conf` has been set to have a `cookie_lifetime` of 600 seconds: + +```bash +admin_gui_session_conf={ "cookie_name": "manager-session", "secret": "this_is_my_other_secret", "storage": "kong", "cookie_secure":true, "cookie_lifetime":600}" +``` + +An authentication cookie is retrieved: + +```bash +curl -k https://api.kong.lan/auth -u "kong_admin:password" -H "kong-admin-user:kong_admin" -c /tmp/cookie +``` + +and `deck ping` initially works fine using the cookie: + +```bash +deck ping --kong-cookie-jar-path /tmp/cookie --tls-skip-verify -w default --kong-addr https://api.kong.lan --headers kong-admin-user:kong_admin --headers "user-agent:curl/7.82.0" +Successfully connected to Kong! +Kong version: 3.14.0.0-enterprise-edition +``` + +After waiting approximately 10 seconds, the same `deck ping` command fails: + +```bash +deck ping --kong-cookie-jar-path /tmp/cookie --tls-skip-verify -w default --kong-addr https://api.kong.lan --headers kong-admin-user:kong_admin --headers "user-agent:curl/7.82.0" +Error: reading Kong version: HTTP status 401 (message: "Unauthorized") +``` + +The session cookie expires before the configured `cookie_lifetime` of 600 seconds elapses. + +## Cause + +When the `deck ping` command is run, a new session cookie is generated on the Kong server. This means that there are now two valid cookies. The original, older cookie is discarded after the `cookie_discard` time, which defaults to 10 seconds. + +## Solution + +Currently, there is no way for `deck` to retrieve the updated session cookie and it will be necessary to increase the `cookie_discard` time to allow the original cookie to be used for the full lifetime. For example, change the `admin_gui_session_conf` as per the example below by adding the `cookie_discard` parameter: + +```bash +admin_gui_session_conf={ "cookie_name": "manager-session", "secret": "this_is_my_other_secret", "storage": "kong", "cookie_secure":true, "cookie_lifetime":600, "cookie_discard":600}" +``` diff --git a/app/_support/dns-resolution-failed-when-setting-http-proxy-for-openid-connect-plugin-behind-proxy.md b/app/_support/dns-resolution-failed-when-setting-http-proxy-for-openid-connect-plugin-behind-proxy.md new file mode 100644 index 0000000000..6dafda6435 --- /dev/null +++ b/app/_support/dns-resolution-failed-when-setting-http-proxy-for-openid-connect-plugin-behind-proxy.md @@ -0,0 +1,49 @@ +--- +title: DNS resolution failed when setting `http_proxy` for the OpenID Connect plugin behind a proxy +content_type: support +description: The `config.http_proxy` and `config.https_proxy` parameters refer to the protocol of the IdP's URL, not the protocol between Kong and the proxy. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does the OpenID Connect plugin fail discovery with a DNS resolution error when `http_proxy` is configured? + a: | + `config.http_proxy` and `config.https_proxy` refer to the protocol of the IdP's discovery URL, not the protocol Kong uses to reach the proxy itself. Set `config.https_proxy` (not `http_proxy`) whenever the IdP's endpoint is `https://`, even if Kong reaches the proxy over plain HTTP. +related_resources: [] +--- + +## Problem + +The user is running Kong in an internal environment and must go through a proxy to access OIDC IdP URLs as below: + +When setting up `http_proxy` to proxy as `config.http_proxy: :` in the OIDC plugin, the user gets DNS resolution errors as below: + +``` + +[openid-connect] loading configuration for https:///auth/realms/demo using discovery, client: 172.26.0.4, server: kong, request: "GET /test HTTP/1.1", host: "kong-ee:8000" + +2026/05/07 02:13:20 [notice] 25#0: *23541 [lua] cache.lua:258: discover(): [openid-connect] loading configuration for https:///auth/realms/demo using discovery failed: [cosocket] DNS resolution failed: dns lookup pool exceeded retries (1): timeout. Tried: ["(short):(na) - cache-miss",":33 - cache-miss/scheduled/querying/try 1 error: timeout/scheduled/querying/try 2 error: timeout/dns lookup pool exceeded retries (1): timeout",":1 - cache-miss/scheduled/querying/try 1 error: timeout/scheduled/querying/try 2 error: timeout/dns lookup pool exceeded retries (1): timeout",":5 - cache-miss/scheduled/querying/try 1 error: timeout/scheduled/querying/try 2 error: timeout/dns lookup pool exceeded retries (1): timeout"] (falling back to previous configuration), client: 172.26.0.4, server: kong, request: "GET /test HTTP/1.1", host: "kong-ee:8000" +``` + +## Cause + +Let's check the description of these two parameters: + +- `http_proxy`: The proxy URL for HTTP communications. +- `https_proxy`: The proxy URL for HTTPS communications. + +These two parameters are not referring to the http or https protocol between Kong and the proxy, they are referring to the http or https protocol of your IdP URL. + +## Solution + +For example, if your IdP's discovery endpoint is `https:///.well-known/openid-configuration`, even though Kong is talking to your proxy via http, you need to set `config.https_proxy=http://:`. + +Once the right parameter is used, you should see OIDC loading the discovery information correctly. + +``` + +2026/05/07 12:43:57 [debug] 25#0: *2468 [lua] handler.lua:96: [openid-connect] loading discovery information +2026/05/07 12:43:57 [debug] 25#0: *2468 [lua] handler.lua:139: [openid-connect] initializing library +``` diff --git a/app/_support/do-upstreams-in-kong-support-tls-v1-3.md b/app/_support/do-upstreams-in-kong-support-tls-v1-3.md new file mode 100644 index 0000000000..32017606fb --- /dev/null +++ b/app/_support/do-upstreams-in-kong-support-tls-v1-3.md @@ -0,0 +1,40 @@ +--- +title: TLS v1.3 support for connecting to Upstreams in Kong +content_type: support +description: Support for Upstream TLS v1.3 is possible but is not enabled by default. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Do Upstreams in Kong support TLS v1.3? + a: | + Yes. Kong's bundled OpenResty already includes TLS 1.3 in its default protocol set, so Upstream connections negotiate it without any `proxy_ssl_protocols` override. Use `proxy_ssl_protocols` only if you need to restrict which protocol versions Kong negotiates with an Upstream. +related_resources: + - text: nginx `proxy_ssl_protocols` directive documentation + url: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_protocols +--- + +## Problem + +When connecting to an Upstream that only supports TLS v1.3 an error is returned for the protocol version and the handshake fails; + +``` + +*3933522 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 172.17.0.1, server: kong, request: "GET /test-v13 HTTP/1.1", upstream: "https://10.0.1.1:8888/", host: "api.kong.lan" +``` + +Is TLS v1.3 supported for connecting to Upstreams? + +## Solution + +TLS v1.3 to Upstreams works by default. Kong's bundled OpenResty (1.27.1.2 as of {{site.base_gateway}} 3.14.0.0) already includes TLS 1.3 in its own default protocol set, so no `proxy_ssl_protocols` override is needed to enable it, and Kong's own templates never set `proxy_ssl_protocols` themselves. + +If you previously added a `proxy_ssl_protocols` override to work around this, you can remove it. The `proxy_ssl_protocols` nginx directive is still useful, but only if you need to *restrict* which protocol versions Kong will negotiate with an Upstream (for example, to disallow older TLS versions). + +To set this in Kong, you can inject the proxy directive directly. For example, in the `kong.conf` add this parameter (changing the values list as appropriate to restrict to only the protocol versions you want to allow); + +```bash +nginx_proxy_proxy_ssl_protocols = TLSv1.2 TLSv1.3 +``` diff --git a/app/_support/does-kong-support-http2-protocol.md b/app/_support/does-kong-support-http2-protocol.md new file mode 100644 index 0000000000..0706214a41 --- /dev/null +++ b/app/_support/does-kong-support-http2-protocol.md @@ -0,0 +1,165 @@ +--- +title: http2 protocol support for client connections in Kong +content_type: support +description: "Kong supports http2 client connections through nginx: by default over TLS ports and via HTTP/2 prior knowledge on plain ports." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Does Kong support http2 protocol? + a: | + Yes. Kong's http2 support follows nginx's, and the default `proxy_listen` for the HTTPS proxy port already includes the `http2` suffix, so http2 client connections work out of the box over TLS. Plain (non-TLS) http2 connections only work when the client uses HTTP/2 prior knowledge, since nginx doesn't negotiate a protocol upgrade on non-TLS ports. +related_resources: + - text: "{{site.base_gateway}} `proxy_listen` configuration reference" + url: /gateway/configuration/#proxy-listen + - text: RFC 7540 section 3.4 (starting HTTP/2 with prior knowledge) + url: https://datatracker.ietf.org/doc/html/rfc7540#section-3.4 + - text: RFC 7540 known-http2 reference (httpwg.org mirror) + url: https://httpwg.org/specs/rfc7540.html#known-http +--- + +## Problem + +Whether Kong supports client connections using the http2 protocol version. + +## Solution + +As Kong is using nginx, support for http2 follows nginx support. The default shipped `proxy_listen` for the SSL proxy port already includes the `http2` suffix (`0.0.0.0:8443 http2 ssl`), so http2 client connections are supported out of the box on the default HTTPS proxy port with no configuration changes needed. A manual `http2` suffix is only required if you override `proxy_listen` with a fully custom listen definition. + +If you use a fully custom `proxy_listen` override, remember to add the `http2` suffix to the listen port definition yourself, as below; + +```bash +proxy_listen="0.0.0.0:4800, 0.0.0.0:8443 http2 ssl" +``` + +You can then make both http1.x and http2 client connections to the proxy port. + +1) Using http1.1 on the https port + +```bash + +# curl -k --http1.1 -v https://0.0.0.0:8443/httpbin/anything +* Trying 0.0.0.0:8443... +* Connected to 0.0.0.0 (127.0.0.1) port 8443 (#0) +* ALPN, offering http/1.1 +* successfully set certificate verify locations: +* CAfile: /etc/ssl/certs/ca-certificates.crt +* CApath: none +* TLSv1.3 (OUT), TLS handshake, Client hello (1): +* TLSv1.3 (IN), TLS handshake, Server hello (2): +* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): +* TLSv1.3 (IN), TLS handshake, Certificate (11): +* TLSv1.3 (IN), TLS handshake, CERT verify (15): +* TLSv1.3 (IN), TLS handshake, Finished (20): +* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): +* TLSv1.3 (OUT), TLS handshake, Finished (20): +* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 +* ALPN, server accepted to use http/1.1 +* Server certificate: +<...snip certificate details...> +> GET /httpbin/anything HTTP/1.1 +> Host: 0.0.0.0:48443 +> User-Agent: curl/7.79.1 +> Accept: */* +> +* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): +* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): +* old SSL session ID is stale, removing +* Mark bundle as not supporting multiuse +< HTTP/1.1 200 OK +``` + +2) Using http2 on the https port + +```bash + +# curl -k --http2 -v https://0.0.0.0:8443/httpbin/anything +* Trying 0.0.0.0:8443... +* Connected to 0.0.0.0 (127.0.0.1) port 8443 (#0) +* ALPN, offering h2 +* ALPN, offering http/1.1 +* successfully set certificate verify locations: +* CAfile: /etc/ssl/certs/ca-certificates.crt +* CApath: none +* TLSv1.3 (OUT), TLS handshake, Client hello (1): +* TLSv1.3 (IN), TLS handshake, Server hello (2): +* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): +* TLSv1.3 (IN), TLS handshake, Certificate (11): +* TLSv1.3 (IN), TLS handshake, CERT verify (15): +* TLSv1.3 (IN), TLS handshake, Finished (20): +* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): +* TLSv1.3 (OUT), TLS handshake, Finished (20): +* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 +* ALPN, server accepted to use h2 +* Server certificate: +<...snip certificate details...> +* Using HTTP2, server supports multiplexing +* Connection state changed (HTTP/2 confirmed) +* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 +* Using Stream ID: 1 (easy handle 0x7fd4eb983a90) +> GET /httpbin/anything HTTP/2 +> Host: 0.0.0.0:48443 +> user-agent: curl/7.79.1 +> accept: */* +> +* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): +* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): +* old SSL session ID is stale, removing +* Connection state changed (MAX_CONCURRENT_STREAMS == 128)! +< HTTP/2 200 +``` + +Support is also available for http2 on plain listener ports via the same configuration suffix; + +```bash +proxy_listen="0.0.0.0:4800 http2, 0.0.0.0:8443 http2 ssl" +``` + +Note, there is a limitation of nginx that plain http2 connections are only supported with "prior knowledge". When a client attempts an http2 connection on a plain (non-TLS) port without prior knowledge, the request actually starts as a normal http1.1 request carrying an `Upgrade: h2c` header. Kong/nginx does not switch protocols for this request: it silently ignores the Upgrade header and serves the request as an ordinary HTTP/1.1 response instead of upgrading to http2 - it does not reject the request or return a protocol error. Depending on the http client, this can show up as a perfectly normal HTTP/1.1 200 response (confirmed with curl 8.7.1) or, on some older http client versions, as a client-side connection error instead (older curl releases could mishandle the unexpected non-101 response and abort with an error such as `Received HTTP/0.9 when not allowed`). Either way, the connection is never actually upgraded to real http2; + +```bash + +# curl --http2 -v http://0.0.0.0:8000/httpbin/anything +* Trying 0.0.0.0:8000... +* Connected to 0.0.0.0 (127.0.0.1) port 8000 (#0) +> GET /httpbin/anything HTTP/1.1 +> Host: 0.0.0.0:8000 +> User-Agent: curl/8.7.1 +> Accept: */* +> Connection: Upgrade, HTTP2-Settings +> Upgrade: h2c +> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA +> +* Request completely sent off +< HTTP/1.1 200 OK +< Content-Type: application/json; charset=utf-8 +< Connection: keep-alive +< +* Connection #0 to host 0.0.0.0 left intact +``` + +Without using prior knowledge, the plain-http2 connection attempt never actually becomes http2 - nginx only supports one protocol version on plain (non-TLS) ports, and it simply falls back to serving the request as HTTP/1.1. You can test a genuine prior-knowledge http2 request using curl and the --http2-prior-knowledge flag; + +```bash + +# curl --http2 --http2-prior-knowledge -v http://0.0.0.0:8000/httpbin/anything +* Trying 0.0.0.0:8000... +* Connected to 0.0.0.0 (127.0.0.1) port 8000 (#0) +* Using HTTP2, server supports multiplexing +* Connection state changed (HTTP/2 confirmed) +* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 +* Using Stream ID: 1 (easy handle 0x7f22c0691a90) +> GET /httpbin/anything HTTP/2 +> Host: 0.0.0.0:8000 +> user-agent: curl/7.79.1 +> accept: */* +> +* Connection state changed (MAX_CONCURRENT_STREAMS == 128)! +< HTTP/2 200 +``` + +Below are a couple of RFC links with details of prior knowledge. + +To summarize support for http2, you will have to either use https connections where both http1 and http2 are supported on the same socket via ALPN negotiation or use prior knowledge for the plain socket connections. diff --git a/app/_support/enabling-ldap-for-kong-manager-failed-with-an-error-invalid-primary-key-id-expected-a-valid-uuid.md b/app/_support/enabling-ldap-for-kong-manager-failed-with-an-error-invalid-primary-key-id-expected-a-valid-uuid.md new file mode 100644 index 0000000000..031eef052e --- /dev/null +++ b/app/_support/enabling-ldap-for-kong-manager-failed-with-an-error-invalid-primary-key-id-expected-a-valid-uuid.md @@ -0,0 +1,59 @@ +--- +title: "Enabling LDAP for Kong Manager failed with an error; invalid primary key: '{id=\"expected a valid UUID\"}'" +content_type: support +description: "LDAP authentication for Kong Manager fails with an `invalid primary key` error when the LDAP user falls back to the anonymous consumer, which Kong Manager doesn't permit." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: "Why does enabling LDAP for Kong Manager fail with an \"invalid primary key: expected a valid UUID\" error?" + a: | + LDAP authentication falls back to the anonymous consumer when it fails, and Kong Manager doesn't allow anonymous access, which surfaces as an `invalid primary key` error. Set `header_type: Basic` in `admin_gui_auth_conf` (LDAP's default `header_type: ldap` triggers this), and confirm the user exists in Kong Admins, `base_dn` is correct, and any LDAP group is mapped to a Kong Role. +related_resources: [] +--- + +## Problem + +I am trying to set up LDAP Authentication for Kong Manager. I cannot log in. In the log, I found the following error message; + +``` + +access.lua:288 failed to load consumer[postgres] invalid primary key: '{id="expected a valid UUID"}' +``` + +What does this mean? How can I fix it? + +## Solution + +The error message indicated that somehow LDAP authentication failed for a user and fell back to the anonymous user as the plugin's default behavior. As LDAP Authentication for Kong Manager does not allow anonymous user, this always fails. + +Note: Kong Manager login now forces `config.consumer_optional=true` for `ldap-auth-advanced`, so on current {{site.base_gateway}} versions an unmapped/failed LDAP user returns a clean `401 Unauthorized` response instead of this primary-key crash. The debug/error sequence below reflects older {{site.base_gateway}} versions, but the underlying cause and the `header_type` fix are unchanged — if you see a 401 immediately after the `ldap-auth-advanced:access` debug message, the same root cause and fix described here still apply. + +If the error message happens right after the `ldap-auth-advanced:access` message as follows, possibly you have set `header_type: ldap` (by default) + +``` + +kong-ee | 2025/04/15 20:54:45 [debug] 25#0: *164 [lua] init.ljbc:0: calling patched method 'ldap-auth-advanced:access' +kong-ee | 2025/04/15 20:54:45 [debug] 25#0: *164 [lua] init.ljbc:0: calling patched method 'ldap-auth-advanced:access' +kong-ee | 2025/04/15 20:54:45 [debug] 25#0: *164 [kong] access.lua:288 failed to load consumer[postgres] invalid primary key: '{id="expected a valid UUID"}' +``` + +Solution: + +Please make sure `header_type: Basic` is set in the `admin_gui_auth_conf` configuration. This is required for LDAP authentication for Kong Manager. + +This error message may be caused by a different reason, too. In such a case, potentially there are other error or debug messages before the error. + +Also, check the following settings; + +1. The user is already in Admins in Kong + +2. `base_dn` is not only roodDN unless `base_dn` is pointing to a Global catalog + +3. The user is under the `base_dn` sub-directory tree in the LDAP/Active Directory server + +4. `kong_admin` does not work unless the `kong_admin` user exists in the LDAP server or is mapped to an LDAP user + +5. For Group mapping, you have to map the existing LDAP group to a Kong Role at first. Note that an LDAP user still needs to be created in Kong Admins even an LDAP group mapped to a Kong Role. diff --git a/app/_support/failed-to-get-from-node-cache-invalid-status-code-received-from-the-userinfo-endpoint-400.md b/app/_support/failed-to-get-from-node-cache-invalid-status-code-received-from-the-userinfo-endpoint-400.md new file mode 100644 index 0000000000..0a2e0efc67 --- /dev/null +++ b/app/_support/failed-to-get-from-node-cache-invalid-status-code-received-from-the-userinfo-endpoint-400.md @@ -0,0 +1,58 @@ +--- +title: "\"failed to get from node cache: invalid status code received from the userinfo endpoint (400)\" error when using the OpenID Connect plugin with Azure AD" +content_type: support +description: "Kong rejects Azure AD access tokens with a `userinfo` endpoint 400 error when the token's `iss` claim doesn't match the issuer configured in the OpenID Connect plugin." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "Reference" + url: "/plugins/openid-connect/examples/azure-ad/#openid-connect-with-azure-ad" +tldr: + q: "Why does Kong return \"failed to get from node cache: invalid status code received from the userinfo endpoint (400)\" when validating tokens from Azure AD with the OpenID Connect plugin?" + a: | + The token's `iss` (issuer) claim doesn't match the issuer configured in the OpenID Connect plugin. This happens when the Azure AD application manifest's `accessTokenAcceptedVersion` isn't set to `2`, which causes Azure AD to issue tokens with the v1 issuer (`https://sts.windows.net/tenant-id/`) instead of the expected v2 issuer (`https://login.microsoftonline.com//v2.0`). Set `accessTokenAcceptedVersion` to `2` in the Azure AD application manifest to fix the mismatch. +--- + +## Problem + +After configuring the Open ID Connect plugin to validate against Microsoft Azure AD, and calling an API with the access token, Kong rejects the request with an error; + +``` + +"message": "Unauthorized" +``` + +Checking the Kong error log shows the below messages; + +``` + +unable to verify bearer token (invalid issuer (https://sts.windows.net/tenant-id/) was specified for access token, https://login.microsoftonline.com//v2.0 was expected) +``` + +``` + +[openid-connect] failed to get from node cache: invalid status code received from the userinfo endpoint (400) +``` + +## Solution + +If the access token is decoded at https://jwt.io/, you can see the issuer details. For example; + +``` + +"iss": "https://sts.windows.net/fa6xxxx-d29xxxxxxxx2ffbd3e39/" +``` + +The `iss` value is different from that configured in the plugin. + +During the setup of the Application in Microsoft Azure, make sure the value of `accessTokenAcceptedVersion` in the manifest file of the Application is set to 2; + +``` + +"accessTokenAcceptedVersion": 2 +``` + +If `accessTokenAcceptedVersion` is set to null, the issuer assigned will be https://sts.windows.net/tenant-id/ rather than the expected https://login.microsoftonline.com//v2.0 and Kong will reject the call. diff --git a/app/_support/getting-error-api-route-collides-with-an-existing-api.md b/app/_support/getting-error-api-route-collides-with-an-existing-api.md new file mode 100644 index 0000000000..954902aa47 --- /dev/null +++ b/app/_support/getting-error-api-route-collides-with-an-existing-api.md @@ -0,0 +1,45 @@ +--- +title: "\"API route collides with an existing API\" error when creating a route in a different workspace" +content_type: support +description: "{{site.ee_product_name}}'s route collision check warns when a new route in one workspace could unintentionally take traffic from a route in another workspace. Disable the check or reorder route creation to avoid it." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "`route_validation_strategy` configuration" + url: "/gateway/configuration/#route-validation-strategy" +tldr: + q: Why does Kong show an "API route collides with an existing API" error when creating a new route? + a: | + {{site.ee_product_name}}'s route collision check assumes different workspaces are managed by different teams, so it warns when a new route in one workspace could unintentionally take traffic from an existing route in another workspace. To resolve it, give the more generic route a distinct `host` value, disable the check with `route_validation_strategy`, or create routes in other workspaces before creating the generic catch-all route. +--- + +## Problem + +When adding a new route with a path that doesn't exist in any workspace, Kong returns the following error: + +``` +API route collides with an existing API +``` + +## Cause + +In {{site.ee_product_name}} Edition, it is assumed that different workspaces are being used by different teams who do not know about each other's routes, and thus the default route collision check is meant to warn one team that a route already exists in a different workspace which will be affected by the new route. The message that is displayed means that the new route would "steal" traffic from the original route, i.e. traffic that would have gone to the original route would now go to the new route. + +An extreme example to illustrate the issue is: if you first add a super generic route with a path `/` without any host in workspace `default`, all requests will match this route. Any other route in a different workspace you might create later would "steal" some traffic from the generic or catch-all route. Because of this potential for redirecting traffic from a route in a different workspace, we send the route collision warning. + +If a route that would "steal" traffic from another route is created in the same workspace, we allow the route creation because we assume that team members in the same workspace are aware of what each other are doing. + +Also, if the order of route creation is reversed, i.e. the narrower route in one workspace is created before the generic route in a different workspace, the route collision detector won't fire, because the generic route does not "steal" traffic from the more specific route which was created first. The algorithm for route collision detection as documented has some limitations. For example, if the generic route has a method defined, and the more specific route in a different workspace does not, the more specific route is still allowed to be created even though there is a potential for it "stealing" traffic from the generic route. + +If the route collision warning message happens, and it is not clear why, it would be necessary to check all the routes in the other workspaces (other than the workspace in which you are trying to create the new route) for a route that would be affected by the new route. + +## Solution + +There are three options to address the issue: + +1. Make sure the more generic routes have a host value set that is different from the other routes. +2. Disable route collision checking with the `route_validation_strategy` setting. +3. Create all other routes in other workspaces first, and then create the more generic catch-all route. diff --git a/app/_support/getting-postgresql-error-failed-to-retrieve-postgresql-server-version-num-the-server-does-not-support-ssl-connections-error-when-connecting-to-postgres.md b/app/_support/getting-postgresql-error-failed-to-retrieve-postgresql-server-version-num-the-server-does-not-support-ssl-connections-error-when-connecting-to-postgres.md new file mode 100644 index 0000000000..5527aaf8a4 --- /dev/null +++ b/app/_support/getting-postgresql-error-failed-to-retrieve-postgresql-server-version-num-the-server-does-not-support-ssl-connections-error-when-connecting-to-postgres.md @@ -0,0 +1,51 @@ +--- +title: "Getting \"[PostgreSQL error] failed to retrieve PostgreSQL server_version_num: the server does not support SSL connections\" error when connecting to PostgreSQL" +content_type: support +description: PostgreSQL must have SSL enabled in `postgresql.conf` before Kong can make a TLS connection to it. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: "Why does Kong show \"the server does not support SSL connections\" when connecting to PostgreSQL over TLS?" + a: | + PostgreSQL rejects the TLS connection because SSL isn't enabled on the server. Set `ssl = on` in `postgresql.conf` (and configure `ssl_cert_file` / `ssl_key_file`) to allow SSL connections. If you're using a self-signed or private CA certificate, don't set `pg_ssl_verify=off` — Kong's `tls_certificate_verify` check blocks that; instead point `lua_ssl_trusted_certificate` to the CA that signed the PostgreSQL certificate. +related_resources: + - text: PostgreSQL documentation - SSL connection settings + url: https://www.postgresql.org/docs/12/runtime-config-connection.html#RUNTIME-CONFIG-CONNECTION-SSL +--- + +## Problem + +When enabling Kong for TLS connections to the underlying PostgreSQL DB, you receive the following error: + +``` + +Error: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: the server does not support SSL connections +``` + +## Solution + +The reason for this, is that SSL needs to be enabled within PostgreSQL for this to work. To enable this, there are a number of settings you need to enable in the `postgresql.conf` file. This file is generally located under the `/var/lib/pgsql/12/data/postgresql.conf` directory. You must enable `ssl` by changing the value to `on` and also provide additional configuration for the other parameters. This includes specifying your `ssl_cert_file` and `ssl_key_file`. Full details on these parameters can be found in the PostgreSQL documentation. The list of parameters (for version 12 of PostgreSQL) is below: + +``` + +# - SSL - + +ssl = on +#ssl_ca_file = '' +#ssl_cert_file = 'server.crt' +#ssl_crl_file = '' +#ssl_key_file = 'server.key' +#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers +#ssl_prefer_server_ciphers = on +#ssl_ecdh_curve = 'prime256v1' +#ssl_min_protocol_version = 'TLSv1' +#ssl_max_protocol_version = '' +#ssl_dh_params_file = '' +#ssl_passphrase_command = '' +#ssl_passphrase_command_supports_reload = off +``` + +Note: if you are connecting to a PostgreSQL server using a self-signed certificate or one issued by a private CA, do not attempt to work around certificate validation by setting `pg_ssl_verify=off`. This no longer disables verification: Kong now enforces a global `tls_certificate_verify` check, and setting `pg_ssl_verify=off` while it is enabled produces the error "attempt to disable certificate verification while global `tls_certificate_verify` option is enabled". Instead, configure `lua_ssl_trusted_certificate` to point to the CA certificate that signed the PostgreSQL server certificate, so the connection can be verified properly. diff --git a/app/_support/how-can-a-consumer-credential-be-created-with-the-admin-api.md b/app/_support/how-can-a-consumer-credential-be-created-with-the-admin-api.md new file mode 100644 index 0000000000..6284b77e96 --- /dev/null +++ b/app/_support/how-can-a-consumer-credential-be-created-with-the-admin-api.md @@ -0,0 +1,42 @@ +--- +title: Creating a consumer credential with the Admin API +content_type: support +description: How to create basic-auth and key-auth consumer credentials directly through the Kong Admin API instead of Kong Manager. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How can a consumer credential be created with the Admin API? + a: | + Call the Admin API endpoint for the credential type you need instead of using Kong Manager — for example, `POST /consumers/{consumer}/basic-auth` for Basic Auth credentials or `POST /consumers/{consumer}/key-auth` for key-auth credentials. +--- + +## Overview + +From within Kong Manager, it is possible to create consumer credentials. Is it possible for consumer credentials to be created directly via an Admin API call? + +## Steps + +To create credentials, you need to call the appropriate endpoint for the required credential type. Some examples are shown below; + +1. To create basic-auth Consumer credentials via an Admin API call, use a request similar to below; + + ```bash + + curl -s -X POST 'https://api.kong.lan:8444/{% raw %}{{workspace}}{% endraw %}/consumers/{% raw %}{{consumer-name}}{% endraw %}/basic-auth' \ + -H 'Content-Type: application/json;charset=UTF-8' \ + --data-raw '{"username":"my-username", + "password":"my-password"}' + ``` + +2. To create key-auth Consumer credentials via an Admin API call, use a request similar to below; + + ```bash + + curl -s -X POST 'https://api.kong.lan:8444/{% raw %}{{workspace}}{% endraw %}/consumers/{% raw %}{{consumer-name}}{% endraw %}/key-auth' \ + -H 'Content-Type: application/json;charset=UTF-8' \ + --data-raw '{"key":"new-key"}' + ``` diff --git a/app/_support/how-can-a-consumer-credential-be-deleted-with-the-admin-api.md b/app/_support/how-can-a-consumer-credential-be-deleted-with-the-admin-api.md new file mode 100644 index 0000000000..fab354797d --- /dev/null +++ b/app/_support/how-can-a-consumer-credential-be-deleted-with-the-admin-api.md @@ -0,0 +1,88 @@ +--- +title: Deleting a consumer credential with the Admin API +content_type: support +description: How to delete a consumer credential (basic-auth, key-auth, ACL, HMAC, JWT, or OAuth 2.0) using the Kong Admin API DELETE endpoint. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How can a consumer credential be deleted with the Admin API? + a: | + Send a `DELETE` request to the credential's Admin API endpoint, for example `DELETE /default/consumers/joe/basic-auth/testcred`. The path segment (`basic-auth`, `key-auth`, `acls`, `hmac-auth`, `jwt`, `oauth2`) and identifier field vary by credential type. +--- + +## Overview + +How can a consumer credential be deleted with the Admin API? + +## Steps + +You can delete a specific credential by sending a DELETE request to + +``` + +http(s)://:/consumers//// +``` + +For example, assuming the below parameters this can be used to delete a basic auth credential: + +``` + +kong-host: localhost +admin-port: 8001 +workspace: default +consumer name: joe +credential type: basic-auth +credential username: testcred +``` + +Request: + +```bash + +curl -X DELETE -H "kong-admin-token:kong" http://localhost:8001/default/consumers/joe/basic-auth/testcred +``` + +The credential types and named fields are as follows: + + +{% table %} +columns: + - title: Credential Type + key: credential_type + - title: Value + key: value + - title: Field Reference + key: field_reference + - title: Example + key: example +rows: + - credential_type: "Basic Authentication" + value: "basic-auth" + field_reference: "Username" + example: ":8001/default/consumers/joe/basic-auth/" + - credential_type: "ACL" + value: "acls" + field_reference: "Group" + example: ":8001/default/consumers/joe/acls/" + - credential_type: "Key Authentication" + value: "key-auth" + field_reference: "Key" + example: ":8001/default/consumers/joe/key-auth/" + - credential_type: "HMAC Authentication" + value: "hmac-auth" + field_reference: "Username" + example: ":8001/default/consumers/joe/hmac-auth/" + - credential_type: "JWT" + value: "jwt" + field_reference: "Key" + example: ":8001/default/consumers/joe/jwt/" + - credential_type: "OAuth 2.0" + value: "oauth2" + field_reference: "Client ID" + example: ":8001/default/consumers/joe/oauth2/" +{% endtable %} + diff --git a/app/_support/how-can-custom-dependencies-be-installed-in-kong-images.md b/app/_support/how-can-custom-dependencies-be-installed-in-kong-images.md new file mode 100644 index 0000000000..22e9227ad7 --- /dev/null +++ b/app/_support/how-can-custom-dependencies-be-installed-in-kong-images.md @@ -0,0 +1,71 @@ +--- +title: Installing custom dependencies in Kong Docker images +content_type: support +description: Extend the standard Kong Docker image with a custom `Dockerfile` to install additional CLI tools and Lua libraries for a custom plugin. +products: + - gateway + - kic +works_on: + - on-prem + - konnect +tldr: + q: How can custom dependencies be installed in Kong images? + a: | + Extend the standard Kong Docker image with a custom `Dockerfile`: switch to the `root` user, install packages with `apt-get` (the default image is Ubuntu-based), use `luarocks` to add Lua rocks, then switch back to the `kong` user and build the image with `docker build`. +related_resources: [] +--- + +## Overview + +When using a custom plugin, there is sometimes a requirement to add some custom libraries for the plugin. How can the libraries be installed in a docker image for use in a Kubernetes deployment? + +## Steps + +To create a custom image using the standard Kong image as a base, it is necessary to create a `Dockerfile`. The below example of a `Dockerfile` can be used to create a custom image that has an additional CLI tool and some lua packages installed; + +```dockerfile + +FROM kong/kong-gateway:3.14.0.0 + +# Use the priviled root user to install new packages +USER root + +# Install additional tools by using APT package manager here +# Example here is for adding the common tool jq into the image +# unzip is required by luarocks to install zip-packaged rocks +RUN apt-get update \ + && apt-get install -y jq unzip \ + && rm -rf /var/lib/apt/lists/* + +# The lua package manger is installed by default with Kong +# use the tool to install additional lua plugins + +# Use the luarocks repository to install rock files +RUN ["luarocks", "install", "lua-resty-jwt"] + +# Copy a local .rock file to the image and install it +COPY lua-zlib-1.2-0.linux-x86_64.rock /tmp +RUN luarocks install /tmp/lua-zlib-1.2-0.linux-x86_64.rock +RUN rm /tmp/lua-zlib-1.2-0.linux-x86_64.rock + +# Switch back to the non-privileged kong user +USER kong +``` + +Note: the Enterprise `-alpine` image variant no longer exists; the default {{site.base_gateway}} image is now Ubuntu-based, so package installation uses `apt-get` rather than `apk`. Installing `unzip` is required before `luarocks install`, since zip-packaged rocks fail to install without it on the Ubuntu-based image. + +To build the custom image, run the below command in the same directory and the `Dockerfile`; + +```bash + +docker build -t kong-with-jq-jwt . +``` + +This will create a local image with the tag `kong-with-jq-jwt`. You can use this in a `docker-compose` file like this; + +```yaml + +image: kong-with-jq-jwt:latest +``` + +You can then push this image to your organizations docker repository with the `docker push` command. diff --git a/app/_support/how-can-i-enable-streaming-of-responses-with-text-event-stream-content-types-instead-of-having-the-responses-buffered-and-sent-in-one-large-payload.md b/app/_support/how-can-i-enable-streaming-of-responses-with-text-event-stream-content-types-instead-of-having-the-responses-buffered-and-sent-in-one-large-payload.md new file mode 100644 index 0000000000..6afc09de20 --- /dev/null +++ b/app/_support/how-can-i-enable-streaming-of-responses-with-text-event-stream-content-types-instead-of-having-the-responses-buffered-and-sent-in-one-large-payload.md @@ -0,0 +1,47 @@ +--- +title: "Enabling streaming of responses with `text/event-stream` content-types instead of buffering them into one large payload" +content_type: support +description: To stream responses with the `text/event-stream` content type instead of having them buffered, disable response body buffering on the specific route by setting `response_buffering` to `false`. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: "How can I enable streaming of responses with `text/event-stream` content-types, instead of having the responses buffered and sent in one large payload?" + a: | + Disable response body buffering on the affected route by setting `response_buffering` to `false` (via a `PATCH` to the Admin API, or in Kong Manager on 3.x+). Kong buffers responses by default because Nginx `proxy_buffering` is on, so turning it off lets `text/event-stream` responses stream from the upstream as they arrive. +related_resources: [] +--- + +## Overview + +How can I enable streaming of responses with `text/event-stream` content-types, instead of having the responses buffered and sent in one large payload? + +## Steps + +To address the issue of responses being buffered and sent in one large payload through Kong (for example, Azure OpenAI endpoint payloads), especially when the content type of the response is `text/event-stream`, you can disable response body buffering for the specific route in question. + +Nginx has a feature called `proxy_buffering` which enables the buffering of responses, and is set to `true` by default. + +Here are the steps to disable response buffering on a route: + +1. Identify the route for which you want to disable response buffering. You can do this by accessing your Kong Admin API or Kong Manager UI and listing all routes associated with the service that communicates with the streaming server. + +2. Once you have identified the route and its ID, you need to update its configuration to disable response buffering. This can be done by making a `PATCH` request to the Kong Admin API with the `response_buffering` property set to `false`. + +Here is an example of how to disable response buffering using a curl command: + +```bash +curl 'https://kong.admin.api:8444/default/routes/' \ +  -X 'PATCH' \ +  -H 'Content-Type: application/json' \ +  -H "kong-admin-token:" \ +  --data-raw '{"response_buffering": false}' +``` + +There is also an option in {{site.base_gateway}} Manager to disable `response_buffering` if using Kong 3.x+. + +After applying this change, Kong will no longer buffer the responses for the specified route, allowing the responses to be streamed from the upstream server endpoints. + +By following these steps, you should be able to configure your service to stream responses directly, avoiding the delay caused by buffering large responses. diff --git a/app/_support/how-can-i-increase-the-connect-timeout-using-the-serverless-functions-plugin.md b/app/_support/how-can-i-increase-the-connect-timeout-using-the-serverless-functions-plugin.md new file mode 100644 index 0000000000..a83c4e966b --- /dev/null +++ b/app/_support/how-can-i-increase-the-connect-timeout-using-the-serverless-functions-plugin.md @@ -0,0 +1,72 @@ +--- +title: Increasing the connect timeout using the Serverless Functions plugin +content_type: support +description: Override Kong's `connect_timeout` to the upstream using a Lua snippet in the Serverless (pre-function) plugin's `config.access` phase. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can I increase the connect timeout using the Serverless Functions Plugin? + a: | + Override the upstream `connect_timeout` (in milliseconds) with a Lua snippet — e.g. `ngx.ctx.balancer_data.connect_timeout = 100000` — set via `config.access` on the pre-function plugin (`config.functions` is not a valid field). This timeout only surfaces visibly against an upstream that's slow or unresponsive at the TCP handshake stage, not against one that is merely slow to respond. +related_resources: [] +--- + +## Overview + +How can I increase the connect timeout to the upstream service using the Serverless (Pre-Function) Plugin? + +## Steps + +Using the Serverless (Pre-function) Plugin in Kong, you can override the connect timeout to the upstream service. Below is an example on how this can be achieved: + +1. Create a file called `function.lua`. + +2. Content of `function.lua`: + + ```lua + + ngx.ctx.balancer_data.connect_timeout = 100000 + ``` + + Note: `connect_timeout` is in milliseconds, so `100000` is 100 seconds. Setting it to a plain `100` (100 milliseconds) would *decrease* the timeout well below the 60000ms (60s) default, causing Kong to fail faster, not slower — the opposite of what this article is demonstrating. + +3. Create a Service (upstream service here will not respond for 10 seconds): + + ```bash + + curl -i -X POST --header 'kong-admin-token: ' --url http://localhost:8001/services/ \ + --data 'name=test' \ + --data 'url=http://httpbin.org/delay/10' + ``` + +4. Create a Route: + + ```bash + + curl -i -X POST --header 'kong-admin-token: ' --url http://localhost:8001/services/test/routes \ + --data 'name=testing-route' \ + --data 'paths[]=/without-timeout' + ``` + +5. Apply the Serverless (Pre-Function) Plugin. + + `config.functions` is not a valid field on the pre-function/post-function plugins. You must use the phase-specific field instead, in this case `config.access`, which takes an array of Lua code snippets to run in the access phase: + + ```bash + + curl -i -X POST --header 'kong-admin-token: ' --url http://localhost:8001/routes/testing-route/plugins \ + -F "name=pre-function" \ + -F "config.access[1]=@function.lua" + ``` + +6. Test. + + ```bash + + curl -i GET http://localhost:8000/without-timeout + ``` + + This request succeeds after Kong's default 60-second response wait, since `httpbin.org/delay/10`'s 10-second delay happens after the TCP connection is already established — the delay is server-side response latency, governed by `read_timeout`/`write_timeout`, not by `connect_timeout`. In other words, this particular example does not actually exercise the raised `connect_timeout` value at all, because connecting to `httpbin.org` itself is fast regardless of the configured value. `connect_timeout` only matters (and only fails/succeeds visibly) against an upstream that is slow or unresponsive at the TCP handshake stage — for example, an unroutable/firewalled IP that never completes a TCP handshake. Live-reproduced: pointing a Service at such an address with the default 60000ms `connect_timeout` makes Kong hang for the full default duration before failing; overriding `ngx.ctx.balancer_data.connect_timeout` to a small value (e.g. `100` for 100ms) in `config.access` makes Kong instead fail fast with a `504 Gateway Timeout` in a few hundred milliseconds — confirming the override mechanism itself works correctly, even though the `httpbin.org/delay/10` example above doesn't demonstrate it. diff --git a/app/_support/how-can-i-integrate-2fa-mfa-otp-or-captcha-recaptcha-with-kong-manager.md b/app/_support/how-can-i-integrate-2fa-mfa-otp-or-captcha-recaptcha-with-kong-manager.md new file mode 100644 index 0000000000..c0fbf568dd --- /dev/null +++ b/app/_support/how-can-i-integrate-2fa-mfa-otp-or-captcha-recaptcha-with-kong-manager.md @@ -0,0 +1,26 @@ +--- +title: Integrating 2FA, MFA, OTP, or captcha/recaptcha with Kong Manager +content_type: support +published: false +description: Kong Manager does not directly offer 2FA, MFA, OTP or captcha/Recaptcha. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can I integrate 2FA, MFA, OTP, or captcha/recaptcha with Kong Manager? + a: | + Kong Manager does not directly support 2FA, MFA, OTP, or captcha/Recaptcha. Configure Kong Manager to use OIDC auth instead, and you can provide MFA (or similar) through the OIDC provider. +related_resources: + - text: Configure OIDC authentication for Kong Manager + url: /gateway/kong-manager/openid-connect/#enable-openid-connect-for-kong-manager +--- + +## Overview + +How can I integrate 2FA, MFA, OTP, or captcha/recaptcha with Kong Manager? + +## Steps + +Kong Manager does not directly offer 2FA, MFA, OTP or captcha/Recaptcha. However, if you configured Kong Manager to use OIDC auth then you could provide MFA etc via the OIDC provider. diff --git a/app/_support/how-can-i-take-a-packet-capture-in-a-kong-pod-in-a-kubernetes-environment.md b/app/_support/how-can-i-take-a-packet-capture-in-a-kong-pod-in-a-kubernetes-environment.md new file mode 100644 index 0000000000..1ab7385eee --- /dev/null +++ b/app/_support/how-can-i-take-a-packet-capture-in-a-kong-pod-in-a-kubernetes-environment.md @@ -0,0 +1,61 @@ +--- +title: Taking a packet capture in a Kong pod in a Kubernetes environment +content_type: support +description: "Add a sidecar container running a `tcpdump` image to the Kong pod to capture packet traffic in a Kubernetes environment where the standard Kong image doesn't include `tcpdump` or root access." +products: + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How can I take a packet capture in a Kong pod in a Kubernetes environment? + a: | + Add a sidecar container based on a `tcpdump` image to the Kong pod (via Helm's `sidecarContainers` values), then `kubectl exec` into that container to run `tcpdump` against the traffic you're interested in, and `kubectl cp` the resulting `.pcap` file out for analysis. +--- + +## Overview + +There are certain situations where it is crucial to get a packet capture of TCP traffic that reaches Kong nodes but the standard Kong images do not have tcpdump installed, and we can not access the Kong nodes with root access. For example, we want to understand why mTLS handshakes are not working or what response a Kong node gets from an upstream or IdP endpoint when the error log does not provide enough information. What is a good way to do a packet capture? + +## Steps + +A valuable way to take a packet capture in a Kubernetes environment is to add an additional sideCar container to the kong pod which contains the a tcpdump image. + +For example you can add the following to a Kong Helm values.yaml file to add the "tcpdump" container to an existing pod with a Kong "Proxy" container (and a Kong Ingress-Controller container): + +```yaml + +deployment: + kong: + enabled: true + sidecarContainers: + - name: tcpdump + securityContext: + runAsUser: 0 + image: corfr/tcpdump + command: + - /bin/sleep + - infinity +``` + +- Once you upgrade the helm deployment, the additional container should get deployed with the tcpdump utility. You should be able to access it with something like this: + +```bash + +kubectl -n exec -it -c tcpdump /bin/sh +``` + +- In the container, you should be able to capture the tcp traffic against whatever host/port you are interested in using something like the following command: + +```bash + +tcpdump -npi any -As0 -w /tmp/packet.pcap host and port +``` + +- Then copy the packet.pcap onto your local machine for further analysis or to make available to Kong Support you can use the following command: + +```bash + +kubectl cp -n :/tmp/packet.pcap ./packet.pcap -c tcpdump +``` diff --git a/app/_support/how-can-kong-be-configured-to-use-azure-key-vaults-with-managed-identities-authentication.md b/app/_support/how-can-kong-be-configured-to-use-azure-key-vaults-with-managed-identities-authentication.md new file mode 100644 index 0000000000..23b0dffed5 --- /dev/null +++ b/app/_support/how-can-kong-be-configured-to-use-azure-key-vaults-with-managed-identities-authentication.md @@ -0,0 +1,64 @@ +--- +title: Configuring Kong to use Azure Key Vaults with Managed Identities authentication +content_type: support +description: "Set up a Managed Identity in Azure, assign it to the Kong VM or AKS instance, grant it access in Key Vault, and configure `AZURE_CLIENT_ID` and `KONG_VAULT_AZURE_VAULT_URI` so Kong can authenticate to Azure Key Vault without explicit credentials." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How can Kong be configured to use Azure Key Vaults with Managed Identities authentication? + a: | + Create an Azure Managed Identity, assign it to the VM or AKS instance running Kong, and grant it `Secrets -> Get` access in Key Vault. Then set `AZURE_CLIENT_ID` and `KONG_VAULT_AZURE_VAULT_URI` as environment variables so Kong's Azure SDK can use the managed identity to authenticate and retrieve secrets, for example `kong vault get {vault://azure/your-secret-name}`. +--- + +## Overview + +How to Configure Azure Key Vaults Backend with Managed Identities in Kong? + +## Steps + +Configuring Azure Key Vaults as a backend for secrets management in Kong using managed identities involves a series of steps to ensure secure and seamless integration. This process allows Kong to authenticate with Azure Key Vaults without the need for explicit credentials, leveraging Azure's managed identities for a more secure and manageable setup. + +The core issue addressed here is the lack of documentation on configuring Kong with Azure Key Vaults using managed identities. The resolution involves setting up the environment correctly and ensuring that Kong can authenticate with Azure Key Vaults using the managed identity assigned to the Azure resource (VM or AKS). + +Here are the steps to achieve this configuration: + +1. Create a Managed Identity in Azure + +- Navigate to the Azure portal and create a new managed identity. Note the `AZURE_CLIENT_ID` of the created identity. + +2. Assign the Managed Identity to Your Azure Resource + +- Assign the managed identity to the Azure VM or AKS instance where Kong is deployed. + +3. Configure Access Policies in Azure Key Vault + +- Go to your Azure Key Vault instance and add an access policy. Grant the managed identity `Secrets -> Get` permission. + +4. Set Required Environment Variables in Kong + +- Set the following environment variables in the Kong environment, replacing the placeholders with actual values: + +``` + +AZURE_CLIENT_ID= +KONG_VAULT_AZURE_VAULT_URI= +``` + +5. Retrieve Secrets from Azure Key Vault in Kong + +- Use the following command to retrieve a secret from Azure Key Vault: + +```bash + +kong vault get {vault://azure/your-secret-name} +``` + +During this process, you might encounter error messages related to authentication or missing environment variables. These errors often indicate a misconfiguration in the environment variables or access policies. Ensure that the `AZURE_CLIENT_ID` and `KONG_VAULT_AZURE_VAULT_URI` are correctly set and that the managed identity has the appropriate permissions in Azure Key Vault. + +Additionally, it's important to note that Kong's integration with Azure Key Vaults using managed identities is designed to work with minimal configuration. The Azure SDK automatically picks up the managed identity of the machine or pod, using it to authenticate and read secrets from Azure Key Vaults. Therefore, apart from setting the `AZURE_CLIENT_ID` as a hint for the SDK, no further specific configuration should be necessary. + +This setup allows for a secure and efficient way to manage secrets in Kong, leveraging Azure's managed identities and Key Vaults. diff --git a/app/_support/how-can-local-files-be-used-to-sign-jwt-tokens-and-what-format-must-the-keys-be-in.md b/app/_support/how-can-local-files-be-used-to-sign-jwt-tokens-and-what-format-must-the-keys-be-in.md new file mode 100644 index 0000000000..41795e29b0 --- /dev/null +++ b/app/_support/how-can-local-files-be-used-to-sign-jwt-tokens-and-what-format-must-the-keys-be-in.md @@ -0,0 +1,106 @@ +--- +title: Using local files to sign JWT tokens and the required key format +content_type: support +description: The `jwt-plugin`'s keyset parameters only support loading keys from an http(s) endpoint, so local key files must be served through a custom nginx endpoint instead. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can local files be used to sign JWT tokens and what format must the keys be in? + a: | + The `jwt-signer` plugin's `*_keyset` parameters only load keys from an http(s) endpoint, not directly from the local filesystem. To serve local key files, add a custom nginx server block (loaded via `nginx_http_include` in `kong.conf`) that exposes them over `http://127.0.0.1:/...` with a `Content-Type: application/jwk-set+json` header, then point the plugin's keyset parameter at that local endpoint. Keys must be in JWK format; PEM keys need to be converted first. +related_resources: + - text: "`jwt-signer` plugin: managing key signing" + url: /plugins/jwt-signer/#manage-key-signing + - text: The JWK format (RFC 7517) + url: https://tools.ietf.org/html/rfc7517 + - text: CLI tool lokey + url: https://github.com/jpf/lokey + - text: Online JS conversion + url: https://irrte.ch/jwt-js-decode/pem2jwk.html + - text: A CLI tool for creating JWK's + url: https://smallstep.com/docs/cli/crypto/jwk/create/ +--- + +## Overview + +When using the `jwt-signer` plugin, how can custom keys be used to sign the tokens? The documentation mentions that the `*_keyset` parameters can be used but there is no detail on how to use this to pass key files from the local filesystem to the plugin, nor the required format for the keys. + +## Steps + +The `jwt-plugin` does not support loading keys from a local filesystem. The keyset parameters can support an http(s) endpoint to load your own keys, or any other value will cause Kong to auto-generate keys. + +It is however possible to set up a custom nginx server block to serve keys from the local filesystem. The keys need to be provided from an endpoint that provides a JWK format key with a content-type of `application/jwk-set+json`. The JWK format is described here. + +Assuming that the keys are stored in the local filesystem in the `/opt/key` directory, you can configure a new nginx server block to return the right content-type for files ending in `.key`. Firstly, create a file `/etc/kong/jwk-keys.conf` with contents like this: + +```nginx +server { + listen 127.0.0.1:4444; + root /opt/keys/; + location / { + types { + application/jwk-set+json key; + } + } +} +``` + +Next, add the below parameter to the `kong.conf` file to load the `jwk-keys.conf` as an nginx include file: + +```bash +nginx_http_include=/etc/kong/jwk-keys.conf +``` + +Note, after restarting Kong, the jwt keys will be available via the localhost interface, i.e. the keys will not be available publicly. + +```bash +curl -v http://127.0.0.1:4444/jwks.key +* About to connect() to 127.0.0.1 port 4444 (#0) +* Trying 127.0.0.1... +* Connected to 127.0.0.1 (127.0.0.1) port 4444 (#0) +> GET /jwks.key HTTP/1.1 +> User-Agent: curl/7.29.0 +> Host: 127.0.0.1:4444 +> Accept: */* +> +< HTTP/1.1 200 OK +< Date: Thu, 14 May 2026 14:08:03 GMT +< Content-Type: application/jwk-set+json +< Content-Length: 705 +< Last-Modified: Thu, 14 May 2026 11:05:31 GMT +< Connection: keep-alive +< ETag: "5ebd25fb-2c1" +< Accept-Ranges: bytes +< +{ + "kid": "74bd86fc61e4c6cb450126ff4e38b069b8f8f35c", + "e": "AQAB", + "alg": "RS256", + "use": "sig", + "n": "q9WQ8_ucw5sLCKMZpWj1WhZXW1C83G6aE7NST1D3cUNnKIN3RhI04EOtJrbfF5wJwmdMurqwIJuhXBC44pyhBkaxJ0-lyrvgLHVhQhxH6K9b-UV0whE0eqiOOl1snKk-N0BRfT5dmCghr7rxcHUJqSFuDpZo2ZJzMiuF2DmeQHaTtusLnU-7xnP4B4eHG_h4nisK1zx8-l-rBYyaGHRf6ZqelTpRDHDVQMGuunbGqVXRgc1OjwPci6ZDzdSFRGST3gCZFirRfOoXMqF2474TD3KjYPdmwfETiPAfOVCA9I2mVj4IhbELDTVVYdh0DBs3mks1j2TBIUniUiDs5c-_ow", + "kty": "RSA" + }, + { + "e": "AQAB", + "alg": "RS256", + "use": "sig", + "n": "xHKHysGHfZby92stAyC4Xkp7t2Ib6TEha1G11UwGgmrv7pgpjKBJkO1XtvvT2L3pEylhcKhLgO8fx5R-rKceezZ_YpTyuT1vHHsWYJxeocV5m0V70_Nvgfysl6lS_gdvfT68dMNk1EL8bIk9uiCMIotVpcq4FIeID75Dendq_oTuXOZVeCi1r8q0qeMWN7nFZEJCnxzayNOTE7-eC8FRMRiu-e3tOtkruga3Cz62nkkrGtQyAaQtUntrDTQxjE2TNhBvWBWDVOfvG-uCe2JkhfDC7CZlE6tpBo-VkyIGGZjR5qBlYTKx6ZJjWeQC13lpwd7WB1vKtBSKGH2vKuBbrQ", + "kty": "RSA", + "kid": "c1771814ba6a70693fb9412da3c6e90c2bf5b927" + } +``` + +You can then configure the `jwt-signer` plugin to use the local http address to load the custom keys: + +```bash +config.access_token_keyset=http://127.0.0.1:4444/jwks.key +``` + +To convert a PEM key to JWK format, there are a few 3rd party utilities. For example, see the below links (these are not validated by Kong so please ensure that you perform your own validation as to the suitability of these or any other tools you choose to use for JWK creation). + +```bash +step crypto jwk create -kty=RSA -alg=RS256 pub.json priv.json +``` diff --git a/app/_support/how-can-the-admin-api-rbac-token-be-reset-programmatically.md b/app/_support/how-can-the-admin-api-rbac-token-be-reset-programmatically.md new file mode 100644 index 0000000000..bb53c20866 --- /dev/null +++ b/app/_support/how-can-the-admin-api-rbac-token-be-reset-programmatically.md @@ -0,0 +1,53 @@ +--- +title: Resetting the Admin API RBAC token programmatically +content_type: support +description: Reset the Admin API RBAC token programmatically by authenticating to get a Kong Manager session cookie and using it to call the `/admins/self/token` endpoint. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can the Admin API RBAC token be reset programmatically? + a: | + Authenticate against `/auth` with the Kong Manager admin credentials to obtain a session cookie, then use that cookie to call `PATCH /admins/self/token`, which generates and returns a new RBAC token. + The token value is always auto-generated and cannot be set to a specific value. +related_resources: [] +--- + +## Overview + +Kong Manager can be used to reset the RBAC token, but how can this be done programmatically via a curl to the Admin API? + +## Steps + +The RBAC tokens are not documented as part of the Admin API as it requires a Kong Manager session cookie as authentication to reset it. It is possible to authenticate and obtain the session cookie and authenticate the token reset using the below example. + +Be advised that this will store the session cookie locally and is provided simply to demonstrate how this is possible. + +```bash +curl 'https://admin-api.konghq.com:/auth' \ +-H 'Kong-Admin-User: kong_admin' \ +-H 'Authorization: Basic a29uZ19hZG1pbjprb25n' -ik -c session.txt + +curl -X PATCH 'https://admin-api.konghq.com:/admins/self/token' \ +-H 'Kong-Admin-User: kong_admin' \ +-b session.txt -ik +``` + +The new RBAC token is automatically generated and cannot be set to a specific value. Below is an example request and response when generating a new token; + +```bash +curl -k -X PATCH 'https://admin-api.konghq.com:/admins/self/token' \ +-H 'Kong-Admin-User: kong_admin' -b session.txt +{"message":"Token reset successfully","token":"SY8YYcwKhkve7FJRrhchjbS0yVP16spc"} +``` + +You could store the new token in an env variable for use with a subsequent command like this; + +```bash +export RBAC_TOKEN=$(curl -sk -X PATCH 'https://admin-api.konghq.com:/admins/self/token' -H 'Kong-Admin-User: kong_admin' -b session.txt | jq -r '.token') + +echo $RBAC_TOKEN +TLqCCXDPlWEre6MPm4vC1WuE4LccMewL +``` diff --git a/app/_support/how-can-the-authorization-headers-be-shown-in-the-kong-log-plugins.md b/app/_support/how-can-the-authorization-headers-be-shown-in-the-kong-log-plugins.md new file mode 100644 index 0000000000..414e728a39 --- /dev/null +++ b/app/_support/how-can-the-authorization-headers-be-shown-in-the-kong-log-plugins.md @@ -0,0 +1,67 @@ +--- +title: Showing Authorization headers in the Kong log plugins +content_type: support +description: Log plugins redact the Authorization header by default; use `custom_fields_by_lua` to log its value to a separate field for debugging. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can the Authorization headers be shown in the Kong log plugins? + a: | + Log plugins redact the Authorization header and there is no config option to unredact it. + Use `custom_fields_by_lua` to log the value into a separate field, for example returning `kong.request.get_header("authorization")` into a field such as `x-original-authorization`. Disable the plugin once debugging is done, given the security exposure. +related_resources: [] +--- + +## Overview + +When sending Authorization headers, the values for these headers are redacted when using one of the logging plugins. The log shows entries like this; + +```json +"headers": { +  "authorization": "REDACTED", +} +``` + +For debugging purposes, it is sometimes necessary to log the actual values. Can this behavior be controlled via a configuration parameter? + +## Steps + +There is no configuration option available to log the value of the Authorization headers. It is possible to use the `custom_fields_by_lua` feature to log the values to a different custom field. For example, you could set a log field called `x-original-authorization` and set this to the value for the Authorization using the Kong PDK. + +As an example, the `udp-log` plugin could be configured like this; + +```bash +curl -s -X POST 'https://api.kong.lan:8444/default/routes/{% raw %}{{routeName}}{% endraw %}/plugins/' \ +-H 'Content-Type: application/json' \ +--data-raw '{ + "tags": [ + "plugin-example" + ], + "name": "udp-log", + "config": { + "custom_fields_by_lua": { + "x-original-authorization": "return kong.request.get_header(\"authorization\")" + }, + "timeout": 10000, + "port": 5555, + "host": "udp-log-hostname" + } +}' +``` + +Calling the Route with an Authorization header; + +```bash +curl -H "Authorization: fred123" http://proxy.kong.lan/echo +``` + +will still log the REDACTED value for the Authorization header, but there will also be an extra entry in the log showing the complete value sent for the Authorization header; + +```json + "x-original-authorization": "fred123" +``` + +Note, there is a potential security implication of saving the Authentication headers, please ensure that you are comfortable with logging these values and disable/delete the plugin once you have completed the debugging. diff --git a/app/_support/how-can-the-kong-declarative-config-file-be-validated-in-db-less-mode.md b/app/_support/how-can-the-kong-declarative-config-file-be-validated-in-db-less-mode.md new file mode 100644 index 0000000000..7f9ea56ba8 --- /dev/null +++ b/app/_support/how-can-the-kong-declarative-config-file-be-validated-in-db-less-mode.md @@ -0,0 +1,65 @@ +--- +title: Validating the Kong declarative config file in db-less mode +content_type: support +description: The Kong binary can be used to validate a config for problems prior to loading using the `config parse` command. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can the Kong declarative config file be validated in db-less mode? + a: | + Run `kong config parse .yaml` to validate a declarative config before loading it — Kong reports the exact location of any schema violation (for example, an invalid `protocols` value). You can also run this externally, without a running Kong node, using the `kong/kong-gateway` Docker image against a mounted config directory. +related_resources: + - text: Kong CLI reference + url: /gateway/cli/reference/ +--- + +## Overview + +The Kong binary can be used to validate a config for problems prior to loading, using the `config parse` command. + +## Steps + +```bash +kong config parse +``` + +For example, in the below configuration (`kong.yaml`) we specify an invalid route protocol `gruber`: + +```yaml +_format_version: "1.1" +services: +- connect_timeout: 60000 + host: localhost + name: httpbin + path: /anything + routes: + - name: httpbin-route + paths: + - /echo + protocols: + - gruber + - http +``` + +When running `kong config parse` we can see where the error resides: + +```bash +kong config parse kong.yaml + +Error: Failed parsing: +in 'services': + - in entry 1 of 'services': + in 'routes': + - in entry 1 of 'routes': + in 'protocols': + - in entry 1 of 'protocols': expected one of: grpc, grpcs, http, https, tcp, tls, tls_passthrough, udp +``` + +This can optionally be performed external to the Kong system using the Docker image, where the config file for validation exists in the current directory. + +```bash +docker run --rm -v ${PWD}:/config -e "KONG_DATABASE=off" --rm kong/kong-gateway:3.14.0.0 kong config parse /config/kong.yaml +``` diff --git a/app/_support/how-can-we-log-more-information-about-client-certificates-used-in-requests-in-the-kong-access-log.md b/app/_support/how-can-we-log-more-information-about-client-certificates-used-in-requests-in-the-kong-access-log.md new file mode 100644 index 0000000000..4c3ccd9400 --- /dev/null +++ b/app/_support/how-can-we-log-more-information-about-client-certificates-used-in-requests-in-the-kong-access-log.md @@ -0,0 +1,30 @@ +--- +title: Logging client certificate information in the Kong access log +content_type: support +published: false +description: Nginx offers several access log variables, such as `$ssl_client_s_dn` and `$ssl_client_raw_cert`, that can log client certificate information in a custom access log format. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can we log more information about client certificates used in requests in the kong access log? + a: | + Nginx exposes client-certificate variables such as `$ssl_client_s_dn` (client DN) and `$ssl_client_raw_cert` (raw cert). + Add them to a custom log format via `KONG_NGINX_HTTP_LOG_FORMAT` and reference that format in `KONG_PROXY_ACCESS_LOG`. +related_resources: [] +--- + +## Overview + +How can we log more information about client certificates used in requests in the kong access log? In particular we are interested in getting the CN and possibly the whole client cert for requests. + +## Steps + +Nginx offers several variables that can be used to log client certificate information in a custom access log format. In particular, there are `$ssl_client_s_dn`, and a `$ssl_client_raw_cert` variable which may be useful to use either or both to log information about the client certificate. You could modify the standard nginx access log format to return any of the variables. One example which would return both the client DN, and the raw client certificate could be achieved with these kong variables: + +```bash +KONG_NGINX_HTTP_LOG_FORMAT: show_client_cert '$$time_iso8601 - $bytes_sent - $$request - $$status - $$remote_addr - $$ssl_client_s_dn - $$ssl_client_raw_cert' +KONG_PROXY_ACCESS_LOG: /dev/stdout show_client_cert +``` diff --git a/app/_support/how-do-i-use-the-prometheus-plugin-with-konnect.md b/app/_support/how-do-i-use-the-prometheus-plugin-with-konnect.md new file mode 100644 index 0000000000..3afddb0d3d --- /dev/null +++ b/app/_support/how-do-i-use-the-prometheus-plugin-with-konnect.md @@ -0,0 +1,30 @@ +--- +title: Using the Prometheus plugin with Konnect +content_type: support +published: false +description: "Enable the `status_listener` on the data plane so its metrics can be scraped by Prometheus or other analytics tools." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I use the Prometheus plugin with Konnect? + a: | + Enable the `status_listener` on the data plane. This exposes a metrics endpoint that Prometheus (or other analytics tools) can scrape. +related_resources: + - text: "{{site.base_gateway}} configuration reference (`status_listen`)" + url: /gateway/configuration/#status-listen + - text: Updating `kong.conf` to turn on the status listener (video) + url: https://youtu.be/6rU_uht_HLo?t=264 + - text: Setting up Prometheus to work with the new status listener (video) + url: https://youtu.be/6rU_uht_HLo?t=467 +--- + +## Overview + +We want to be able to use the Prometheus plugin in our Konnect environment. How do we accomplish this? + +## Steps + +You need to enable the `status_listener` on the data plane. This will allow the metrics to be scraped by Prometheus or other analytics tools. diff --git a/app/_support/how-to-accurately-calculate-upstream-target-weights-for-traffic-distribution.md b/app/_support/how-to-accurately-calculate-upstream-target-weights-for-traffic-distribution.md new file mode 100644 index 0000000000..85a5e0adbf --- /dev/null +++ b/app/_support/how-to-accurately-calculate-upstream-target-weights-for-traffic-distribution.md @@ -0,0 +1,108 @@ +--- +title: How to accurately calculate upstream target weights for traffic distribution +content_type: support +description: "How Kong's upstream load balancer treats each IP address returned by a DNS A record as a separate weighted target, and how to calculate weights so traffic is distributed as intended." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why doesn't Kong's upstream weighting produce the expected traffic split when a target's hostname resolves to multiple IP addresses? + a: | + Each IP address returned by a target's DNS A record is added to the balancer as a separate target with the same weight as the original hostname target, so a hostname with multiple A records gets a multiplied effective weight. To achieve an intended ratio (e.g. 50:50), divide the desired weight by the number of resolved IP addresses, then scale all weights to whole numbers, since Kong weights must be integers. +related_resources: + - text: Kong upstream load-balancing reference documentation + url: /gateway/traffic-control/load-balancing-reference/ +--- + +## Overview + +How to accurately calculate upstream target weights for traffic distribution + +## Steps + +The behavior of the Kong upstream load-balancer mechanism is described in the documentation. The most misunderstood mechanism is the way to weight targets, which is usually due to a misunderstanding of the load balancing mechanism itself. + +Specifically, if you have a hostname that has an A record that returns multiple IP addresses, each IP address gets added to the balancer as a separate upstream target with the same weight as the initial hostname target. + +50:50 - Example + +Target 1: httpbin.org:80 + +Weight: 100 + +Returns: 200 Status + +Target 2: kong-loopback.me.test:8000 + +Weight: 100 + +Returns: 202 Status - Loopback DNS that points back to Kong and a request termination plugin + +Target setup in my test instance: + +The implication here that most believe, is that the requests should be distributed equally among the 2 targets due to their weighting. + +However if you run a test and review the Vitals information for that service, you'll see the following distribution: + +If you review the DNS A records for the 2 targets, the reason becomes clear: + +```bash +nslookup -type=A httpbin.org +Server: 172.18.0.100 +Address: 172.18.0.100:53 + +Non-authoritative answer: +Name: httpbin.org +Address: 34.231.5.222 +Name: httpbin.org +Address: 18.215.122.215 +Name: httpbin.org +Address: 44.195.242.112 +Name: httpbin.org +Address: 54.91.120.77 +Name: httpbin.org +Address: 3.229.191.75 +Name: httpbin.org +Address: 52.55.211.119 +Name: httpbin.org +Address: 52.7.224.181 +Name: httpbin.org +Address: 54.157.76.102 +``` + +```bash +nslookup -type=A kong-loopback.me.test +Server: 172.18.0.100 +Address: 172.18.0.100:53 + +Name: kong-loopback.me.test +Address: 172.18.0.8 +``` + +httpbin.org is actually 8 different IP addresses and kong-loopback only has 1. + +This means the weighting is actually 800 for httpbin.org and 100 for kong-loopback, resulting in an ~11% distribution going to the loopback, despite the apparent identical weight of both hostname targets. 100/900 = ~11.1% + +To correctly weight the above example as 50:50, the easiest way would be to set the weights like so: + +Target 1: httpbin.org:80 + +Weight: (100 weight / 8 targets = 12.5) + +Target 2: kong-loopback.me.test:8000 + +Weight: 100 + +You can not use decimals for weights as they need to be integers so 12.5 needs to be converted to a whole number with respect to the other weights. + +To do this, we multiply both weights by 10 to remove the decimal place. + +The weights then become: + +httpbin.org weight: 125 + +kong-loopback weight: 1000 + +The resulting distribution is close to 50:50: diff --git a/app/_support/how-to-add-additional-port-for-the-proxy.md b/app/_support/how-to-add-additional-port-for-the-proxy.md new file mode 100644 index 0000000000..d78f4d32f8 --- /dev/null +++ b/app/_support/how-to-add-additional-port-for-the-proxy.md @@ -0,0 +1,81 @@ +--- +title: How to add an additional port for the Kong proxy +content_type: support +description: "Add an additional port for the Kong proxy by setting `proxy_listen` in `kong.conf` or the `KONG_PROXY_LISTEN` environment variable." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can I add an additional proxy port for Kong to listen to? + a: | + Add another entry to `proxy_listen` in `kong.conf` (or the `KONG_PROXY_LISTEN` environment variable) with the desired host and port, then restart Kong. For example, adding `0.0.0.0:9000 reuseport backlog=16384` alongside the existing listeners makes Kong also listen on port 9000. +related_resources: + - text: "{{site.base_gateway}} configuration reference (`proxy_listen`)" + url: /gateway/configuration/#proxy-listen +--- + +## Overview + +How can I add an additional proxy port for Kong to listen to? + +## Steps + +You can add an additional proxy port by adding the port to `proxy_listen` in `kong.conf`, or by using the environment variable `KONG_PROXY_LISTEN`. + +For more details, see the configuration reference. + +Example: + +If you want to add an additional http proxy port that listens on 9000, you would set `proxy_listen` as below and restart kong afterward. + +```bash +proxy_listen=0.0.0.0:8000 reuseport backlog=16384, 0.0.0.0:8443 http2 ssl reuseport backlog=16384 , 0.0.0.0:9000 reuseport backlog=16384 +``` + +Kong now will listen to 8000 and 9000 and you should be able to send an http request to these ports now. + +``` +[~] http http://localhost:8000 +HTTP/1.1 404 Not Found +Date: Thu, 06 Aug 2026 14:44:59 GMT +Content-Type: application/json; charset=utf-8 +Connection: keep-alive +Content-Length: 103 +X-Kong-Response-Latency: 0 +Server: kong/3.14.0.0-enterprise-edition +X-Kong-Request-Id: bba38b6151d61f0bb93ff01f60825d2c + +{ + "message":"no Route matched with those values", + "request_id":"bba38b6151d61f0bb93ff01f60825d2c" +} + +[~] http http://localhost:9000 +HTTP/1.1 404 Not Found +Date: Thu, 06 Aug 2026 14:44:59 GMT +Content-Type: application/json; charset=utf-8 +Connection: keep-alive +Content-Length: 103 +X-Kong-Response-Latency: 1 +Server: kong/3.14.0.0-enterprise-edition +X-Kong-Request-Id: 7cd5d07f69a88a4db8fa6e087854241b + +{ + "message":"no Route matched with those values", + "request_id":"7cd5d07f69a88a4db8fa6e087854241b" +} +``` + +Additional Note: + +For https, you could just copy the existing configuration in `proxy_listen` and change the port. + +Example: + +Config below will add 9443 as an https port: + +```bash +proxy_listen=0.0.0.0:8000 reuseport backlog=16384, 0.0.0.0:8443 http2 ssl reuseport backlog=16384 , 0.0.0.0:9443 http2 ssl reuseport backlog=16384 +``` diff --git a/app/_support/how-to-add-status-code-to-tracing-spans-with-opentelemetry-plugin.md b/app/_support/how-to-add-status-code-to-tracing-spans-with-opentelemetry-plugin.md new file mode 100644 index 0000000000..cd7bb584bf --- /dev/null +++ b/app/_support/how-to-add-status-code-to-tracing-spans-with-opentelemetry-plugin.md @@ -0,0 +1,44 @@ +--- +title: How to add status code to tracing spans with the OpenTelemetry plugin +content_type: support +description: Add a `pre-function` plugin that sets the `http.status_code` attribute on the OpenTelemetry root tracing span, since the OpenTelemetry plugin doesn't include it by default. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I add a status code attribute to OpenTelemetry tracing spans in Kong? + a: | + Kong's OpenTelemetry plugin doesn't include a status code attribute in spans by default. Add a `pre-function` plugin that calls `kong.tracing.active_span():set_attribute("http.status_code", kong.response.get_status())` in `config.header_filter` — but note this only reaches the true root span when `tracing_instrumentations` is set to a lighter level like `request` or `router`; at `tracing_instrumentations = all` it attaches to a nested child span instead. +related_resources: + - text: Built-in tracing instrumentations + url: /plugins/opentelemetry/#built-in-tracing-instrumentations + - text: Customize OpenTelemetry spans as a developer + url: /plugins/opentelemetry/#create-a-custom-span +--- + +## Overview + +How to add status code to tracing spans with OpenTelemetry plugin? + +## Steps + +By default, the OpenTelemetry plugin will not include a status code attribute in spans. + +We could apply a `pre-function` plugin to include status code attribute in spans. + +Below is an example configuration of the `pre-function` plugin + +```lua + +scope: global or the same service/route entity with OpenTelemetry plugin + +config.header_filter: +local root_span = kong.tracing.active_span() +root_span:set_attribute("http.status_code", kong.response.get_status()) +``` + +Next send some requests through kong and confirm the `http.status_code` attribute is in the root span. + +Note: `kong.tracing.active_span()` only reliably returns the true root span at lighter `tracing_instrumentations` levels (e.g. `request` or `router`). At `tracing_instrumentations = all`, the active span in the `header_filter` phase is a nested per-plugin child span rather than the root span, so setting the attribute this way no longer attaches it to the root span in that configuration. diff --git a/app/_support/how-to-allow-anonymous-access-to-a-route-using-the-oidc-plugin.md b/app/_support/how-to-allow-anonymous-access-to-a-route-using-the-oidc-plugin.md new file mode 100644 index 0000000000..d9d696b884 --- /dev/null +++ b/app/_support/how-to-allow-anonymous-access-to-a-route-using-the-oidc-plugin.md @@ -0,0 +1,315 @@ +--- +title: How to allow anonymous access to a route using the OIDC plugin +content_type: support +description: Configure the OpenID Connect plugin's anonymous consumer so that requests without a valid token fall through to an anonymous consumer instead of being rejected. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I allow anonymous access to a route that has the OIDC plugin applied? + a: | + Create a consumer to represent anonymous access, then set the OpenID Connect plugin's `config.anonymous` to that consumer's ID. + Requests with a valid token authenticate as the real consumer; requests without one fall through to the anonymous consumer instead of being rejected. This works whether the plugin is applied at the service or route level. +related_resources: [] +--- + +## Overview + +How to enable anonymous access to a route when an OIDC plugin is installed at the service level or the route level + +## Steps + +Here's an example of a service with a route that allows access for an anonymous consumer. + +We have a service that’s configured like so: + +``` +❯ curl -k -H "Kong-Admin-Token:password" https://api.kong.lan/services/e399a95c-5a02-4594-814e-8d364167903e | jq 'with_entries(select(.value != null))' +{ + "created_at": 1680192808, + "id": "e399a95c-5a02-4594-814e-8d364167903e", + "retries": 5, + "write_timeout": 60000, + "name": "local-echo-server", + "connect_timeout": 60000, + "read_timeout": 60000, + "updated_at": 1680192808, + "protocol": "http", + "host": "echo-server", + "port": 8080, + "enabled": true +} +``` + +and this route: + +``` +❯ curl -k -H "Kong-Admin-Token:password" https://api.kong.lan/services/e399a95c-5a02-4594-814e-8d364167903e/routes/auth-oidc | jq 'with_entries(select(.value != null))' +{ + "created_at": 1680192808, + "paths": [ + "/auth/oidc" + ], + "https_redirect_status_code": 426, + "regex_priority": 0, + "name": "auth-oidc", + "path_handling": "v0", + "request_buffering": true, + "response_buffering": true, + "service": { + "id": "e399a95c-5a02-4594-814e-8d364167903e" + }, + "strip_path": true, + "preserve_host": false, + "id": "4625ebe0-8287-44a2-8cc0-27a182e5a06d", + "protocols": [ + "http", + "https" + ], + "updated_at": 1680192808 +} +``` + +An “anonymous” consumer has also been configured and looks like this: + +``` +❯ curl -k -H "Kong-Admin-Token:password" https://api.kong.lan/consumers/anonymous | jq 'with_entries(select(.value != null))' +{ + "created_at": 1680192808, + "id": "b4d6a39a-5dc6-4ce8-b85a-fee580b5513b", + "type": 0, + "username": "anonymous", + "username_lower": "anonymous" +} +``` + +and that the oidc-config + +``` +"created_at": 1680192883, + "id": "d7f831f7-0323-49f3-8f87-1a55a127bca0", + "name": "openid-connect", + "service": { + "id": "e399a95c-5a02-4594-814e-8d364167903e" + }, + ... + "enabled": true, + "config": { + ... + "introspection_check_active": true, + "login_methods": [ + "authorization_code" + ], + "session_strategy": "default", + "session_memcache_prefix": "sessions", + "cache_token_exchange": true, + "session_compressor": "none", + "preserve_query_args": false, + "authorization_cookie_samesite": "off", + "forbidden_error_message": "Forbidden", + "unauthorized_error_message": "Unauthorized", + "userinfo_accept": "application/json", + "upstream_access_token_header": "authorization:bearer", + "authorization_cookie_httponly": true, + "authorization_cookie_path": "/", + "authorization_cookie_lifetime": 600, + "authorization_cookie_name": "authorization", + "credential_claim": [ + "sub" + ], + "issuer": "http://keycloak.kong.lan:8080/auth/realms/kong", + "refresh_token_param_type": [ + "header", + "query", + "body" + ... + "forbidden_destroy_session": true, + "session_redis_ssl_verify": false, + "logout_revoke_access_token": true, + "logout_revoke_refresh_token": true, + "logout_revoke": false, + "cache_tokens_salt": "rDrzmZtubsQt6ZRTWldsi8Y01FCNfiIJ", + "login_redirect_mode": "fragment", + "session_cookie_maxsize": 4000, + "session_cookie_httponly": true, + "session_cookie_path": "/", + ... + ], + "session_cookie_lifetime": 3600, + "scopes_claim": [ + "scope" + ], + "verify_nonce": true, + "response_mode": "query", + "resolve_distributed_claims": false, + "enable_hs_signatures": false, + "session_cookie_name": "session", + "cache_ttl": 3600, + "groups_claim": [ + "groups" + ], + "scopes_required": [ + "profile", + "email", + "nope-scope" + ], + "rediscovery_lifetime": 30, + "audience_claim": [ + "aud" + ], + ... + "scopes": [ + "openid" + ], + "session_redis_prefix": "sessions", + "auth_methods": [ + "bearer" + ], + "introspection_accept": "application/json", + "session_memcache_host": "127.0.0.1", + "cache_introspection": true, + "refresh_tokens": true, + "anonymous": "b4d6a39a-5dc6-4ce8-b85a-fee580b5513b", + "run_on_preflight": true, + "consumer_by": [ + "username", + "custom_id" + ], + "verify_claims": true, + "verify_parameters": false, + "consumer_claim": [ + "email" + ], + "consumer_claims": [ + ["email"] + ], +... + } +} +``` + +includes: + +``` +"config": { +... +... + "anonymous": "b4d6a39a-5dc6-4ce8-b85a-fee580b5513b", +... + } +``` + +If we access the route with the correct token, the echo service would return an output that looks like this: + +``` +> curl -H "Authorization: Bearer $TOKEN" http://proxy.kong.lan/auth/oidc + +Request served by 97f243d248ea + +HTTP/1.1 GET / + +Host: echo-server:8080 +Connection: keep-alive +X-Forwarded-Path: /auth/oidc +Accept: */* +Authorization: Bearer XXX +X-Consumer-Custom-Id: stu+keycloak_user@konghq.com +X-Consumer-Username: oidc-consumer +X-Consumer-Id: accb0537-af5f-4d29-8948-e78eaf0e677f +X-Forwarded-For: 172.28.0.1, 172.28.0.16 +X-Forwarded-Prefix: /auth/oidc +X-Real-Ip: 172.28.0.1 +X-Forwarded-Proto: http +X-Forwarded-Host: proxy.kong.lan +X-Forwarded-Port: 48000 +User-Agent: curl/7.86.0 +``` + +If however a wrong token is used, this is what's returned: + +``` +❯ curl -H "Authorization: Bearer $XXXX" http://proxy.kong.lan/auth/oidc + +Request served by 97f243d248ea + +HTTP/1.1 GET / + +Host: echo-server:8080 +X-Forwarded-Proto: http +X-Forwarded-Host: proxy.kong.lan +X-Forwarded-Path: /auth/oidc +X-Consumer-Id: b4d6a39a-5dc6-4ce8-b85a-fee580b5513b +X-Consumer-Username: anonymous +Connection: keep-alive +X-Forwarded-Prefix: /auth/oidc +Authorization: Bearer +User-Agent: curl/7.86.0 +Accept: */* +X-Anonymous-Consumer: true +X-Forwarded-For: 172.28.0.1, 172.28.0.16 +X-Forwarded-Port: 48000 +X-Real-Ip: 172.28.0.1 +``` + +Note that the same plugin could be removed from the service and be applied only to the route, and the behavior would remain the same, i.e. the anonymous consumer will act as a “catch-all” case for non-authorized calls to the route. + +```bash +curl -k -H "Kong-Admin-Token:password" https://api.kong.lan/routes/auth-oidc/plugins + + "data": [ + { + "created_at": 1680192883, + "id": "d7f831f7-0323-49f3-8f87-1a55a127bca0", + "route": { + "id": "4625ebe0-8287-44a2-8cc0-27a182e5a06d" + }, + "name": "openid-connect", + "protocols": [ + "grpc", + "grpcs", + "http", + "https" + ], + "enabled": true, + "config": { + "display_errors": false, + "response_type": [ + "code" +... +... + "session_redis_prefix": "sessions", + "auth_methods": [ + "bearer" + ], + "introspection_accept": "application/json", + "session_memcache_host": "127.0.0.1", + "cache_introspection": true, + "refresh_tokens": true, + "anonymous": "b4d6a39a-5dc6-4ce8-b85a-fee580b5513b", + "run_on_preflight": true, + "consumer_by": [ + "username", + "custom_id" + ], + "verify_claims": true, + "verify_parameters": false, + "consumer_claim": [ + "email" + ], + "consumer_claims": [ + ["email"] + ], + "session_storage": "cookie", + "leeway": 0, + "http_version": 1.1, + "session_cookie_samesite": "Lax", + "cache_tokens": true, + "cache_user_info": true, + "ssl_verify": false + } + } + ] +} +``` diff --git a/app/_support/how-to-avoid-openid-connect-plugin-obtain-a-token-from-cache-that-is-almost-expired.md b/app/_support/how-to-avoid-openid-connect-plugin-obtain-a-token-from-cache-that-is-almost-expired.md new file mode 100644 index 0000000000..d25b79d94c --- /dev/null +++ b/app/_support/how-to-avoid-openid-connect-plugin-obtain-a-token-from-cache-that-is-almost-expired.md @@ -0,0 +1,59 @@ +--- +title: How to avoid the OpenID Connect plugin obtaining a token from cache that is almost expired +content_type: support +description: Explains how to set `config.cache_ttl_max` on the OpenID Connect plugin so Kong requests a new access token before the cached token expires. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I avoid the OpenID Connect plugin obtaining a token from cache that is almost expired? + a: | + Set `config.cache_ttl_max` on the OpenID Connect plugin to a value lower than the access token's actual lifetime (as configured in the IDP). Kong will then request a new token from the IDP once the cached token's age passes that threshold, instead of reusing a token that is about to expire. For example, with a 180s access token lifetime, setting `config.cache_ttl_max=165` gets a new token 15s before expiry. +related_resources: + - text: OpenID Connect plugin documentation + url: /plugins/openid-connect/ +--- + +## Overview + +How to avoid the OpenID Connect plugin obtaining a token from cache that is almost expired? Is there a setting where kong can obtain a new token just 10 or 15 secs before the cached token expired? + +## Steps + +Yes, it is possible by setting the `config.cache_ttl_max` to limit the max TTL for cache. + +"Whether kong will get new token from IDP" depends on below 2 factor + +(Here the IDP means OpenID Connect providers like Okta/Keycloak/AzureAD...) + +(1) access token lifetime configured in the IDP side(2) `config.cache_tokens` configured in OpenID Connect plugin + +Firstly please set `config.cache_tokens=true` in OpenID Connect plugin to enable cache function. + +And kong will cache the token until it expired. + +Next please confirm the access token lifetime in the IDP side and set `config.cache_ttl_max` + +to avoid obtaining a token from cache that is almost expired. + +For example, assuming the access token lifetime is 180s and we want to get a new token 15s before the access token expire. Then we need to set `config.cache_ttl_max=165`. (180-15=165) + +The behavior as below + +``` + +1. We send first request to Kong at 0s, +OpenID Connect plugin will get a new access token from IDP and cache it. +This access token is valid for 180s. + +2. We send request again to Kong at 150s, +then OpenID Connect plugin will get the existing access token from cache. +Here the access token still valid for 30s(180-150=30). + +3. We send request again to Kong at 166s, +then OpenID Connect plugin will get a new access token from IDP. +Because 166s is larger than 165s which has exceed `config.cache_ttl_max`. +This new access token is valid for 180s. +``` diff --git a/app/_support/how-to-check-dns-querying-when-installed-kong-in-k8s.md b/app/_support/how-to-check-dns-querying-when-installed-kong-in-k8s.md new file mode 100644 index 0000000000..0f778a1c92 --- /dev/null +++ b/app/_support/how-to-check-dns-querying-when-installed-kong-in-k8s.md @@ -0,0 +1,153 @@ +--- +title: How to check DNS querying when Kong is installed in k8s +content_type: support +description: Explains how to inspect the DNS queries Kong makes in Kubernetes by enabling query logging in the cluster's CoreDNS configuration. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I check the DNS queries Kong makes when it's installed in Kubernetes? + a: | + Kong resolves DNS through the cluster's CoreDNS service, which doesn't log query details by default. Add the `log` and `whoami` plugins to CoreDNS's `Corefile` (in the `coredns` ConfigMap in the `kube-system` namespace), apply the updated ConfigMap, and restart the CoreDNS deployment. Then tail the CoreDNS pod logs to see each DNS query and its result, for example `NOERROR` for a successful lookup or `NXDOMAIN` when the name doesn't resolve. +related_resources: + - text: CoreDNS log plugin documentation + url: https://coredns.io/plugins/log/#examples +--- + +## Overview + +When Kong is installed in k8s, kong will send DNS query to DNS server. How can we check DNS query results? + +## Steps + +K8S uses `coredns` pods as DNS server by default, you could find `coredns` in the `kube-system` namespace as below + +```bash + +kubectl get all -n kube-system +NAME READY STATUS RESTARTS AGE +pod/coredns-558bd4d5db-9tmw2 1/1 Running 1 64d +pod/coredns-558bd4d5db-bzlnq 1/1 Running 2 64d + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 64d + +NAME READY UP-TO-DATE AVAILABLE AGE +deployment.apps/coredns 2/2 2 2 64d + +NAME DESIRED CURRENT READY AGE +replicaset.apps/coredns-558bd4d5db 2 2 2 64d +``` + +Step 0: Let's create below upstream/target/service/route for testing + +```bash + +# create httpbin upstream +curl :/upstreams -H "kong-admin-token:admin" \ +-d 'name=httpbin' + +# create target with existing service: http://httpbin.org:80 +curl :/upstreams/httpbin/targets -H "kong-admin-token:admin" \ +-d 'target=httpbin.org:80' + +# create target with dummy service: http://httpbin.org.dummy:80 +curl :/upstreams/httpbin/targets -H "kong-admin-token:admin" \ +-d 'target=httpbin.org.dummy:80' + +# create service +curl :/services -H "kong-admin-token:admin" \ +-d 'name=httpbin' \ +-d 'host=httpbin' \ +-d 'path=/anything' + +# create route +curl :/services/httpbin/routes -H "kong-admin-token:admin" \ +-d 'name=httpbin' \ +-d 'paths=/test' + +# testing +curl :/test -i +>200 response +``` + +`coredns` does not show DNS query access and result in log by default. + +We need to modify `coredns` configuration as below + +Step 1: Get original `coredns` config + +```bash + +kubectl get configmap coredns -n kube-system -o yaml > coredns.yaml +``` + +Step 2: Modify `coredns` config as below + +# Please modify this file based on your environment. you have to add "log" and "`whoami`" into it. + +```yaml + +vi coredns.yaml + +apiVersion: v1 +kind: ConfigMap +metadata: + name: coredns + namespace: kube-system +data: + Corefile: | + .:53 { ++ log ++ whoami + errors + health { + lameduck 5s + } + ready + kubernetes cluster.local in-addr.arpa ip6.arpa { + pods insecure + fallthrough in-addr.arpa ip6.arpa + ttl 30 + } + prometheus :9153 + forward . /etc/resolv.conf { + max_concurrent 1000 + } + cache 30 + loop + reload + loadbalance + } +``` + +Step 3: Apply new `coredns` config + +```bash + +kubectl apply -f coredns.yaml +configmap/coredns configured + +# Restart coredns +kubectl rollout restart deployments/coredns -n kube-system +deployment.apps/coredns restarted +``` + +Step 4: Check `coredns` log + +```bash + +kubectl logs -f -n kube-system + +Then you will be able to see below log +[INFO] 10.1.3.45:49488 - 31240 "A IN httpbin.org. udp 29 false 512" NOERROR qr,aa,rd,ra 191 0.0001016s + +It shows query result for "A IN httpbin.org" is "NOERROR"(success) and DNS response time is 0.0001016s. + +You will be able to see below log too +[INFO] 10.1.3.46:43281 - 44662 "A IN httpbin.org.dummy. udp 35 false 512" NXDOMAIN qr,rd,ra 35 0.0079283s + +It shows query result for "A IN httpbin.org.dummy" is "NXDOMAIN"(not found) and DNS response time is 0.0079283s +``` diff --git a/app/_support/how-to-check-whether-kong-is-oss-version-or-enterprise-version.md b/app/_support/how-to-check-whether-kong-is-oss-version-or-enterprise-version.md new file mode 100644 index 0000000000..77ecc7e4bd --- /dev/null +++ b/app/_support/how-to-check-whether-kong-is-oss-version-or-enterprise-version.md @@ -0,0 +1,41 @@ +--- +title: How to check whether Kong is OSS version or enterprise version +content_type: support +published: false +description: Explains how to confirm whether a {{site.base_gateway}} installation is OSS or Enterprise using Kong Manager, the `kong version` command, or proxy logs. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can I check whether my Kong installation is OSS or Enterprise? + a: | + Check the info panel in Kong Manager, run `kong version` (which prints "Enterprise" before the version number for Enterprise installations), or inspect the Kong proxy logs. +related_resources: [] +--- + +## Overview + +How to check whether Kong is using an OSS version or an Enterprise version + +## Steps + +There are several methods to check whether Kong is using an OSS or Enterprise version. + +Method 1. By Kong Manager. + +Open the Kong Manager and click the 'i' mark on the right top. + +It will show you detailed information about the Kong you are using. + +Method 2. By the `kong version` command. + +For example, the result shows "Enterprise" before the version for an Enterprise version. + +```bash +kubectl exec -it -c proxy -n -- kong version +Kong Enterprise 3.14.0.0 +``` + +Method 3. By checking Kong proxy logs. diff --git a/app/_support/how-to-configure-a-certificate-key-with-secret-management-and-environment-variable-backend.md b/app/_support/how-to-configure-a-certificate-key-with-secret-management-and-environment-variable-backend.md new file mode 100644 index 0000000000..1e1faf0bbf --- /dev/null +++ b/app/_support/how-to-configure-a-certificate-key-with-secret-management-and-environment-variable-backend.md @@ -0,0 +1,75 @@ +--- +title: How to configure a certificate key with Secrets Management and the Environment Variable backend +content_type: support +description: When referencing a certificate key from an environment variable with Kong's Secrets Management feature, set the certificate's `key` field to `{vault://env/}` in your declarative configuration. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I configure a certificate key using Kong's Secrets Management Environment Variable backend? + a: | + Set the certificate's `key` field to `{vault://env/}` in your declarative config (or Kong Manager) and export the referenced environment variable with the real PEM contents — e.g. `export ENV_REFERENCE_API_KEY="$(cat key.pem)"` — since a single-quoted string with literal `\n` escapes won't produce real newlines and will fail to parse. +related_resources: [] +--- + +## Overview + +We are trying to test the beta Secrets Management feature using the Environment Variable backend but we can not get a certificate key to work. What would an example look like? + +## Steps + +If you have a declarative configuration file to be used with deck or in a db-less configuration, you need to make sure to configure the certificate key as `{vault://env/}` + +The declarative config file should look like this: + +```yaml + +- cert: |- + -----BEGIN CERTIFICATE----- + MIIFeDCCBGCgAwIBAgIUAusYGP9BwoLFFAJdB/jY6eUzUyQwDQYJKoZIhvcNAQEL + BQAwgZIxCzAJBgNVBAYTAlVLMRIwEAYDVQQIDAlIYW1wc2hpcmUxEjAQBgNVBAcM + CUFsZGVyc2hvdDEQMA4GA1UECgwHS29uZyBVSzEQMA4GA1UECwwHU3VwcG9ydDEY + MBYGA1UEAwwPU3VwcG9ydCBSb290IENBMR0wGwYJKoZIhvcNAQkBFg5zdHVAa29u + Z2hxLmNvbTAeFw0yMTAxMjAxNTA0NDVaFw0yMjAxMjAxNTA0NDVaMIGQMQswCQYD + VQQGEwJVSzESMBAGA1UECAwJSGFtcHNoaXJlMRIwEAYDVQQHDAlBbGRlcnNob3Qx + EDAOBgNVBAoMB0tvbmcgVUsxEDAOBgNVBAsMB1N1cHBvcnQxFjAUBgNVBAMMDW10 + bHMtY29uc3VtZXIxHTAbBgkqhkiG9w0BCQEWDnN0dUBrb25naHEuY29tMIICIjAN + BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA1/+83/YNiEVKYvcuVwYGve6afsg1 + BYCn1+E9Uwgh0uwAenT/DKB8NhqoVxc7cZ2HaTI146IGmFICmctlTWvLPLglHmTo + byOUV6tIJAjvzyEOpC458hLGgbv8mhGXJWPxBVu7Wy6Hapz2bk0cEscfL7PHKaRu + 3D6r8/zbhhWAqe4EIt+NVYT6baaYBs7bPZQXs/sluKI+DNYuDeaAmoSuCc4ein6z + 0xDqCSMmPebzjns03ttB29vWL3eYY9dvgoCd+CPhXT/C4CHtvKbH+hOQYDtVF6MO + 1mmABAQTQWMR/00+QI0xtvuXtEPurla5dA0TN6ddCTOOcILKx62z5oc3Kqr+nHHa + 71zNzARUVaZ2vy1pRVr0DZgB7KqcFXhy/oy8IpmxUR1ASBDZl6B6RKrdQwvgLgmn + 3M/roNLAU+3nz4itpt/zf+X0suwdthrflic1R68z1SlYbyoGARWkZ/pOl6kLNVK2 + OsqQuICaajnW7t1oDd7z1+3hm+uoryDwvG6f3T9ZvWjKXYcKg7b+BjbFdahbDywD + PgnhSz9AaoVWhR+GHIPrjRClMpEkra/yGJFvH3UpXhgg9d0DrLZE51Z75a9SvnAj + vdLuNhx4bJbwLBgNGsJMkupzBrw4iCfbKFcBbP8o0Xjtarj7T/mkWuQ1GjWqfyrD + 55NecBPNw5C9BR0CAwEAAaOBxTCBwjAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQE + AwIFoDAzBglghkgBhvhCAQ0EJhYkT3BlblNTTCBHZW5lcmF0ZWQgQ2xpZW50IENl + cnRpZmljYXRlMB0GA1UdDgQWBBSV3F+eicU8SVT4LcDJ6eMzP0todzAfBgNVHSME + GDAWgBR2ySl/TAlFDGO3NAVlyJaZR+XZtzAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0l + BBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMA0GCSqGSIb3DQEBCwUAA4IBAQB5L0OZ + WELG9Pw6Ol1BsZYgpLR4PGNBB9dKm/9dd+q+ohZVFCfXcjZ3YOU1vh/HHQrALRNY + I58JxcVCOx/qIW2uA0iSCqIT0sNb9cJLxfZf7X+BzPPPnu0ugUJp7GzLNnHitrLC + Xb1nmmefwgraNzp+a5IrR8RcQG1mYDuS+2HK/rybo22XcCxhob8OiDEn8+ytkKyQ + Ipmrf9D+/68/ih6az0w1aakASMmFe8z/p6VgVQkCySCWWFG525BRdGmSImqVZ4xa + aQFN3L+oN+JJcCFTthLOAYo32JH+xLMz7PokzSL84g3b68h59hXDoMSwB10GthL5 + T8tqV6i5miKWwvfZ + -----END CERTIFICATE----- + id: f3ae1bb2-ea6a-4caf-a7a7-2f078b7842db + key: '{vault://env/env_reference_api_key}' +``` + +In Kong Manager the certificate key needs to be entered as in the below screenshot: + +The environment variable `ENV_REFERENCE_API_KEY` can be configured in the preferred way depending on the set up, e.g. set in a Linux shell. The value must contain real newline characters, not a literal `\n`-escaped single-quoted string — a single-quoted shell string with literal `\n` sequences does not produce actual newlines, and the resulting PEM will fail to parse (e.g. `PEM_read_bio_PrivateKey() failed`, causing the TLS handshake to fail). The safest way to set this is to load the key file directly so real newlines are preserved: + +```bash + +export ENV_REFERENCE_API_KEY="$(cat key.pem)" +``` + +or set in a docker container etc., making sure the value is passed with real newlines rather than escaped `\n` sequences. diff --git a/app/_support/how-to-configure-a-kong-service-to-perform-upstream-mtls.md b/app/_support/how-to-configure-a-kong-service-to-perform-upstream-mtls.md new file mode 100644 index 0000000000..e79d243401 --- /dev/null +++ b/app/_support/how-to-configure-a-kong-service-to-perform-upstream-mtls.md @@ -0,0 +1,84 @@ +--- +title: How to configure a Kong service to perform upstream MTLS +content_type: support +description: Configure a Kong service for upstream MTLS by setting a client certificate and CA certificate at the service level or globally. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I configure a Kong service to perform mutual TLS with an upstream? + a: | + Upstream MTLS needs a client certificate to present and a CA certificate to verify the upstream. + Per service, upload the client cert and set `config.client_certificate`, and upload the CA cert and set `config.ca_certificates`. To apply globally instead, use `KONG_CLIENT_SSL_CERT` / `KONG_CLIENT_SSL_CERT_KEY` and `KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE`. +related_resources: [] +--- + +## Overview + +How can Kong services be configured to negotiate MTLS with an upstream? + +## Steps + +MTLS with an upstream service requires several components: + +- The ability to provide a client certificate with the proxied request. +- The ability to verify the upstream's presented server certificate. + +1. Set up the client certificate (service level). + + a) Install the client certificate into Kong Manager's Certificates section or via the Admin API. See the Certificate API documentation. + + Example: + + ```bash + curl -X POST \ + http://localhost:8001/certificates \ + -H 'Content-Type: multipart/form-data' \ + -F cert=@./client-cert.pem \ + -F key=@./client-cert.key \ + -H "kong-admin-token:admin" + ``` + + If installing the certificate via CLI / Admin API, a successful response will include a certificate object id. Kong Manager will also show this ID and allow it to be retrieved. + + b) Use the certificate ID obtained in step 1, and enter it into the `config.client_certificate` field in the service. See the Service API documentation. + + Whenever a request is proxied to that service, the certificate associated with that id will be used as a client certificate. + + Further reading: Why do I have to add the private key with my client certificate? + +2. Set up the CA certificate (service level). + + a) Install the CA certificate as in the previous step 1a, except use the CA Certificates API this time. + + Example: + + ```bash + curl -X POST \ + http://localhost:8001/ca_certificates \ + -H 'Content-Type: multipart/form-data' \ + -F cert=@./ca-cert.pem \ + -H "kong-admin-token:admin" + ``` + + Note: You only need to add the certificate(s) for CA Certs. Private keys are not required / accepted. + + b) As with step 1b, use the CA certificate object id and enter it into the `config.ca_certificates` field in the service. + + When Kong does TLS negotiation with the upstream service, the CA Cert associated with that id will be used to verify the upstream server cert. + +3. Set up the client certificate (global). + + Nginx can use 2 directives to set a global client certificate which will attach to all services. + + `KONG_CLIENT_SSL_CERT` and `KONG_CLIENT_SSL_CERT_KEY` can be used to indicate the path of the certificate and key that will be used with client certificate negotiation. + +4. Set up the CA certificate (global). + + Nginx can use a directive to set a CA cert to verify all upstream server certificates without needing to modify all service configs. + + `KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE` contains the path to a PEM file that can hold multiple CA Root certificates for verifying ALL upstream server certificates. + +Further reading: Defining SSL certs and where to use them diff --git a/app/_support/how-to-configure-kong-aws-lambda-plugin-to-return-the-same-4xx-status-codes-sent-by-an-aws-lambda-function-back-to-the-client.md b/app/_support/how-to-configure-kong-aws-lambda-plugin-to-return-the-same-4xx-status-codes-sent-by-an-aws-lambda-function-back-to-the-client.md new file mode 100644 index 0000000000..d06d1629ce --- /dev/null +++ b/app/_support/how-to-configure-kong-aws-lambda-plugin-to-return-the-same-4xx-status-codes-sent-by-an-aws-lambda-function-back-to-the-client.md @@ -0,0 +1,49 @@ +--- +title: Configuring the Kong AWS Lambda plugin to return the same 4xx status codes sent by an AWS Lambda function back to the client +content_type: support +description: Explains how to enable `is_proxy_integration` on the Kong AWS Lambda plugin so it returns the same status codes sent by the AWS Lambda function to the client. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I configure the Kong AWS Lambda plugin to return the same 4xx status codes sent by an AWS Lambda function back to the client? + a: | + Enable `is_proxy_integration` on the Kong AWS Lambda plugin so it treats Lambda responses the way an AWS API Gateway Lambda proxy integration would, preserving the status code. Also make sure your Lambda function returns a proxy-integration-shaped response (a `statusCode` field, plus `headers` and `body`). +related_resources: [] +--- + +## Overview + +How to configure Kong AWS Lambda plugin to return the same 4xx status codes sent by an AWS Lambda function back to the client? + +## Steps + +To ensure Kong AWS Lambda plugin returns the same 4xx, 5xx, 2xx or any status codes sent by an AWS Lambda function back to the client, you need to enable the `is_proxy_integration` configuration in the Kong AWS Lambda plugin. This setting allows Kong to properly interpret and forward the response/status codes received from the Lambda function. + +Here's a step-by-step guide to enable this configuration: + +1. Locate the AWS Lambda plugin configuration in your Kong setup. + +2. Modify the plugin configuration to enable or include `"is_proxy_integration": true`. This tells Kong to handle the response from AWS Lambda as if it were coming from an AWS API Gateway configured with Lambda proxy integration, which includes preserving the status code. + +3. Ensure your Lambda function is returning the correct response structure for proxy integration. Here's an example AWS Lambda function: + + ```python + def lambda_handler(event, context): + return { + "statusCode": 400, # or any other 4xx/5xx code as per your logic + "headers": { + "Content-Type": "application/json" + }, + "body": json.dumps({ + "message": "Hello, this is a RESTful response!", + "data": event + }) + } + ``` + +4. After making these changes, deploy the updated plugin configuration and test to ensure that the correct status codes are being returned to the client. + +By following these steps, Kong will forward the status codes from AWS Lambda function to the client, preserving the application's intended response behavior. diff --git a/app/_support/how-to-configure-rate-limiting-by-consumer-from-the-openid-connect-plugin.md b/app/_support/how-to-configure-rate-limiting-by-consumer-from-the-openid-connect-plugin.md new file mode 100644 index 0000000000..91169b8a55 --- /dev/null +++ b/app/_support/how-to-configure-rate-limiting-by-consumer-from-the-openid-connect-plugin.md @@ -0,0 +1,35 @@ +--- +title: How to configure rate limiting by consumer from the openid-connect plugin +content_type: support +description: Configure rate limiting by consumer from the openid-connect plugin using `config.credential_claim` without mapping consumers in Kong. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I rate limit by consumer from the openid-connect plugin without mapping consumers in Kong? + a: | + Keep the rate limiting plugin on its default `config.limit_by = consumer`, and leave `config.consumer_claim` empty with `config.consumer_optional = false` so consumers aren't mapped in Kong. + The openid-connect plugin then rate limits by the token's `sub` claim via `config.credential_claim` (default `sub`); point it at another claim, such as `tenant`, if you identify the consumer differently. +related_resources: [] +--- + +## Overview + +When configuring rate limiting by consumer, the consumer should be taken from the subject authenticated by the openid-connect plugin, but the consumer should not exist in Kong. How can this be configured? + +## Steps + +Regarding the rate limiting (or rate limiting advanced) plugin, you need to make sure it has configured `config.limit_by = consumer` (default value). Then, the openid-connect plugin by default tries to search claim `sub` inside the token and if that exists, the rate limiting plugin can rate limit by that, which is configurable with `config.credential_claim`. + +`sub` is the Subject Identifier: a locally unique and never reassigned identifier within the Issuer for the End-User, which is intended to be consumed by the Client. + +You can also use `config.consumer_claim` and `config.consumer_optional` and then apply rate limit to specific consumers. So, if it is not required to map the consumers in Kong, you need to set: + +``` +config.consumer_claim = // empty, default value +config.consumer_optional = false // default value - you don't want to map the consumers in Kong +``` + +Then, you also need to make sure the token includes the `sub` claim. If so, you can keep the default value `config.credential_claim = sub` - but if you identify the consumer by the tenant for example you should set `config.credential_claim = tenant`. diff --git a/app/_support/how-to-configure-upstream-healthchecks-with-the-ingress-controller.md b/app/_support/how-to-configure-upstream-healthchecks-with-the-ingress-controller.md new file mode 100644 index 0000000000..a499eed7a9 --- /dev/null +++ b/app/_support/how-to-configure-upstream-healthchecks-with-the-ingress-controller.md @@ -0,0 +1,79 @@ +--- +title: How to configure upstream healthchecks with the Ingress Controller +content_type: support +description: Explains how to configure upstream circuit breakers and active healthchecks through the {{site.kic_product_name}}, covering both the legacy `KongIngress` resource (KIC 2.5/2.12 LTS) and the annotation-based approach (KIC 3.1+). +products: + - kic +works_on: + - on-prem + - konnect +tldr: + q: How do I configure upstream circuit breakers and active healthchecks with the {{site.kic_product_name}}? + a: | + For KIC 2.5 LTS/2.12 LTS, define a `KongIngress` resource with `upstream.healthchecks` (passive and active) and reference it from the Kubernetes Service using the `konghq.com/override` annotation. For KIC 3.1 and onwards, `KongIngress` is removed in favor of annotations and the `KongUpstreamPolicy` custom resource, which cover the same healthcheck configuration. +related_resources: + - text: Gateway Upstream healthchecks documentation + url: /gateway/traffic-control/health-checks-circuit-breakers/ + - text: KongIngress upstream-policy annotation reference (KIC 3.1+) + url: /kubernetes-ingress-controller/reference/annotations/#konghqcomupstream-policy + - text: KongUpstreamPolicy custom resource reference (KIC 3.1+) + url: /kubernetes-ingress-controller/reference/custom-resources/#kongupstreampolicy +--- + +## Overview + +When using the Kubernetes Ingress Controller and defining the Kong Objects using the Kubernetes Ingress resource, how can I define circuit breakers and active healthchecks? + +## Steps + +### For KIC versions 2.5 LTS and 2.12 LTS + +If you need to configure circuit breakers and active healthchecks in Kubernetes, you can do it using the `KongIngress` resource object and annotating the Kubernetes Service object so that it references the defined `KongIngress`. As this is changing the upstream parameters, the annotation needs to go on the Kubernetes Service and not on the Ingress. + +Once the Service has been annotated to use the `KongIngress` object that has been set with the upstream healthcheck parameters, Kong will update the upstream objects accordingly and provide the passive healthchecks (circuit breakers) and active healthchecks should you require them. + +For example, you can define the following `KongIngress`: + +```yaml + +apiVersion: configuration.konghq.com/v1 +kind: KongIngress +metadata: + name: sample-kong-ingress +upstream: + hash_on: header + hash_on_header: x-lb + hash_fallback: ip + algorithm: consistent-hashing + healthchecks: + passive: + healthy: + successes: 1 + unhealthy: + http_failures: 5 + tcp_failures: 5 + timeouts: 3 + active: + healthy: + interval: 10 + successes: 1 + unhealthy: + interval: 10 + http_failures: 5 + tcp_failures: 5 + timeouts: 3 +``` + +Then, you need to annotate the Kubernetes Service as `konghq.com/override` so that the upstream gets configured accordingly: + +```yaml + +annotations: + konghq.com/override: sample-kong-ingress +``` + +You can find the Gateway Upstream healthchecks documentation in the related resources below. + +### For KIC version 3.1 and onwards + +For Kubernetes Ingress Controller 3.1 and onwards, `KongIngress` is going to be removed and it is replaced with annotations in most cases, including healthchecks. See the annotation and custom resource references below. diff --git a/app/_support/how-to-create-a-kong-manager-and-rbac-user-via-admin-api.md b/app/_support/how-to-create-a-kong-manager-and-rbac-user-via-admin-api.md new file mode 100644 index 0000000000..c63016fc73 --- /dev/null +++ b/app/_support/how-to-create-a-kong-manager-and-rbac-user-via-admin-api.md @@ -0,0 +1,127 @@ +--- +title: How to create a Kong Manager and RBAC user via Admin API +content_type: support +description: Explains how to create a Kong Manager admin user, assign it the `super-admin` RBAC role, and activate it entirely through the Admin API using an existing RBAC token. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I create a Kong Manager admin user and assign the super-admin RBAC role using only the Admin API? + a: | + With an existing RBAC token, POST to `//admins` to create the user, then POST to `//admins//roles` to assign the `super-admin` role. If SMTP isn't configured, generate a registration URL directly via `?generate_register_url=true` on the admin's endpoint and use it to activate the account and set a password, bypassing email verification. +related_resources: [] +--- + +## Overview + +If the user does not have access to Kong manager to invite user, how can they use Admin API (with RBAC token) to create new users? + +## Steps + +Prerequisites: + +- A user needs to have an existing RBAC token + +### Create new user + +As I will be creating a super-admin, the user will be created in the default workspace. + +```bash + +curl http://://admins \ +-H "Kong-Admin-Token:" \ +-H "Content-Type: application/json" \ +-d '{"email":"","rbac_token_enabled":true,"username":""}' +``` + +Response should be similar to below. Please note down `id`. + +```json + +{ + "admin": { + "created_at": 1614061705, + "updated_at": 1614061705, + "id": "4cb58015-3394-4ee7-9eb7-519e820cec94", + "rbac_token_enabled": true, + "status": 4, + "username": "", + "email": "" + } +} +``` + +### Assign role to the new user + +Here we will be assigning `super-admin` role to this user, id `4cb58015-3394-4ee7-9eb7-519e820cec94`. + +```bash + +curl http://://admins/4cb58015-3394-4ee7-9eb7-519e820cec94/roles \ +-H "Kong-Admin-Token:" \ +-H "Content-Type: application/json" \ +-d '{"roles":"super-admin"}' +``` + +The response should come back as below. + +```json + +{ + "roles": [ + { + "comment": "Full access to all endpoints, across all workspaces", + "created_at": 1614084842, + "id": "e72111a7-47c8-46b3-844f-895f0af4189a", + "name": "super-admin", + "is_default": false, + "role_source": "local", + "ws_id": "d478d5c9-4dc9-4bb6-9c67-fa4b6b76b823" + } + ] +} +``` + +If you have SMTP set up, this user should receive the invite to create password. + +### Activate user and create password + +To bypass the email verification, an RBAC user can generate the register url directly with Admin API. + +```bash + +curl http://://admins/4cb58015-3394-4ee7-9eb7-519e820cec94?generate_register_url=true \ +-H 'Kong-Admin-Token:' +``` + +You should get similar response as below: + +```json + +{ + "rbac_token_enabled": true, + "belong_workspace": { + "id": "d478d5c9-4dc9-4bb6-9c67-fa4b6b76b823", + "name": "default", + "created_at": 1614061000, + "updated_at": 1614061000 + }, + "email": "", + "username": "", + "id": "1b8959ab-d45d-4e2b-8190-3ce47d6bd498", + "status": 4, + "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTQzNDYzODIsImlkIjoiZGVjN2M0MTktNzMwNC00YTU5LWFkNWItZDRmYTk2NzdhMWIwIn0.syMkow-XC3yCWuVCTViP47oubvECbv7IqhMNtRRiBJ4", + "workspaces": [ + { "name": "*" }, + { "is_admin_workspace": true, "id": "d478d5c9-4dc9-4bb6-9c67-fa4b6b76b823", "name": "default" } + ], + "register_url": "http://:/register?email=&username=&token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTQzNDYzODIsImlkIjoiZGVjN2M0MTktNzMwNC00YTU5LWFkNWItZDRmYTk2NzdhMWIwIn0.syMkow-XC3yCWuVCTViP47oubvECbv7IqhMNtRRiBJ4", + "updated_at": 1614087069, + "groups": {}, + "created_at": 1614087069 +} +``` + +We can go to `http://:/register?email=&username=&token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTQzNDYzODIsImlkIjoiZGVjN2M0MTktNzMwNC00YTU5LWFkNWItZDRmYTk2NzdhMWIwIn0.syMkow-XC3yCWuVCTViP47oubvECbv7IqhMNtRRiBJ4` to create password for new user ``. diff --git a/app/_support/how-to-create-a-new-admin-without-kong-manager.md b/app/_support/how-to-create-a-new-admin-without-kong-manager.md new file mode 100644 index 0000000000..fd0b0e4a81 --- /dev/null +++ b/app/_support/how-to-create-a-new-admin-without-kong-manager.md @@ -0,0 +1,153 @@ +--- +title: How to create a new admin without Kong Manager +content_type: support +description: Provides a bash script that creates a new Kong Manager admin, sets their password, and generates an RBAC token, all via the Admin API. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I create a new Kong admin with RBAC credentials without using Kong Manager? + a: | + Using only the Admin API and an existing `Kong-Admin-Token`, run a script that creates the admin, generates an invite/registration token, sets the admin's password, authenticates as the new admin to get a session ID, and then generates a new RBAC token for them, all without going through the Kong Manager UI. +related_resources: [] +--- + +## Overview + +How to create a new admin without Kong Manager + +## Steps + +There are 5 steps to creating a new administrator with RBAC credentials, without using the Admin API. + +Beforehand you will need the following variables: + +`$KONG_ADMIN` = Admin API endpoint + +`$ADMIN_TOKEN` = The admin token of the creating admin, not the new admin. + +`$EMAIL` = The email of the new admin + +`$USERNAME` = The name of the new admin + +`$PASSWORD` = The new password of the admin + +The following script does the following operations all at once: + +1. Create a new admin using `$USERNAME` and `$EMAIL` as the details + +2. Generates the invite token + +3. Sets the newly created admins password to the value of: `$PASSWORD` using the token generated in step 2 + +4. Authenticates with the Admin API using basic authentication with the new admins credentials + +5. Generates a new RBAC Token for the new admin + +```bash + +#!/bin/bash + +USERNAME="test-admin" +PASSWORD="test-password" +EMAIL="test-admin@test.com" +ADMIN_TOKEN="admin" +KONG_ADMIN="https://localhost:8444" + +# 1. Creates a new admin +CREATE_DATA="{\"email\":\"$EMAIL\" ,\"username\":\"$USERNAME\",\"rbac_token_enabled\":true}" +printf "Generating new admin with the following JSON: $CREATE_DATA ..." + +CREATE_RESPONSE=$(curl -sk --location --request POST $KONG_ADMIN/admins \ +--header "Kong-Admin-Token:$ADMIN_TOKEN" \ +--header "Content-Type:application/json" \ +--data-raw "$CREATE_DATA") + +NAME=$(echo $CREATE_RESPONSE | jq .admin.username -j ) + +if [[ $USERNAME == $NAME ]] +then + printf "SUCCESS" +else + printf "FAILED" + exit 1 +fi + +# 2. Generates the invite token +printf "\nGenerating invite token ..." + +INVITE_RESPONSE=$(curl -sk --location --request GET $KONG_ADMIN/admins/$NAME \ +--header "Content-Type:application/json" \ +--header "Kong-Admin-Token:$ADMIN_TOKEN" \ +--data-raw '{"generate_register_url":true}') + +TOKEN=$(echo $INVITE_RESPONSE | jq .token -j) + +if [[ $TOKEN != "" ]] +then + printf "SUCCESS" +else + printf "FAILED" + exit 1 +fi + +# 3. Sets a password for the admin +printf "\nSetting new admin password ..." +SET_PASSWORD_DATA="{\"username\":\"$NAME\",\"email\":\"$EMAIL\",\"token\":\"$TOKEN\",\"password\":\"$PASSWORD\"}" + +SET_PASSWORD_RESPONSE=$(curl -skv --location --request POST $KONG_ADMIN/admins/register \ +--header "Kong-Admin-Token:$ADMIN_TOKEN" \ +--header "Content-Type:application/json" \ +--data-raw "$SET_PASSWORD_DATA" 2>&1) + +SUB_STRING="201 Created" +if [[ $SET_PASSWORD_RESPONSE == *"$SUB_STRING"* ]] +then + printf "SUCCESS" +else + printf "FAILED" + exit 1 +fi + +BASE64_ENCODED_AUTH=$(echo -n $USERNAME:$PASSWORD | base64 ) + +# printf "\n\n$BASE64_ENCODED_AUTH" + +# 4. Authenticates and gets a session ID +printf "\nAuthorising new admin and retrieving session id ..." +GET_SESSION_RESPONSE=$(curl -skv --location --request GET $KONG_ADMIN/auth \ + -H "Authorization: Basic $BASE64_ENCODED_AUTH" \ + -H "Kong-Admin-User: $USERNAME" 2>&1 ) + +SESSION_ID=$(echo "$GET_SESSION_RESPONSE" | grep "Set-Cookie" | cut -d ";" -f 1 | cut -d " " -f 3) + +if [[ $SESSION_ID != "" ]] +then + printf "SUCCESS" + printf "\nSession ID: $SESSION_ID" +else + printf "\nFailed to get session id" + exit 1 +fi + +# 5. Generates a new RBAC Token for the new admin +printf "\nGenerating RBAC Token for new admin ..." +RBAC_TOKEN_RESPONSE=$(curl -sk --location --request PATCH $KONG_ADMIN/admins/self/token \ + -H 'Content-Type: application/json' \ + -H "Cookie: $SESSION_ID" \ + -H "Kong-Admin-User: $USERNAME" \ + -d '{}') + +NEW_RBAC_TOKEN=$(echo $RBAC_TOKEN_RESPONSE | jq .token -j ) + +if [[ $NEW_RBAC_TOKEN != "" ]] +then + printf "SUCCESS" + printf "\nNew admin RBAC token: $NEW_RBAC_TOKEN" +else + printf "FAILED" + exit 1 +fi +``` diff --git a/app/_support/how-to-create-global-plugins-when-converting-openapi-spec-to-declarative-kong-config.md b/app/_support/how-to-create-global-plugins-when-converting-openapi-spec-to-declarative-kong-config.md new file mode 100644 index 0000000000..1ae8573582 --- /dev/null +++ b/app/_support/how-to-create-global-plugins-when-converting-openapi-spec-to-declarative-kong-config.md @@ -0,0 +1,57 @@ +--- +title: How to create global plugins when converting OpenAPI spec to declarative Kong config +content_type: support +description: The `x-kong-plugin` extension only adds plugins at the service level, so global plugins must be added manually to the declarative config after conversion. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I create global plugins when converting an OpenAPI spec to declarative Kong config? + a: | + The `x-kong-plugin` extension only attaches plugins at the service or route level, not globally. + After running `deck file openapi2kong`, add the global plugin to a `plugins:` entry in the declarative config by hand, then sync it with a tag, for example `deck gateway sync --select-tag global-plugins`, to avoid affecting the rest of the workspace. +related_resources: + - text: rate limiting advanced plugin basic example + url: /plugins/rate-limiting-advanced/examples/ + - text: "`deck_sync` flags reference" + url: /deck/gateway/sync/#command-usage +--- + +## Overview + +When using the `x-kong-plugin` property in the OpenAPI spec file it adds the plugin to the service level. How can I add the plugin globally? + +## Steps + +The decK `file openapi2kong` command converts OpenAPI spec to declarative config for Kong using Kong APIOps. When using the `x-kong-plugin` custom extension in the OpenAPI spec file can help to generate plugin config in the declarative config for Kong for routes and services. But for the plugin at the global level, it is not supported. You may need to manually add a plugins entry in the declarative configuration file after you have generated the declarative config. + +For example, if you want to enable the rate limiting advanced plugin globally you can create the following `rla-global.yaml` file: + +```yaml +_format_version: "3.0" +_workspace: test +plugins: +- name: rate-limiting-advanced + tags: + - global-plugins + config: + limit: + - 5 + window_size: + - 30 + identifier: consumer + sync_rate: -1 + namespace: example_namespace + strategy: local + hide_client_headers: false +``` + +Then, you can use the decK command `deck gateway sync --select-tag global-plugins` with the `--workspace` flag to determine which Kong workspace you want to sync. + +Note it is important to add `--select-tag global-plugins` in this case, otherwise it will affect all the configuration in the workspace. Using the created `rla-global.yaml` file you can enable the rate-limiting-advanced plugin globally in the workspace “test” with the following command: + +```bash +deck gateway sync --select-tag global-plugins --workspace test rla-global.yaml +``` diff --git a/app/_support/how-to-customise-a-plugin-response-to-make-it-more-meaningful.md b/app/_support/how-to-customise-a-plugin-response-to-make-it-more-meaningful.md new file mode 100644 index 0000000000..bb42b33dec --- /dev/null +++ b/app/_support/how-to-customise-a-plugin-response-to-make-it-more-meaningful.md @@ -0,0 +1,72 @@ +--- +title: How to customize a plugin response to make it more meaningful +content_type: support +description: Use the `exit-transformer` plugin to append meaningful messages to plugin responses based on the HTTP status code returned by Kong. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I customize a plugin response to make it more meaningful? + a: | + Use the `exit-transformer` plugin to rewrite responses based on the HTTP status code Kong returns. + Point its `config.functions` at a Lua function that inspects `status` and sets `body.message` (for example, a clearer message on `403`), then apply the plugin to the route or service. +related_resources: + - text: OpenTelemetry plugin + url: /plugins/opentelemetry/ +--- + +## Overview + +How can I customize the output/response of plugins e.g. bot detection to illustrate the reasons behind certain responses e.g. "Forbidden" + +## Steps + +To provide more informative responses from, for example, the bot detection plugin, you can use the `exit-transformer` plugin to customize the output. This approach allows you to append specific messages to the response body, based on the HTTP status codes returned by Kong, indicating the reason for the response, such as detection as a bot. + +Here are the steps to set up the `exit-transformer` plugin for verbose output: + +1. Create a Lua script named `transform.lua` with the following content: + + ```lua + return function(status, body, headers) + if status == 401 then + body.message = "Unauthorized" + end + if status == 403 then + body.message = "Forbidden, detected as bot" + end + if status == 429 then + body.message = "Retry again later" + end + return status, body, headers + end + ``` + +2. Apply the `exit-transformer` plugin to a route or service. Use the following command as an example to set it up on a route: + + ```bash + curl -X POST -H "Kong-Admin-Token:password" http://api.kong.lan/routes/local-httpbin/plugins \ + --form 'name="exit-transformer"' \ + --form 'config.functions=@"transform.lua"' + ``` + +3. Test the setup by making a request that would trigger the plugin. For example, using a user-agent that is detected as a bot: + + ```bash + curl http://localhost:8000/httpbin -A "Googlebot" + ``` + + The response should now include the customized message: + + ```json + {"request_id":"61b4f06cc3719e4410464120c3a83054","message":"Forbidden, detected as bot"} + ``` + +Additionally, when logging plugins like `http-log` or `file-log` are enabled, the user-agent and other relevant information are logged in the Data Plane logs, which can be useful for debugging purposes. + +Additional Tips: + +- For a more comprehensive view of the plugins involved in processing a request, consider using the OpenTelemetry plugin, which provides visibility into the spans captured and can help developers understand the flow of requests through the plugins. +- Remember to remove or adjust the verbose output settings when moving from the development stage to production to avoid exposing unnecessary information. diff --git a/app/_support/how-to-customize-kong-access-logs.md b/app/_support/how-to-customize-kong-access-logs.md new file mode 100644 index 0000000000..98b30655ab --- /dev/null +++ b/app/_support/how-to-customize-kong-access-logs.md @@ -0,0 +1,104 @@ +--- +title: How to customize Kong access logs +content_type: support +description: Explains how to write a custom Nginx template with a `map` block so Kong's access log skips 2xx and 3xx requests. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I customize Kong's access logs to skip 2xx and 3xx requests? + a: | + Write a custom Nginx template based on Kong's default one, adding an nginx `map` block (`map $status $loggable { ~^[23] 0; default 1; }`) and setting `proxy_access_log` to use `if=$loggable`. Start Kong with `--nginx-conf` pointing at the custom template, escaping the `$` as `$$` or `\$` when it's set via an environment variable, so only non-2xx/3xx requests are logged. +related_resources: [] +--- + +## Overview + +How can I customize Kong access log, for example not logging 2xx and 3xx requests? + +## Steps + +To customize Kong access logs, basically you need to write a custom Nginx template and use the `map` module to remove the 2xx and 3xx requests. For the template, take as model the file `/usr/local/kong/kong.conf` and add the map: + +```nginx +pid pids/nginx.pid; +error_log /dev/stderr notice; +# injected nginx_main_* directives +daemon off; +worker_processes auto; +worker_rlimit_nofile 16384; +env KONG_LICENSE_DATA; +env KONG_LICENSE_PATH; +include 'nginx-inject.conf'; + +events { + # injected nginx_events_* directives + multi_accept on; + worker_connections 16384; +} + +http { + map $status $loggable { + ~^[23] 0; + default 1; + } + include 'nginx-kong.conf'; +} +``` + +This file needs to be located in the `/usr/local/kong/` directory as references `/usr/local/kong/nginx-kong.conf` + +Then you need to start Kong to use this config: + +```bash +kong start --nginx-conf /usr/local/kong/custom-nginx.conf +``` + +And configure `proxy_access_log` as below: + +``` +proxy_access_log = /dev/stdout combined if=$loggable +``` + +If using environment variables you may need to escape the `$` character: + +``` +KONG_PROXY_ACCESS_LOG: "/dev/stdout combined if=$$loggable" +``` + +If running docker from the terminal directly, you also need to escape the `$` character as `\$`: + +```bash +docker run --network bridge -d --name kong-dbless \ + -v $(pwd)/custom_nginx.conf:/usr/local/kong/custom-nginx.conf \ + -v "$(pwd)/declarative:/kong/declarative/" \ + -v "$(pwd)/tmp_volume:/tmp" \ + -v "$(pwd)/prefix_volume:/var/run/kong" \ + -e "KONG_PREFIX=/var/run/kong" \ + -e "KONG_DATABASE=off" \ + -e "KONG_DECLARATIVE_CONFIG=/kong/declarative/kong3.yaml" \ + -e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" \ + -e "KONG_PROXY_ERROR_LOG=/dev/stderr" \ + -e "KONG_ADMIN_ERROR_LOG=/dev/stderr" \ + -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \ + -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \ + -e "KONG_PROXY_ACCESS_LOG=/dev/stdout combined if=\$loggable" \ + -e "KONG_LICENSE_DATA=$KONG_LICENSE_DATA" \ + -p 8000:8000 \ + -p 8443:8443 \ + -p 8001:8001 \ + -p 8444:8444 \ + -p 8002:8002 \ + -p 8445:8445 \ + -p 8003:8003 \ + -p 8004:8004 \ + kong/kong-gateway \ + kong start --nginx-conf /usr/local/kong/custom-nginx.conf +``` + +### Additional information + +- Kong docs: How to Customize Gateway Logs +- Nginx docs: Configuring Logging - Enabling Conditional Logging diff --git a/app/_support/how-to-debug-the-control-plane-dataplane-web-socket-communication.md b/app/_support/how-to-debug-the-control-plane-dataplane-web-socket-communication.md new file mode 100644 index 0000000000..c2ac212bb1 --- /dev/null +++ b/app/_support/how-to-debug-the-control-plane-dataplane-web-socket-communication.md @@ -0,0 +1,111 @@ +--- +title: How to debug the Control Plane/DataPlane web socket communication +content_type: support +description: Use stunnel to terminate TLS on the Control Plane/Data Plane WebSocket connection and tcpdump to capture the resulting plaintext traffic, so you can inspect the declarative configuration payload sent over `cluster_listen`. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I debug the Control Plane/Data Plane WebSocket communication in a hybrid {{site.base_gateway}} deployment? + a: | + Use `stunnel` to terminate TLS on the CP/DP `cluster_listen` WebSocket connection as a man-in-the-middle proxy, and `tcpdump` to capture the resulting plaintext traffic. Inspect the captured packets (e.g. in Wireshark) to extract the gzipped declarative configuration JSON that the CP sends to each DP. +related_resources: [] +--- + +## Overview + +When using a hybrid installation, the Control Plane and the Data Plane use a Secure Web Socket for communication. This traffic is encrypted and not observable using tools such as `tcpdump`. How can the actual communication content be observed? + +## Steps + +For this example, we will assume that the default (Debian-based) {{site.base_gateway}} images are being used and that the Hybrid installation is using shared certificates. This is the simplest Hybrid deployment architecture, but the same principles can be used for PKI mode. + +When a DP starts, it opens a Web Socket to the CP and the CP pushes the configuration to the DP. This is done via a Web Socket connection on the `cluster_listen` port (8005). To view this traffic, we are going to use `stunnel` to setup a MITM attack and `tcpdump` to capture the plain text traffic. The example also captures traffic on the CP as this will capture all traffic for all DP nodes. You could also setup the capture on the DP node if you only want to capture traffic from a single node. + +1. Configure the CP to listen on a non-default port for the `cluster_listen` port. In our example, we are using `docker-compose` so have a line below to use port 48005. There is no need to expose this port externally, as we will be using `stunnel` to forward traffic from the standard port (8005) to this custom port. + + ```yaml + + KONG_CLUSTER_LISTEN: "0.0.0.0:48005" + ``` + +2. On the CP, install `stunnel` and `tcpdump` + + ```bash + + apt-get update + apt-get install -y stunnel4 tcpdump + ``` + +3. Create a `stunnel` configuration file; + + ```bash + + echo "debug = 3 + foreground = no + pid = + + [server] + client = no + cert = /tmp/hybrid/cluster.crt + key = /tmp/hybrid/cluster.key + accept = 0.0.0.0:8005 + connect = 127.0.0.1:58005 + + [client] + client = yes + cert = /tmp/hybrid/cluster.crt + key = /tmp/hybrid/cluster.key + accept = 127.0.0.1:58005 + connect = 0.0.0.0:48005" > stunnel-mitm-proxy.conf + ``` + + This configuration will start a `stunnel` server listener on port 8005. This is the port that the DP will be connecting to. The configuration uses the shared Hybrid cluster certificate pair for this listen port and forwards traffic as plain text to port 58005 from the client section. + + The client is listening on port 58005 and forwards traffic to the local port that Kong is using for the `cluster_listen` port (48005) + +4. Start `stunnel` + + ```bash + + stunnel stunnel-mitm-proxy.conf + ``` + +5. Start a `tcpdump` running on the plain text port + + ```bash + + tcpdump -s 0 -i any -w /tmp/cluster.pcap port 58005 + ``` + +6. Start the DP. This will connect to port 8005 on the CP and `stunnel` will accept the connection and forward to itself as plain text before forwarding to the `cluster_listen` port + +7. Wait for the DP to have downloaded the configuration (check the `/clustering/status` endpoint) + +8. Once the DP has the configuration, you can stop the `tcpdump` and copy the `/tmp/cluster.pcap` file to your local machine + +When the DP connects to the CP, it sends a json object with details of the plugins it has installed and their versions. You can see this in the screenshot below; + +The CP verifies that the DP has the required plugins of the correct versions to ensure that the entity sync can succeed. If this check passes, then the CP sends a gzipped json file with the declarative configuration; + +To see what is in the zipped archive, it is necessary to extract the content. To do this, right click on the Data for the TCP frame and select "Export Packet Bytes...". + +Save the file locally (you will need to use a `.bin` extension for the filename, for example `temp.json.bin`). + +Unzip the file (if using `gunzip`, then you will need to use the `-S` parameter to allow the `.bin` extension) + +```bash + +gunzip -S .bin temp.json.bin +``` + +Check the start of the json payload; + +```bash + +head -c250 temp.json +``` + +You now have a copy of the declarative configuration json file that the CP sends to the DP. The DP will save this file in `/usr/local/kong/config.cache.json.gz` diff --git a/app/_support/how-to-deploy-multiple-custom-plugins-in-kong-in-docker.md b/app/_support/how-to-deploy-multiple-custom-plugins-in-kong-in-docker.md new file mode 100644 index 0000000000..70a47c981f --- /dev/null +++ b/app/_support/how-to-deploy-multiple-custom-plugins-in-kong-in-docker.md @@ -0,0 +1,72 @@ +--- +title: How to deploy multiple custom plugins in Kong in Docker +content_type: support +description: Deploy multiple custom plugins in a Dockerized Kong by mounting them outside the standard plugins directory and configuring `lua_package_path` and `KONG_PLUGINS`. +tldr: + q: How do I deploy multiple custom plugins in Kong running in Docker? + a: | + Mount your custom plugins to a path outside the standard Kong plugins directory, then point `KONG_LUA_PACKAGE_PATH` at it (e.g. `/usr/local/custom_plugins/?.lua;;`) and list the plugins in `KONG_PLUGINS` alongside `bundled`. + Restart the container to pick up code changes. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +--- + +## Overview + +How can multiple custom plugins be configured for Kong when using Docker? + +## Steps + +Implementing and updating custom plugins in Kong, particularly within a Docker environment, requires careful consideration of volume mounts and the `lua_package_path` configuration to avoid overwriting existing plugins and to enable seamless updates. Here's a step-by-step guide based on a successful resolution: + +1. Mount custom plugin code to a Docker volume. + + To avoid overwriting the standard Kong plugins directory and ensure that your custom plugins are correctly recognized, you should mount your custom plugins directory outside the standard Kong plugins directory. + + First, organize your custom plugins on your host machine with the correct directory structure. For example, if you have custom plugins named `log-filter` and `custom-handler`, your directory structure should look like this: + + ``` + custom_plugins/ + ├─ kong/ + └─── plugins/ + ├───── log-filter/ + └───── custom-handler/ + ``` + + Then, mount this directory to the container using a Docker volume that points outside the standard Kong plugins directory: + + ```bash + -v /path/to/custom_plugins:/usr/local/custom_plugins + ``` + +2. Configure `lua_package_path`. + + Adjust the `lua_package_path` to include your custom plugins directory. This tells Kong where to find your custom plugins without interfering with the default plugin directory. + + ```bash + -e "KONG_LUA_PACKAGE_PATH=/usr/local/custom_plugins/?.lua;;" + ``` + + The `?` in the `lua_package_path` is expanded to `kong/plugins/`. So for our example plugins, the path will be expanded to: + + ``` + /usr/local/custom_plugins/kong/plugins/ + ``` + + where `` will be either `log-filter` or `custom-handler`. + +3. Load custom plugins. + + Specify your custom plugins in the `KONG_PLUGINS` environment variable, along with the bundled plugins, to ensure they are loaded by Kong. + + ```bash + -e "KONG_PLUGINS=bundled, log-filter, custom-handler" + ``` + +4. Update custom plugins or add new ones. + + Updating the plugin code on the host requires restarting the Kong container to reflect the changes. For adding new plugins, simply follow the steps above to include them in your custom plugins directory and update the `KONG_PLUGINS` environment variable accordingly. diff --git a/app/_support/how-to-determine-and-test-which-aws-subnets-service-kongs-s3-buckets.md b/app/_support/how-to-determine-and-test-which-aws-subnets-service-kongs-s3-buckets.md new file mode 100644 index 0000000000..68b6ca83d2 --- /dev/null +++ b/app/_support/how-to-determine-and-test-which-aws-subnets-service-kongs-s3-buckets.md @@ -0,0 +1,90 @@ +--- +title: How to determine and test which AWS subnets service Kong's S3 buckets. +content_type: support +description: "Use AWS's published IP range list and `jq` to determine which subnets serve a Kong S3 bucket in a given region." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "AWS's published list of subnets for all their services" + url: https://ip-ranges.amazonaws.com/ip-ranges.json +tldr: + q: How do I determine which subnets AWS will use to provide IP addresses to Kong S3 buckets? + a: | + AWS publishes a list of subnets for all its services at `ip-ranges.json`. Filter that list with `jq` for the `S3` service and your target region, then compare the resulting subnets against the IP returned by `ping`/`dig` for your bucket's hostname to find the matching range. +--- + +## Overview + +How do I determine which subnets AWS will use to provide IP addresses to Kong S3 buckets? + +## Steps + +AWS publishes a list of subnets for all their services. + +We can use this list along with `jq` to filter for specific regions and services from AWS. + +Example: + +Using bucket URL: pulp-cloud-02-eks-01-prod-us-east-2-20220929122921552700000002.s3.amazonaws.com + +This bucket holds Kong binaries which are available for download. + +Note: this particular bucket name is used only to illustrate the methodology. It no longer resolves to the `us-east-2` region implied by its name — it currently resolves via `us-east-1`. Rather than relying on this specific example, use a bucket/endpoint you've freshly verified for your region, or a region-specific S3 endpoint (for example `s3..amazonaws.com`), and confirm the resolved region with `dig`/`ping` before filtering `ip-ranges.json`. The `ip-ranges.json` + `jq` methodology below remains valid regardless of which bucket you check. + +1. Determine the AWS service we are targeting, in this case, it is S3 . + +2. If possible, extrapolate the region the resource is in. The URL makes it easy in this case: us-east-2 + +3. Download the list of IP addresses and filter it based on the values: + +```bash + +curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.service=="S3") | select(.region=="us-east-2") | .ip_prefix' | sort + +1.178.8.0/24 +16.12.60.0/22 +16.12.64.0/22 +18.34.252.0/22 +18.34.72.0/21 +3.141.102.208/28 +3.141.102.224/28 +3.2.67.0/24 +3.5.100.0/22 +3.5.104.0/22 +3.5.108.0/22 +3.5.128.0/22 +3.5.132.0/23 +3.5.88.0/22 +3.5.92.0/23 +52.219.141.0/24 +52.219.142.0/24 +52.219.143.0/24 +52.219.176.0/22 +52.219.212.0/22 +52.219.224.0/22 +52.219.228.0/22 +52.219.232.0/22 +52.219.80.0/20 +52.219.96.0/20 +``` + +This represents a list of all IP subnet ranges that will cover any S3 bucket address in the us-east-2 region. + +To prove that the URL is indeed using these subnets, a simple `ping` and lookup can be performed. + +4. `Ping` the url to determine the current IP: + +```bash + +ping pulp-cloud-02-eks-01-prod-us-east-2-20220929122921552700000002.s3.amazonaws.com +PING s3-w.us-east-2.amazonaws.com (52.219.84.188): 56 data bytes +``` + +5. Find the range in the previous list that is the closest to the returned IP. + +The closest related subnet is: 52.219.80.0/20 + +6. Use a subnet calculator so show the IP addresses that are covered by the range. Results diff --git a/app/_support/how-to-disable-ssl-chipers.md b/app/_support/how-to-disable-ssl-chipers.md new file mode 100644 index 0000000000..a672d702ac --- /dev/null +++ b/app/_support/how-to-disable-ssl-chipers.md @@ -0,0 +1,51 @@ +--- +title: How to disable SSL ciphers in Kong-Gateway Proxy +content_type: support +description: "Configure Kong's `ssl_cipher_suite` and `ssl_ciphers` settings to enable only the SSL/TLS ciphers you want, instead of disabling the ones you don't." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I disable SSL ciphers I don't want, as many are enabled by default? + a: | + Set `ssl_cipher_suite` to `custom` in `kong.conf` (or via the `KONG_SSL_CIPHERS` / `KONG_SSL_CIPHER_SUITE` environment variables) and list only the ciphers you want enabled, rather than trying to disable the ones you don't want. +--- + +## Overview + +How do I disable SSL ciphers I don't want, as many are enabled by default? + +## Steps + +Kong allows you to configure SSL ciphers by enabling only the ones you require rather than disabling unwanted ones. Configure this by setting the `ssl_cipher_suite` to `custom` and specifying your preferred ciphers. + +Set the following in your `kong.conf` file: + +```conf +ssl_cipher_suite = custom +``` + +```conf +ssl_ciphers = +``` + +Example using environment variables: + +```bash +KONG_SSL_CIPHERS=ECDHE-RSA-AES256-GCM-SHA384,DHE-PSK-ARIA256-GCM-SHA384,ECDHE-RSA-CHACHA20-POLY1305 +KONG_SSL_CIPHER_SUITE=custom +KONG_SSL_PROTOCOLS=TLSv1.2 +``` + +After restarting Kong, verify the loaded SSL ciphers with: + +```bash +/usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf -T | grep ssl_ciphers +``` + +Note: running `nginx -T` on its own (without `-p`/`-c`) fails on current images because the binary's compiled-in default config path doesn't match Kong's runtime layout — the `-p /usr/local/kong -c nginx.conf` flags are required. + +For more information, see the `ssl_cipher_suite` and `ssl_protocols` property references. diff --git a/app/_support/how-to-disable-the-x-forwarded-x-headers-from-a-single-route.md b/app/_support/how-to-disable-the-x-forwarded-x-headers-from-a-single-route.md new file mode 100644 index 0000000000..3ee3059119 --- /dev/null +++ b/app/_support/how-to-disable-the-x-forwarded-x-headers-from-a-single-route.md @@ -0,0 +1,44 @@ +--- +title: How to disable the x-forwarded-x headers from a single route +content_type: support +description: Explains how to use the `post-function` plugin to disable specific `X-Forwarded-*` headers, such as `x-forwarded-proto` or `x-forwarded-host`, on a single route. +products: + - gateway + - kic +works_on: + - on-prem + - konnect +tldr: + q: How do I disable specific X-Forwarded-* headers on a single route? + a: | + Install the `post-function` plugin on the route and, in its `access` phase configuration, set the corresponding Nginx variable to `nil` (for example, `ngx.var.upstream_x_forwarded_proto=nil` to drop `x-forwarded-proto`, or `ngx.var.upstream_x_forwarded_host=nil` to drop `x-forwarded-host`) so the header isn't sent to the upstream. +related_resources: [] +--- + +## Overview + +How to disable the x-forwarded-x headers on a single route so they do not get sent to the upstream. + +## Steps + +The easiest way to disable these headers is with a `post-function` plugin. + +If you wanted to disable the `x-forwarded-proto` header for example: + +1. Install the `post-function` plugin on the route you wish the header removed from. + +2. Add the following line to the access phase of the post function header field: + +```lua +ngx.var.upstream_x_forwarded_proto=nil +``` + +If you wanted to disable the `x-forwarded-host` header for example: + +1. Install the `post-function` plugin on the route you wish the header removed from. + +2. Add the following line to the access phase of the post function header field: + +```lua +ngx.var.upstream_x_forwarded_host=nil +``` diff --git a/app/_support/how-to-display-xff-headers-in-access-log.md b/app/_support/how-to-display-xff-headers-in-access-log.md new file mode 100644 index 0000000000..afaae3fb50 --- /dev/null +++ b/app/_support/how-to-display-xff-headers-in-access-log.md @@ -0,0 +1,35 @@ +--- +title: How to display XFF headers in Proxy Access Logs +content_type: support +published: false +description: "Steps to configure Kong to log X-Forwarded-For and other `x_forwarded_*` header values in proxy access logs using a custom NGINX log format." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: available NGINX variables + url: http://nginx.org/en/docs/varindex.html +tldr: + q: How do I display X-Forwarded-For (XFF) headers in Kong's proxy access logs? + a: | + Set `KONG_PROXY_ACCESS_LOG` and `KONG_NGINX_HTTP_LOG_FORMAT` to a custom NGINX log format that includes the `x_forwarded_for`, `x_forwarded_proto`, `x_forwarded_host`, `x_forwarded_port`, `x_forwarded_path`, and `x_forwarded_prefix` variables. +--- + +## Overview + +How to display XFF headers in Proxy Access Logs + +## Steps + +Assuming you are running Kong by using a container, + +below environment parameters need to be added to show `x_forwarded_*` headers. + +```bash +KONG_PROXY_ACCESS_LOG=/dev/stdout show_everything +KONG_NGINX_HTTP_LOG_FORMAT=show_everything ', x_forwarded_for:$upstream_x_forwarded_for, x_forwarded_proto:$upstream_x_forwarded_proto, x_forwarded_host:$upstream_x_forwarded_host, x_forwarded_port:$upstream_x_forwarded_port, x_forwarded_path:$upstream_x_forwarded_path, x_forwarded_prefix:$upstream_x_forwarded_prefix' +``` + +Please find available NGINX variables. diff --git a/app/_support/how-to-enable-data-plane-resilience-when-using-helm.md b/app/_support/how-to-enable-data-plane-resilience-when-using-helm.md new file mode 100644 index 0000000000..4bf908311e --- /dev/null +++ b/app/_support/how-to-enable-data-plane-resilience-when-using-helm.md @@ -0,0 +1,123 @@ +--- +title: How to enable Data Plane Resilience when using Helm +content_type: support +description: "How to achieve Data Plane Resilience with the Kong Helm chart by separating the config-exporter data plane from the data plane that proxies traffic." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I enable Data Plane Resilience when deploying Kong with Helm? + a: | + The Kong Helm chart has no setting to stop a data plane from proxying traffic, so a node used only to export config to fallback storage (via `cluster_fallback_config_storage` and `cluster_fallback_config_import`) can't be prevented from also serving requests. The workaround is to deploy two separate data plane node groups with Helm — one dedicated to the config-exporter role and one for routing traffic — and send client traffic only to the second node group (for example, by leaving the exporter's Service out of your load balancer). +related_resources: + - text: Data Plane Resilience documentation + url: /gateway/cp-outage/ + - text: Example minimal {{site.ee_product_name}} hybrid data plane Helm values + url: https://github.com/Kong/charts/blob/main/charts/kong/example-values/minimal-kong-enterprise-hybrid-data.yaml +--- + +## Overview + +I would like to enable Data Plane Resilience, but I can't find anything in the Kong helm charts for this. + +## Steps + +The backup node can be a data plane or a control plane node. + +We will use a data plane node as an example. It cannot be stopped from taking traffic, e.g. turning off proxy listen for the data plane would produce errors. Hence the approach would be to not direct traffic to it, e.g. not to add it to the load balancer or have a different deployment for this node. + +More specifically, since there's no config that would prohibit the exporter data plane node from proxying traffic, the solution would be to create two different dataplanes, i.e. one for the configuration exporter and one for routing traffic, and only use the second one for all requests. + +For example based on the config shown here, there could be two different configurations, one for the exporter and one for the importer. + +e.g. similar to + +```yaml +> cat data-importer.yaml + +image: +repository: kong/kong-gateway +tag: "3.14" + +env: +role: data_plane +cluster_control_plane: URL:8005 +cluster_telemetry_endpoint: URL:8006 +lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt +cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt +cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key +cluster_fallback_config_storage: s3://test-bucket/test-prefix +cluster_fallback_config_import: "on" + +customEnv: +AWS_REGION: 'us-east-2' +AWS_ACCESS_KEY_ID: XXX +AWS_SECRET_ACCESS_KEY: XXX + +secretVolumes: +- kong-cluster-cert + +ingressController: +enabled: false + +enterprise: +enabled: true +# See instructions regarding enterprise licenses at https://github.com/Kong/charts/blob/master/charts/kong/README.md#kong-enterprise-license +license_secret: kong-enterprise-license # CHANGEME +vitals: +enabled: false + +manager: +enabled: false + +portal: +enabled: false + +portalapi: +enabled: false + +proxy: +http: +servicePort: 8001 +containerPort: 8000 + +tls: +servicePort: 8444 +containerPort: 8443 +parameters: +- http2 +``` + +``` +❯ k get all -n kong +NAME READY STATUS RESTARTS AGE +pod/my-controlplane-kong-5c4d5f4cfd-svcdg 0/1 Init:0/2 0 22m +pod/my-controlplane-kong-init-migrations-qzdjf 0/1 Init:0/1 0 22m +pod/my-controlplane-postgresql-0 1/1 Running 0 22m +pod/my-dataplane-exporter1-kong-84966fb599-4x4bd 0/1 Init:0/1 0 6m2s +pod/my-dataplane-importer1-kong-789567c6fb-6zs4f 0/1 Init:0/1 0 4s + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +... +... +service/my-dataplane-exporter1-kong-proxy LoadBalancer 10.99.130.222 localhost 8000:31120/TCP,8443:30266/TCP 6m2s +service/my-dataplane-importer1-kong-proxy LoadBalancer 10.98.75.141 localhost 8001:30414/TCP,8444:31552/TCP 4s + +NAME READY UP-TO-DATE AVAILABLE AGE +deployment.apps/my-controlplane-kong 0/1 1 0 22m +deployment.apps/my-dataplane-exporter1-kong 0/1 1 0 6m2s +deployment.apps/my-dataplane-importer1-kong 0/1 1 0 4s + +NAME DESIRED CURRENT READY AGE +replicaset.apps/my-controlplane-kong-5c4d5f4cfd 1 1 0 22m +replicaset.apps/my-dataplane-exporter1-kong-84966fb599 1 1 0 6m2s +replicaset.apps/my-dataplane-importer1-kong-789567c6fb 1 1 0 4s + +NAME READY AGE +statefulset.apps/my-controlplane-postgresql 1/1 22m + +NAME COMPLETIONS DURATION AGE +job.batch/my-controlplane-kong-init-migrations 0/1 22m 22m +``` diff --git a/app/_support/how-to-enable-debug-log.md b/app/_support/how-to-enable-debug-log.md new file mode 100644 index 0000000000..f91e718a0e --- /dev/null +++ b/app/_support/how-to-enable-debug-log.md @@ -0,0 +1,61 @@ +--- +title: How to enable debug log +content_type: support +description: "There are multiple ways to change Kong's log level depending on your deployment method: config file, environment variable, Kubernetes, Helm, or dynamically without a restart." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I enable debug logging in {{site.base_gateway}}? + a: | + Set `log_level = debug` in `kong.conf`, or set the `KONG_LOG_LEVEL=debug` environment variable (for Docker, Kubernetes, or Helm deployments), then reload or restart Kong. {{site.base_gateway}} also supports Dynamic Log Levels, which let you raise or lower the log level without restarting Kong, except in DB-less mode where the Admin API isn't available. +related_resources: + - text: "{{site.base_gateway}} log level configuration documentation" + url: /gateway/logs/#configure-log-levels +--- + +## Overview + +How do we enable debug log on different platforms? + +## Steps + +VM: + +There are two ways of changing the log level. + +1. On the `/etc/kong/kong.conf` file, change `log_level=debug` + +2. Adding the environment variable `KONG_LOG_LEVEL=debug` will override the setting on the config file. + +Docker: + +Normally we use environment variables to configure Kong containers. Please add `KONG_LOG_LEVEL=debug` to your docker run command or docker compose file under `environment`. + +Kubernetes: + +You can modify the deployment and add the below to the Kong container + +```yaml + - name: KONG_LOG_LEVEL + value: debug +``` + +If you deploy with Helm, you can add `log_level` under `env` as below + +```yaml +env: + log_level: debug +``` + +Once the changes are made, please remember to reload/restart Kong. For Kubernetes, please remember to apply the deployment file or upgrade your Helm release to bring up a new pod with this configuration. + +Dynamically: + +{{site.base_gateway}} supports Dynamic Log Levels to allow log levels to be increased and decreased without the need to restart Kong. + +Caveat: Cannot work in DBLess mode as there is no admin-api available. + +For more details, please review the documentation. diff --git a/app/_support/how-to-enable-debug-logging-for-kuma-sidecar-without-using-port-forward.md b/app/_support/how-to-enable-debug-logging-for-kuma-sidecar-without-using-port-forward.md new file mode 100644 index 0000000000..ae5ed7c4d7 --- /dev/null +++ b/app/_support/how-to-enable-debug-logging-for-kuma-sidecar-without-using-port-forward.md @@ -0,0 +1,47 @@ +--- +title: How to enable debug logging for kuma sidecar without using port-forward +content_type: support +description: Steps to enable debug logging on the kuma sidecar by logging in to the container and sending an HTTP POST request to its admin API with the `wget` command. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I enable debug logging for the kuma sidecar without using `kubectl port-forward`? + a: | + Use `kubectl exec` to open a shell in the kuma sidecar container, then send an HTTP POST request to its local admin API with `wget` to set the log level to `debug`. The same approach can set the log level back to `info` afterward. +--- + +## Overview + +We do not have permission to use port-forward, how could we enable debug logging for kuma sidecar? (The default log level of kuma sidecar is info.) + +## Steps + +You have to log in to the kuma sidecar and send an HTTP POST request with the `wget` command. + +1. Log in to the kuma sidecar. + + ```bash + kubectl exec -it -c kuma-sidecar -n -- sh + ``` + +2. Enable debug logging with the following `wget` command. + + ```bash + wget http://localhost:9901/logging?level=debug --post-data='' -O /tmp/res + ``` + +3. Exit the kuma sidecar container and confirm the log level. + + ```bash + kubectl logs -f -c kuma-sidecar -n + ``` + +For the case you want to set the log level back to info, please log in to the kuma sidecar and run the following `wget` command. + +```bash +wget http://localhost:9901/logging?level=info --post-data='' -O /tmp/res +``` diff --git a/app/_support/how-to-enable-debug-logs-for-the-kubernetes-ingress-controller.md b/app/_support/how-to-enable-debug-logs-for-the-kubernetes-ingress-controller.md new file mode 100644 index 0000000000..ce164dfd10 --- /dev/null +++ b/app/_support/how-to-enable-debug-logs-for-the-kubernetes-ingress-controller.md @@ -0,0 +1,42 @@ +--- +title: How to enable debug logs for the Kubernetes Ingress Controller +content_type: support +description: "Set the Kubernetes Ingress Controller's log level to debug using the `log-level` CLI flag, the `CONTROLLER_LOG_LEVEL` environment variable, or the `log_level` key under `ingressController.env` in the Helm chart." +products: + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I enable debug logs for the Kubernetes Ingress Controller? + a: | + Set the KIC log level to `debug` via the `log-level` CLI flag, the `CONTROLLER_LOG_LEVEL` environment variable, or the `log_level` key under `ingressController.env` in the Helm chart. The chart prepends `CONTROLLER_` and upper-cases the key, so it must be `log_level`, not `controller_log_level`. +--- + +## Overview + +How do I enable debug logs for the Kubernetes Ingress Controller? + +## Steps + +The KIC log level can be set with the `log-level` flag, documented here: Kubernetes Ingress Controller CLI Arguments . + +The flag can also be configured using an environment variable. The name of the environment variable is: `CONTROLLER_LOG_LEVEL=debug` . + +When using Helm, you can add this flag in the `env` section under the `ingress` config, as part of the Ingress Controller Parameters , for example: + +The chart automatically prepends `CONTROLLER_` and upper-cases every key placed under `ingressController.env`, so the key must be `log_level`, not `controller_log_level` — using `controller_log_level` produces the never-read environment variable `CONTROLLER_CONTROLLER_LOG_LEVEL` and the flag is silently ignored. + +```yaml +ingressController: + enabled: true + installCRDs: false + env: + log_level: "debug" + kong_admin_token: + valueFrom: + secretKeyRef: + name: kong-enterprise-superuser-password #CHANGEME + key: password #CHANGEME +``` diff --git a/app/_support/how-to-enable-response-compression-using-gzip.md b/app/_support/how-to-enable-response-compression-using-gzip.md new file mode 100644 index 0000000000..a4accd81e1 --- /dev/null +++ b/app/_support/how-to-enable-response-compression-using-gzip.md @@ -0,0 +1,47 @@ +--- +title: How to enable response compression using gzip +content_type: support +description: "Enable gzip response compression in Kong by setting the nginx `gzip` directives (`nginx_proxy_gzip`, `nginx_proxy_gzip_proxied`, `nginx_proxy_gzip_types`) in `kong.conf` or as `KONG_NGINX_PROXY_GZIP*` environment variables." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I enable gzip response compression in Kong? + a: | + Inject nginx's gzip directives (`nginx_proxy_gzip`, `nginx_proxy_gzip_proxied`, `nginx_proxy_gzip_types`) via `kong.conf` or the equivalent `KONG_NGINX_PROXY_GZIP*` environment variables (available since Kong EE 0.35; earlier versions need a custom nginx template). Clients must send an `Accept-Encoding: gzip` header to receive a compressed response. +related_resources: + - text: nginx gzip module documentation + url: http://nginx.org/en/docs/http/ngx_http_gzip_module.html + - text: Kong nginx-injected directives configuration reference + url: /gateway/configuration/#nginx-injected-directives-section +--- + +## Overview + +How to enable response compression using the "gzip" method + +## Steps + +It is possible to add the relevant nginx directives to the Kong configuration to enable the features that the nginx module supports. + +For example, you could add the following properties to `kong.conf` or `KONG_` environment variables to enable compression for all requests: + +```conf +nginx_proxy_gzip = on +nginx_proxy_gzip_proxied = any +nginx_proxy_gzip_types = * +``` + +Or use equivalent environment variables: + +```bash +KONG_NGINX_PROXY_GZIP=on +KONG_NGINX_PROXY_GZIP_PROXIED=any +KONG_NGINX_PROXY_GZIP_TYPES=* +``` + +Those are then injected into the proxy server block portion of the template with the `nginx_proxy_` part removed. + +Only requests which set the `Accept-Encoding: gzip` header will result in compressed responses. Without this header, the response will be sent uncompressed even after enabling the relevant nginx directives. diff --git a/app/_support/how-to-expose-kong-metrics-with-prometheus-in-k8s.md b/app/_support/how-to-expose-kong-metrics-with-prometheus-in-k8s.md new file mode 100644 index 0000000000..6a513aa75d --- /dev/null +++ b/app/_support/how-to-expose-kong-metrics-with-prometheus-in-k8s.md @@ -0,0 +1,101 @@ +--- +title: How to expose Kong metrics with Prometheus in K8s +content_type: support +description: "Install Prometheus and configure a Kong Helm deployment with `serviceMonitor.enabled: true` so the Prometheus Operator can scrape Kong's metrics endpoint." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I expose Kong's metrics to Prometheus when Kong is deployed with Helm in Kubernetes? + a: | + Install Prometheus (e.g. via the `kube-prometheus-stack` Helm chart), set `serviceMonitor.enabled: true` in the Kong Helm chart's `values.yaml` so a `ServiceMonitor` object is created, and enable the Prometheus plugin globally to scrape Kong's proxy traffic metrics. +related_resources: + - text: Kong Helm chart Prometheus Operator integration documentation + url: https://github.com/Kong/charts/blob/main/charts/kong/README.md#prometheus-operator-integration + - text: "{{site.base_gateway}} Kubernetes install guide (Konnect)" + url: /gateway/install/kubernetes/konnect/ + - text: Kong Helm chart `ServiceMonitor` parameters + url: https://github.com/Kong/charts/blob/kong-2.41.0/charts/kong/README.md#general-parameters + - text: Prometheus plugin documentation + url: /plugins/prometheus/ +--- + +## Overview + +I installed Kong using a Helm chart in Kubernetes — how do I expose Kong's metrics to Prometheus? + +## Steps + +1. Install Prometheus + + ```bash + helm repo add prometheus-community https://prometheus-community.github.io/helm-charts + kubectl create ns monitoring + helm upgrade -i prometheus prometheus-community/kube-prometheus-stack \ + --namespace monitoring \ + --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \ + --set fullnameOverride=prometheus + kubectl get all -n monitoring + ``` + + This Prometheus deployment is only for demo purposes and isn't suitable for a production environment. + +2. Install Kong + + The following parameter is required in `values.yaml` when installing Kong with the Helm chart: + + ```yaml + serviceMonitor: + enabled: true + ``` + + Set this parameter in `values.yaml` and install Kong. + + This configuration creates a `ServiceMonitor` object in the same namespace as Kong, as shown in the example below, which tells Prometheus how to scrape metrics from Kong. + + ```yaml + apiVersion: monitoring.coreos.com/v1 + kind: ServiceMonitor + metadata: + labels: + app.kubernetes.io/instance: kong + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: kong + app.kubernetes.io/version: "3.14" + helm.sh/chart: kong-2.41.0 + name: kong-kong + namespace: kong + spec: + endpoints: + - scheme: http + targetPort: status + - scheme: http + targetPort: cmetrics + jobLabel: kong + namespaceSelector: + matchNames: + - kong + selector: + matchLabels: + app.kubernetes.io/instance: kong + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: kong + app.kubernetes.io/version: "3.14" + enable-metrics: "true" + helm.sh/chart: kong-2.41.0 + ``` + +3. Enable a global Prometheus plugin + + Enable the Prometheus plugin globally, then create `Service` and `Route` objects in Kong for proxy requests to upstreams. After that, send a few requests to Kong. + +4. Forward port 9090 of the Prometheus Kubernetes service to localhost and check the metrics + + ```bash + # Execute command to port forward 9090 port of the Prometheus K8S SVC to localhost + kubectl port-forward service/prometheus-prometheus -n monitoring 9090:9090 + ``` + + Now you can access `http://localhost:9090/` to open the Prometheus dashboard. You can see Kong's metrics on the Prometheus dashboard. diff --git a/app/_support/how-to-fake-down-status-endpoint.md b/app/_support/how-to-fake-down-status-endpoint.md new file mode 100644 index 0000000000..939365d059 --- /dev/null +++ b/app/_support/how-to-fake-down-status-endpoint.md @@ -0,0 +1,132 @@ +--- +title: "How to fake down `/status` endpoint" +content_type: support +description: "Simulate the `/status` endpoint going down on a standalone Kong node by looping it back through an upstream and marking the target unhealthy via the Admin API." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I temporarily disable or fake down the `/status` endpoint on a standalone Kong node? + a: | + On a standalone/traditional Kong node, loop the `/status` endpoint back through an upstream and mark its target unhealthy via the Admin API (`PUT /upstreams/{id}/targets/{id}/unhealthy`) so `/status` returns `503`; mark it healthy again to restore it. This doesn't work against a hybrid Control Plane's Admin API, which returns `404` for that endpoint. +--- + +## Overview + +We currently have an LB doing health checks for the `/status` endpoint at every nth second interval. If we want to do an upgrade to the Kong node, we will take down the Admin API but we don’t want the LB to route the request when the node is down during the interval gap. Is there any way we can turn off the `/status` endpoint? + +## Steps + +`/status` endpoint can be controlled by `status_listen` configuration in `kong.conf`. + +To turn it off, you need to restart kong but in case a restart is not possible, you can loopback the `/status` endpoint in Kong and use `Mark Healthy/Unhealthy` to turn it off/on using Kong upstream health check. + +This technique applies to standalone/traditional Kong nodes only. On a hybrid-mode Admin API, `PUT /upstreams/{id}/targets/{id}/unhealthy` returns `404`, so this approach does not work against a hybrid Control Plane's Admin API. + +On a standalone node, note that the health checker is "lazy": the first health-check-related request against a freshly created upstream/target can return `400` until a request has actually been proxied through it at least once. Send one warm-up request through the upstream before marking targets healthy/unhealthy; after that, the endpoint behaves as described below. + +Here are the steps to set it up: + +1. Create an upstream for loopbacking the admin API + +```bash + +http :8001/upstreams name=status_check -f +``` + +2. Set Admin API host as the target for the upstream `status_check` + +```bash + +http :8001/upstreams/status_check/targets target=:8001 -f +``` + +3. Create a service with path `/status` and a route for the upstream + +```bash + +http :8001/services name=status_health host=status_check path=/status -f +http :8001/services/status_health/routes name=status_route paths=/status -f +``` + +4. Right now we have Kong serving `/status` at the proxy port + +```bash + +http :8000/status +HTTP/1.1 200 OK + +{ + "database": { + "reachable": true + }, + "memory": { + "lua_shared_dicts": { + "kong": { + "allocated_slabs": "0.04 MiB", + "capacity": "5.00 MiB" + }, + "kong_healthchecks": { + "allocated_slabs": "0.04 MiB", + "capacity": "5.00 MiB" + }, + ... + }, + "workers_lua_vms": [ + { + "http_allocated_gc": "182.51 MiB", + "pid": 2914 + }, + ... + ] + }, + "server": { + "connections_accepted": 137, + "connections_active": 7, + "connections_handled": 137, + "connections_reading": 0, + "connections_waiting": 0, + "connections_writing": 7, + "total_requests": 137 + } +} +``` + +5. Mark the target unhealthy to fake the `/status` endpoint down + +```bash + +# the template +# :8001/upstreams//targets//[healthy/unhealthy] + +http :8001/upstreams/f3fa8cc2-003a-49fe-8734-c3f433281a46/targets/7355da4e-6684-4d3d-9890-863e2e444510/unhealthy +HTTP/1.1 204 No Content +Access-Control-Allow-Origin: * +Connection: keep-alive +``` + +6. Try to curl the `status` endpoint again + +```bash + +http :8000/status +HTTP/1.1 503 Service Temporarily Unavailable +Connection: keep-alive +Content-Length: 58 +``` + +You can see the endpoint returns 503 as it is currently unhealthy as an upstream. To set it healthy, use the endpoint in step 5. + +```bash + +http :8001/upstreams/f3fa8cc2-003a-49fe-8734-c3f433281a46/targets/7355da4e-6684-4d3d-9890-863e2e444510/healthy +HTTP/1.1 204 No Content +Access-Control-Allow-Origin: * +Connection: keep-alive + +http :8000/status +HTTP/1.1 200 OK +``` diff --git a/app/_support/how-to-force-a-login-into-the-idp-after-logging-out-of-kong-manager.md b/app/_support/how-to-force-a-login-into-the-idp-after-logging-out-of-kong-manager.md new file mode 100644 index 0000000000..e701a0c23b --- /dev/null +++ b/app/_support/how-to-force-a-login-into-the-idp-after-logging-out-of-kong-manager.md @@ -0,0 +1,49 @@ +--- +title: How to force a login into the IdP after logging out of Kong Manager +content_type: support +description: "Add `authorization_query_args_names` and `authorization_query_args_values` to your `admin_gui_auth_config` (or `KONG_ADMIN_GUI_AUTH_CONF`) to force a login into your IdP after logging out of Kong Manager." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I force a login into my IdP after an Admin logs out of Kong Manager, instead of Kong Manager silently logging them back in? + a: | + Add `"authorization_query_args_names": ["prompt"]` and `"authorization_query_args_values": ["login"]` to your `admin_gui_auth_config` (or the `KONG_ADMIN_GUI_AUTH_CONF` environment variable) for the `openid-connect` admin GUI auth method. These fields aren't overridden by Kong Manager, so they force the `prompt=login` parameter on the IdP redirect, requiring the admin to log in again instead of being silently re-authenticated. +--- + +## Overview + +We have enabled Kong Manager with openid-connect authentication. Currently when Admins log out of the Kong Manager, they can log back into the Manager without having to log into our IdP.We would like to make sure that after logging out of the Kong Manager that Admins will have to explicitly log into our IdP when they want to log into the Kong Manager again. + +## Steps + +This requirement can be achieved by adding two additional properties to the `admin_gui_auth_config` property or `KONG_ADMIN_GUI_AUTH_CONF` environment variable compared to the documented configuration : + +`"authorization_query_args_names": ["prompt"]` and `"authorization_query_args_values": ["login"]` + +A complete configuration would be similar to this. Note that for `admin_gui_auth=openid-connect`, the `logout_methods`, `logout_query_arg`, and `auth_methods` fields shown below are silently overridden/ignored by Kong (Kong Manager forces its own values for these so the admin login/logout flow works correctly), so they have no effect — they're harmless to leave in but aren't required for this technique. The `authorization_query_args_*` fields are not overridden and are what actually forces `prompt=login`: + +```bash + +admin_gui_auth_conf={ \ + "issuer": "", \ + "client_id": [""], \ + "client_secret": [""], \ + "consumer_by": ["username","custom_id"], \ + "ssl_verify": false, \ + "consumer_claim": ["email"], \ + "leeway": 60, \ + "redirect_uri": ["http://localhost:8002"], \ + "login_redirect_uri": ["http://localhost:8002"], \ + "logout_methods": ["GET", "DELETE"], \ + "logout_query_arg": "logout", \ + "logout_redirect_uri": ["http://localhost:8002"], \ + "scopes": ["openid","profile","email","offline_access"], \ + "auth_methods": ["authorization_code"], \ + "authorization_query_args_names": ["prompt"], + "authorization_query_args_values": ["login"] +} +``` diff --git a/app/_support/how-to-get-id-token-and-access-token-used-for-kong-manager-or-dev-portal-login.md b/app/_support/how-to-get-id-token-and-access-token-used-for-kong-manager-or-dev-portal-login.md new file mode 100644 index 0000000000..0d3fc5dbdb --- /dev/null +++ b/app/_support/how-to-get-id-token-and-access-token-used-for-kong-manager-or-dev-portal-login.md @@ -0,0 +1,73 @@ +--- +title: how to get id token and access token used for Kong Manager or Dev Portal login +content_type: support +description: "Configure `downstream_id_token_header` and `downstream_access_token_header` in the OpenID Connect config, then inspect the browser's network tab to find the id token and access token used for Kong Manager or Dev Portal login." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I find the id token and access token used for Kong Manager or Dev Portal OpenID Connect login? + a: | + Add `downstream_id_token_header` and `downstream_access_token_header` to your `admin_gui_auth_conf`/`KONG_ADMIN_GUI_AUTH_CONF` (Kong Manager) or `portal_auth_conf`/`KONG_PORTAL_AUTH_CONF` (classic Dev Portal), restart Kong, then log in and inspect the `id-token` and `access-token` response headers in your browser's network tab. Note: the classic Dev Portal is hard-deprecated and license-gated on current {{site.ee_product_name}}, so the Dev Portal steps are likely non-functional for most current customers. +--- + +## Overview + +I am using openid-connect to login Kong Manager and Dev Portal, but I am encountering a login issue. How could I check what is the id token and access token used in the login procedure? + +## Steps + +1. For the case you are using openid-connect to login Kong Manager + + 1. Please add below 2 parameters to `admin_gui_auth_conf` or `KONG_ADMIN_GUI_AUTH_CONF` + + ```json + "downstream_id_token_header": "id-token" + "downstream_access_token_header": "access-token" + ``` + + 2. Restart Kong + + 3. Login Kong Manager by using Chrome + + Keep opening the "Network" tab of "Developer Tools" in this procedure, + + Find below API and check its response header. + + ``` + http(s)://:/auth?state=xxx&session_state=yyy&code=zzz + ``` + + Then we will find the id-token response header shows the id token, + + the access-token response header shows the access token. + +2. For the case you are using openid-connect to login Dev Portal + + Note: the classic Dev Portal is hard-deprecated and license-gated on current {{site.ee_product_name}}, so this section is very likely non-functional for most current customers. The Kong Manager steps above remain fully valid. + + 1. Please add below 2 parameters to `portal_auth_conf` or `KONG_PORTAL_AUTH_CONF` + + ```json + "downstream_id_token_header": "id-token" + "downstream_access_token_header": "access-token" + ``` + + 2. Restart Kong + + 3. Login Dev Portal by using Chrome + + Keep opening the "Network" tab of "Developer Tools" in this procedure, + + Find below API and check its response header. + + ``` + http(s)://://session + ``` + + Then we will find the id-token response header shows the id token, + + the access-token response header shows the access token. diff --git a/app/_support/how-to-get-kong-audit-logs-out-to-an-external-system.md b/app/_support/how-to-get-kong-audit-logs-out-to-an-external-system.md new file mode 100644 index 0000000000..cf65de5590 --- /dev/null +++ b/app/_support/how-to-get-kong-audit-logs-out-to-an-external-system.md @@ -0,0 +1,28 @@ +--- +title: How to get Kong Audit Logs out to an external system +content_type: support +published: false +description: "Kong doesn't push Audit Logs to an external system; use the Admin API to pull them into your SIEM tool." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Can Kong push Audit Logs to an external system? + a: | + No — Kong doesn't support pushing Audit Logs to an external system. Use the Admin API to pull Audit Logs, since most SIEM tools support pulling from an API. +related_resources: + - text: Kong Audit Logs documentation + url: /gateway/audit-logs/ +--- + +## Overview + +Can the Kong Audit Logs be uploaded to an external system via a push mechanism from Kong? + +## Steps + +No, this feature is not available as a push operation from Kong. The Kong Admin API can be used to retrieve the Audit Logs as most SIEM tools have a mechanism to pull from an API. + +Please refer to the Kong documentation for examples of our Audit Logging. diff --git a/app/_support/how-to-grab-konnect-license-usage-via-admin-api.md b/app/_support/how-to-grab-konnect-license-usage-via-admin-api.md new file mode 100644 index 0000000000..ed040b5a4d --- /dev/null +++ b/app/_support/how-to-grab-konnect-license-usage-via-admin-api.md @@ -0,0 +1,35 @@ +--- +title: "{{site.konnect_product_name}}: How to grab Konnect license usage via Admin API" +content_type: support +published: false +description: "Query the Konnect Admin API's `/kbilling/v1/usage` endpoint with a personal access token (PAT) to check current Konnect license usage." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I check Konnect license usage using the Admin API? + a: | + Generate a personal access token (PAT) in the Konnect UI (Organization Settings > Personal Access Tokens), then call `GET https://global.api.konghq.com/kbilling/v1/usage` with that token to retrieve the same monthly request count shown in Konnect's Billing Settings. +--- + +## {{site.konnect_product_name}}: How to grab Konnect license usage via Admin API + +We see it is possible to view the license usage inside Konnect but we want to be able to pull this data programmatically. Is there anyway to use the Admin API to grab the license usage report? + +To grab this data using the Admin API you will need to generate a personal access token (PAT). You can generate one from the Konnect UI under Organization Settings > Personal Access Tokens. + +Once you have your PAT you can use the following command: + +```bash + +curl -X GET 'https://global.api.konghq.com/kbilling/v1/usage' --header 'Authorization: Bearer ' -s | jq .data.us.total_request_count +``` + +This will return the same monthly call count shown on the Konnect Billing Settings page. + +Command output: + +178241 diff --git a/app/_support/how-to-implement-guard-logic-for-buffering-and-body-existence-in-serverless-plugins.md b/app/_support/how-to-implement-guard-logic-for-buffering-and-body-existence-in-serverless-plugins.md new file mode 100644 index 0000000000..aa532075cd --- /dev/null +++ b/app/_support/how-to-implement-guard-logic-for-buffering-and-body-existence-in-serverless-plugins.md @@ -0,0 +1,92 @@ +--- +title: How to implement guard logic for buffered proxying and body existence in serverless plugins +content_type: support +description: Add guard logic to a `pre-function` plugin that checks `ngx.ctx.buffered_proxying` and confirms the request or response body is non-nil before serializing it, to prevent errors from nil bodies or disabled buffering. +tldr: + q: How do I guard against missing bodies and disabled buffering in a serverless plugin? + a: | + In a `pre-function` plugin, check `ngx.ctx.buffered_proxying` before reading a body, call `kong.service.request.enable_buffering()` when it is off, and verify the raw body is non-nil before serializing it. + This prevents errors when the body is `nil` or buffering is disabled. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +--- + +## Overview + +When working with request and response bodies, buffered proxying needs to be enabled and the body needs to exist to perform operations. + +Without guard logic in place, empty request / response bodies and incorrect buffering settings can cause the serverless plugin to throw errors in the Kong logs when the body is nil or buffering is disabled. + +## Steps + +Guard logic can be put in place to protect any operations that expect a body to be present. + +This is an example of how to configure a `pre-function` plugin to check for whether buffering is enabled, and if so serialize the request and response bodies to a log plugin output. + +1. Create `access-phase.lua` and add the following contents: + + ```lua + kong.log.warn("Access Start") + kong.log.warn("Checking buffering") + + if ngx.ctx.buffered_proxying then + kong.log.warn("Buffering already enabled.") + else + kong.log.warn("Enabling buffering...") + kong.service.request.enable_buffering() + end + + if not ngx.ctx.buffered_proxying then + kong.log.warn("Buffering was not enabled.") + else + kong.log.warn("Buffering check passed, retrieving request body") + local raw_body = kong.request.get_raw_body() + + if raw_body then + kong.log.warn("Valid request body found, serialising to log") + kong.log.set_serialize_value("request.body", raw_body) + else + kong.log.warn("No body to serialise") + end + end + + kong.log.warn("Access End") + ``` + +2. Create `log-phase.lua` and add the following contents: + + ```lua + kong.log.warn("Log Start") + + kong.log.warn("Checking buffering") + + if not ngx.ctx.buffered_proxying then + kong.log.warn("Buffering is disabled, cannot log the response body.") + else + kong.log.warn("Buffering check passed, retrieving response body") + local raw_response_body = kong.service.response.get_raw_body() + + if raw_response_body then + kong.log.warn("Valid response body found, serialising to log") + kong.log.set_serialize_value("response.body", raw_response_body) + else + kong.log.warn("No body to serialise") + end + end + ``` + +3. Add the `pre-function` plugin to a route/service using the files as inputs: + + ```bash + curl -X POST http://localhost:8001/routes//plugins \ + -F "name=pre-function" \ + -F "config.access=@./access-phase.lua" \ + -F "config.log=@./log-phase.lua" \ + -H "Kong-Admin-Token:" + ``` + +The Kong error log will show the various log messages as they are executed. diff --git a/app/_support/how-to-import-certificate-object-from-environment-variables-to-kong-by-using-secret-management-function.md b/app/_support/how-to-import-certificate-object-from-environment-variables-to-kong-by-using-secret-management-function.md new file mode 100644 index 0000000000..f4edc71bbc --- /dev/null +++ b/app/_support/how-to-import-certificate-object-from-environment-variables-to-kong-by-using-secret-management-function.md @@ -0,0 +1,61 @@ +--- +title: How to import a certificate object from environment variables to Kong by using the secret management function +content_type: support +description: "Export your certificate and private key as environment variables (e.g. `MY_SECRET_CERT` and `MY_SECRET_KEY`), then reference them with the `{vault://env/...}` syntax when creating a certificate object in Kong." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I import a certificate object into Kong from environment variables using the secrets management function? + a: | + Export the certificate and key as environment variables (e.g. `MY_SECRET_CERT` and `MY_SECRET_KEY`), then reference them in the certificate object's `cert` and `key` fields using `{vault://env/}` syntax — for example, `cert: '{vault://env/my-secret-cert}'` — either in declarative config or in Kong Manager. Restart Kong afterward. +related_resources: + - text: Kong deck sensitive data reference + url: /deck/gateway/sensitive-data/#main + - text: "{{site.base_gateway}} secrets management documentation" + url: /gateway/secrets-management/ +--- + +## Overview + +Kong now allows you to import a certificate object from environment variables. What are the steps to implement this? + +## Steps + +1. Export your certificate and key as environment variables. + + Here we use `MY_SECRET_CERT` for the public certificate and `MY_SECRET_KEY` for the private key. + + ```bash + export MY_SECRET_CERT='-----BEGIN CERTIFICATE----- + + -----END CERTIFICATE-----' + + export MY_SECRET_KEY='-----BEGIN PRIVATE KEY----- + + -----END PRIVATE KEY-----' + ``` + +2. Reference `MY_SECRET_CERT` and `MY_SECRET_KEY` to create the certificate object in Kong. + + 1. If you use a declarative configuration YAML file, create the certificate object like this: + + ```yaml + _format_version: "3.0" + _transform: true + + certificates: + - id: b0dbe8fd-e5e6-414a-a0dc-0160665620ab + cert: '{vault://env/my-secret-cert}' + key: '{vault://env/my-secret-key}' + ``` + + 2. If you use Kong Manager, create the certificate object as shown below: + +3. Restart Kong. + + ```bash + kong restart + ``` diff --git a/app/_support/how-to-inject-the-same-nginx-directive-with-multiple-different-values.md b/app/_support/how-to-inject-the-same-nginx-directive-with-multiple-different-values.md new file mode 100644 index 0000000000..c92f0990ee --- /dev/null +++ b/app/_support/how-to-inject-the-same-nginx-directive-with-multiple-different-values.md @@ -0,0 +1,33 @@ +--- +title: How to inject the same nginx directive with multiple different values +content_type: support +description: Kong doesn't support specifying the same NGINX directive multiple times with different values, but you can work around this by defining multiple complete directive values in a single config property. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I set the same NGINX directive multiple times with different values in Kong? + a: | + Kong doesn't support repeating the same NGINX directive with different values directly. Work around this by defining multiple complete directive values within a single `nginx_proxy_*` config property (or `KONG_NGINX_PROXY_*` environment variable), separated by semicolons, with the last value not ending in a semicolon. +--- + +## Overview + +I need to use the Nginx directive injection feature documented here but I have multiple values for the same directive, and specifying the directive multiple times or adding multiple values to the same directive doesn't work. How can I overcome this issue? + +## Steps + +Currently, it is not possible to specify the same directive multiple times with different values but it is possible to work around the issue by adding the multiple complete directive definitions for one variable injection. + +For example if you want to add multiple `add_header` nginx directives with different values, you could achieve this by specifying them in the relevant kong property or KONG variable like this, i.e the first value is specified normally, and all other values are added to that value by specifying the whole directive. Please note that the directives need to be separated by a semicolon but that the last directive must NOT end in a semicolon: + +``` +nginx_proxy_add_header = Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0'; add_header X-Frame-Options 'sameorigin'; add_header X-XSS-Protection '1; mode=block'; add_header X-Content-Type-Options 'nosniff'; add_header X-Permitted-Cross-Domain-Policies 'master-only'; add_header Strict-Transport-Security 'max-age=31536000' always +``` + +``` +KONG_NGINX_PROXY_ADD_HEADER: Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0'; add_header X-Frame-Options 'sameorigin'; add_header X-XSS-Protection '1; mode=block'; add_header X-Content-Type-Options 'nosniff'; add_header X-Permitted-Cross-Domain-Policies 'master-only'; add_header Strict-Transport-Security 'max-age=31536000' always +``` diff --git a/app/_support/how-to-log-response-body-and-upstream-by-using-pre-function-plugin.md b/app/_support/how-to-log-response-body-and-upstream-by-using-pre-function-plugin.md new file mode 100644 index 0000000000..6db9ec2e44 --- /dev/null +++ b/app/_support/how-to-log-response-body-and-upstream-by-using-pre-function-plugin.md @@ -0,0 +1,45 @@ +--- +title: How to log response body and upstream by using pre-function plugin +content_type: support +description: Use the pre-function plugin to log the response body and the resolved upstream for a service. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: pre-function plugin + url: /plugins/pre-function/ +tldr: + q: How do I log the response body and upstream target using the pre-function plugin? + a: | + Enable a `pre-function` plugin on the service with a `config.log` phase function that calls `kong.log(kong.service.response.get_raw_body())`, and a `config.access` phase function that enables request buffering and logs `ngx.ctx.balancer_data` fields. The response body and resolved upstream then appear in Kong's error log for each request. Logging the response body forces Nginx to buffer the full body, which affects proxy performance for large responses. +--- + +## Overview + +How to log response body and upstream by using pre-function plugin + +## Steps + +See the pre-function plugin docs to learn how to enable a pre-function plugin. + +Use the command below to enable a pre-function plugin with a Lua function to log the response body on a service: + +```bash +curl -X POST http://\:8001/services/\/plugins \ + --data "name=pre-function" \ + --data "config.log[1]=kong.log(string.gsub(kong.service.response.get_raw_body(), '\n', ''))" \ + --data "config.access[1]=kong.service.request.enable_buffering()" \ + --data "config.access[2]=kong.log(ngx.ctx.balancer_data.scheme, '://', ngx.ctx.balancer_data.host, ':', ngx.ctx.balancer_data.port, ngx.var.upstream_uri)" +``` + +If you send a request to the route associated with the service that has the pre-function plugin enabled, you should be able to see the response body inside `kong error.log`: + +``` +2026/05/07 07:03:01 [notice] 27#0: *207173 [kong] ',..."]:1 [pre-function] https://httpbin.org:443/anything, client: 172.21.0.1, server: kong, request: "GET /ttt HTTP/1.1", host: "localhost:8000" +... +2026/05/07 07:03:02 [notice] 27#0: *207173 [kong] [string "kong.log(string.gsub(kong.service.response.ge..."]:1 [pre-function] { "args": {}, "data": "", "files": {}, "form": {}, "headers": { "Accept": "*/*", "Host": "httpbin.org", "User-Agent": "curl/7.64.1", "X-Amzn-Trace-Id": "Root=1-6094e626-44a735441ee0fc2b51ef72a0", "X-Forwarded-Host": "localhost", "X-Forwarded-Path": "/ttt", "X-Forwarded-Prefix": "/ttt" }, "json": null, "method": "GET", "origin": "172.21.0.1, 175.177.45.138", "url": "https://localhost/anything"}19 while logging request, client: 172.21.0.1, server: kong, request: "GET /ttt HTTP/1.1", host: "localhost:8000" +``` + +Be aware that logging the response body requires Nginx to buffer the whole body, which affects proxy performance when the body is large. diff --git a/app/_support/how-to-mount-a-persistentvolume-using-the-kong-helm-chart.md b/app/_support/how-to-mount-a-persistentvolume-using-the-kong-helm-chart.md new file mode 100644 index 0000000000..0e08e61977 --- /dev/null +++ b/app/_support/how-to-mount-a-persistentvolume-using-the-kong-helm-chart.md @@ -0,0 +1,47 @@ +--- +title: How to mount a PersistentVolume using the Kong Helm Chart +content_type: support +description: To mount a `PersistentVolume` using the Kong Helm chart, declare the volume under `deployment.userDefinedVolumes` and `deployment.userDefinedVolumeMounts` in `values.yaml`. +products: + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: the documentation + url: https://github.com/Kong/charts/blob/main/charts/kong/README.md#user-defined-volumes + - text: the Kubernetes docs + url: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims +tldr: + q: How do I mount a PersistentVolume using the Kong Helm Chart? + a: | + Declare the volume under `deployment.userDefinedVolumes` and `deployment.userDefinedVolumeMounts` in your Helm `values.yaml` (for example, referencing a `PersistentVolumeClaim`) so it's mounted in all Kong containers. The PV and PVC must already exist in Kubernetes, or the pods will wait indefinitely for the claim. +--- + +## Overview + +How do I mount a PersistentVolume using the Kong Helm Chart? + +## Steps + +To mount a `PersistentVolume`, you need to declare the volume in the `deployment.userDefinedVolumes` and `deployment.userDefinedVolumeMounts` sections. You can find the details in the documentation. + +After declaring the `userDefinedVolumes` and `userDefinedVolumeMounts`, the volume will be mounted in all the Kong containers. + +If you want to use a `PersistentVolumeClaim`, you can declare it in the same way as an `emptyDir`, but you must create the PV and PVC claim in Kubernetes, otherwise the pods will not be created, waiting for the PVC. You can find the details in the Kubernetes docs. + +As an example, the `values.yaml` file will be like: + +```yaml +deployment: + # Define any volumes and mounts you want present in the Kong proxy container + userDefinedVolumes: + - name: "test-volume" + persistentVolumeClaim: + claimName: myclaim + userDefinedVolumeMounts: + - name: "test-volume" + mountPath: "/tmp/kong_volumes/mount" +``` + +Assuming the PVC `myclaim` exists in the Kong namespace. diff --git a/app/_support/how-to-mount-configmap-and-secret-when-kong-installed-by-helm.md b/app/_support/how-to-mount-configmap-and-secret-when-kong-installed-by-helm.md new file mode 100644 index 0000000000..251d6bc856 --- /dev/null +++ b/app/_support/how-to-mount-configmap-and-secret-when-kong-installed-by-helm.md @@ -0,0 +1,120 @@ +--- +title: How to mount configmap and secret when kong installed by helm +content_type: support +published: false +description: Steps to mount `ConfigMaps` and `Secrets` as folders or files in Kong pods using `extraConfigMaps` and `extraSecrets` in the Kong Helm chart values. +products: + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I mount ConfigMaps and Secrets as folders or files in Kong pods installed with the Helm chart? + a: | + Create the `ConfigMap` or `Secret` with `kubectl`, then reference it under `extraConfigMaps` or `extraSecrets` in the Kong Helm chart values (with an optional `subPath` to mount a single file instead of a folder), and reinstall or upgrade the Helm release. See the FAQs below for a full walkthrough and multi-resource examples. +faqs: + - q: How to mount a `configmap` to a folder in the pod? + a: | + step 1: create a configmap from a file + + ```bash + kubectl create configmap --from-file= -n + ``` + + step 2: add extraConfigMaps section in helm chart like below + + ```yaml + extraConfigMaps: + - name: + mountPath: (e.g /test) + ``` + + step 3: re-install /upgrade kong helm chart + + * Please notice the whole folder mount in the pod will be overwritten by the configmap + - q: How to mount a `configmap` to a file in the pod? + a: | + step 1: create a configmap from a file + + ```bash + kubectl create configmap --from-file= -n + ``` + + step 2: add extraConfigMaps section in helm chart like below + + ```yaml + extraConfigMaps: + - name: + mountPath: (e.g /test/a.txt) + subPath: (e.g a.txt) + ``` + + step 3: re-install /upgrade kong helm chart + - q: How to mount a `secret` to a folder in the pod? + a: | + step 1: create a secret from a file + + ```bash + kubectl create secret generic --from-file= -n + ``` + + step 2: add `extraSecrets` section in helm chart like below + + ```yaml + extraSecrets: + - name: + mountPath: (e.g /test) + ``` + + step 3: re-install /upgrade kong helm chart + + * Please notice the whole folder mount in the pod will be overwritten by the secret + - q: How to mount a `secret` to a file in the pod? + a: | + step 1: create a secret from a file + + ```bash + kubectl create secret generic --from-file= -n + ``` + + step 2: add `extraSecrets` section in helm chart like below + + ```yaml + extraSecrets: + - name: + mountPath: (e.g /test/a.txt) + subPath: (e.g a.txt) + ``` + + step 3: re-install /upgrade kong helm chart + - q: Is it possible to mount multiple `configmaps` or `secrets` in the pod? + a: | + Yes, you could write multiple extraConfigMaps in helm chart like below + + ```yaml + extraConfigMaps: + - name: + mountPath: (e.g /test/a.txt) + subPath: (e.g a.txt) + - name: + mountPath: (e.g /test/b.txt) + subPath: (e.g b.txt) + ``` + + you could write multiple `extraSecrets` in helm chart like below + + ```yaml + extraSecrets: + - name: + mountPath: (e.g /test/a.txt) + subPath: (e.g a.txt) + - name: + mountPath: (e.g /test/b.txt) + subPath: (e.g b.txt) + ``` +--- + +## Overview + +Kong pods can mount `ConfigMaps` and `Secrets` as folders or files when Kong is installed with the Helm chart, including mounting multiple `ConfigMaps` or `Secrets` at once. See the FAQs below for step-by-step instructions. diff --git a/app/_support/how-to-pass-optional-field-as-empty-string-in-request-body-using-the-equest-validator-plugin.md b/app/_support/how-to-pass-optional-field-as-empty-string-in-request-body-using-the-equest-validator-plugin.md new file mode 100644 index 0000000000..03f7d2bc70 --- /dev/null +++ b/app/_support/how-to-pass-optional-field-as-empty-string-in-request-body-using-the-equest-validator-plugin.md @@ -0,0 +1,84 @@ +--- +title: How to pass optional field as empty string in request body using the request validator plugin +content_type: support +published: false +description: Set the request validator plugin's `len_min` parameter to `0` to allow an optional string field to accept an empty string value. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: the related documentation + url: /plugins/request-validator/#body-schema-definition +tldr: + q: How do I let an optional field accept an empty string in the request validator plugin's schema? + a: | + Set the field's `len_min` parameter to `0` in the request validator plugin's schema. Without it, an empty string value fails with a "request body doesn't conform to schema" error. +--- + +## Overview + +How to pass an optional field as empty string in request body using request validator. + +When using request validator plugin to validate request body and trying to pass empty string value in one of the optional fields in request body using the code below, produces an error that the request body doesn't conform to schema + +```json +{ +"location": { +"type": "string", +"required": false +} +``` + +## Steps + +The solution would be to set the `len_min` parameter to `0` like so: + +```json +[ { "location": { "type": "string", "required": false, "len_min": 0 }} ] +``` + +With that setting it works with a 0 length string. + +```bash +curl -k --header 'Content-Type: application/json' https://proxy.kong.lan/httpbin/anything --data '{ "location": "" }' +{ + "args": {}, + "data": "{ \"location\": \"\" }", + "files": {}, + "form": {}, + "headers": { + "Accept": "*/*", + "Connection": "keep-alive", + "Content-Length": "18", + "Content-Type": "application/json", + "Host": "kongpose_httpbin_1", + "User-Agent": "curl/7.79.1", + "X-Forwarded-Host": "proxy.kong.lan", + "X-Forwarded-Path": "/httpbin/anything", + "X-Forwarded-Prefix": "/httpbin" + }, + "json": { + "location": "" + }, + "method": "POST", + "origin": "172.20.0.1, 172.20.0.31", + "url": "http://proxy.kong.lan/anything" +} +``` + +But with this body schema + +```json +[ { "location": { "type": "string", "required": false }} ] +``` + +we get an error + +```bash +curl -k --header 'Content-Type: application/json' https://proxy.kong.lan/httpbin/anything --data '{ "location": "" }' +{"message":"request body doesn't conform to schema"} +``` + +The related documentation can be found here. diff --git a/app/_support/how-to-prevent-data-planes-from-accepting-traffic-before-they-have-downloaded-the-configuration-from-the-control-plane.md b/app/_support/how-to-prevent-data-planes-from-accepting-traffic-before-they-have-downloaded-the-configuration-from-the-control-plane.md new file mode 100644 index 0000000000..fef334fd4d --- /dev/null +++ b/app/_support/how-to-prevent-data-planes-from-accepting-traffic-before-they-have-downloaded-the-configuration-from-the-control-plane.md @@ -0,0 +1,41 @@ +--- +title: How to prevent data planes from accepting traffic before they have downloaded the configuration from the control plane +content_type: support +description: An easy way to address this issue is to change the default readiness probe to use a health check route. +products: + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I prevent Kong data planes from accepting traffic before they've downloaded their configuration from the control plane? + a: | + Kong ships a native `/status/ready` endpoint on the status port that returns 503 until the data plane has received and applied its configuration from the control plane, then returns 200. It's the default `readinessProbe` shipped in the Kong Helm chart, so most deployments don't need any custom configuration. Point your Kubernetes readiness probe at `/status/ready` instead of relying on a custom `/health` route workaround. +--- + +## Overview + +We are using Kong helm charts with a hybrid mode setup, and noticed that when new data planes come up while the control plane is unavailable that the data planes are marked as ready, and receive traffic even though they don't have the configuration yet. This results in 404 errors. How can we avoid Kong data plane pods being marked as ready before they have received the configuration from the control plane? + +## Steps + +Kong now ships a native `/status/ready` endpoint on the status port that already gates readiness on config-sync status: it returns a 503 until the data plane has received and applied its configuration from the control plane, then returns 200. This is the recommended solution, and it's the default `readinessProbe` shipped in the Kong Helm chart (`Kong/charts`) — most deployments won't need any custom configuration. + +With a helm chart deployment, the `readinessProbe` for data-planes should look like this in `values.yaml`: + +```yaml + +readinessProbe: + httpGet: + path: "/status/ready" + port: "status" + scheme: HTTP + initialDelaySeconds: 5 + timeoutSeconds: 5 + periodSeconds: 10 + successThreshold: 1 + failureThreshold: 3 +``` + +Previously, before `/status/ready` was available, a workaround was to create a custom `/health` route with a request-termination plugin returning 200, and point the readiness probe at that route on the proxy port instead. That approach still works, but it has a real downside compared to `/status/ready`: every readiness check is a proxy request and counts towards your overall request quota. Since `/status/ready` runs on the status port and is purpose-built for this check, the custom `/health` workaround is very likely unnecessary on current {{site.base_gateway}} versions. diff --git a/app/_support/how-to-proxy-request-to-a-specific-upstream-path-when-the-upstream-root-path-already-exists.md b/app/_support/how-to-proxy-request-to-a-specific-upstream-path-when-the-upstream-root-path-already-exists.md new file mode 100644 index 0000000000..4ab7be311d --- /dev/null +++ b/app/_support/how-to-proxy-request-to-a-specific-upstream-path-when-the-upstream-root-path-already-exists.md @@ -0,0 +1,58 @@ +--- +title: How to proxy request to a specific upstream path when the upstream root path already exists by using `request-transformer` plugin +content_type: support +description: This article shows how to create a dynamic route path by using the `request-transformer` plugin. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: request-transformer plugin documentation + url: /plugins/request-transformer/ +tldr: + q: How do I route a request to a specific upstream path when the Route already points to the upstream's root path? + a: | + Use the `request-transformer` plugin with a regex capture group on the Route path, then set `Config.Replace.Uri` to `/$(uri_captures.g1)` so Kong rewrites the request to the captured upstream path before proxying it. +--- + +## Overview + +It is a common case that there are several routes pointing to one service. For example, assuming we are using an echo application as upstream and running it in `http://kong.example` for testing purpose. There is an existing service entity called `exampleservice` in Kong, and its URL parameter is defined as `http://kong.example`. And, we have a route entity with a path `/example`, and `strip_path: true`. This route will proxy requests to `http://kong.example` via Kong. Now, you'd like to enable a plugin only on `http://kong.example/api/v1/secret`. One method is to create another pair of service and route. But with Kong `request-transformer` plugin, you can realize it without creating a duplicated service entity of the same application. + +## Steps + +This article shows how to create a dynamic route path by using the `request-transformer` plugin. + +1. Define the route path in regex format in the Route entity and point this route to the service `exampleservice`. + + In versions prior to 3.x: + + ``` + /example/(?api/v1/secret) + ``` + + In versions from 3.x onwards, regex paths have to be prefixed with `~`: + + ``` + ~/example/(?api/v1/secret) + ``` + +2. Enable the `request-transformer` plugin for the route. + +3. Set `Config.Replace.Uri` of the `request-transformer` plugin as below: + + ``` + /$(uri_captures.g1) + ``` + +4. Test the settings. The request is transformed and sent to `/api/v1/secret` to your application. + + ```bash + curl -i {kong proxy}/example/api/v1/secret + HTTP/1.1 200 OK + .... + + HTTP/1.1 GET /api/v1/secret. <--- request URI has been transformed to /api/v1/secret + (echo application shows which path has been accessed as above) + ``` diff --git a/app/_support/how-to-proxy-soap-requests-in-kong.md b/app/_support/how-to-proxy-soap-requests-in-kong.md new file mode 100644 index 0000000000..e1967c91a7 --- /dev/null +++ b/app/_support/how-to-proxy-soap-requests-in-kong.md @@ -0,0 +1,100 @@ +--- +title: How to proxy SOAP requests in Kong +content_type: support +description: "Kong proxies SOAP requests like any other HTTP request, but proxying an HTTPS SOAP upstream requires uploading the full CA certificate chain to Kong's `ca_certificates` entity." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I proxy SOAP requests in Kong? + a: | + Kong proxies SOAP requests like any other HTTP request — no special configuration is needed to route them. If the SOAP upstream uses HTTPS, upload each certificate in the trust chain (including the root CA) as its own `ca_certificates` entity and list all the resulting IDs on the service, since Kong only accepts one certificate per `ca_certificates` entity. +--- + +## Overview + +How to proxy SOAP requests in Kong + +## Steps + +Kong can proxy SOAP requests like any other HTTP request. The limitations come in when dealing with transforming and analyzing the data inside the request, however proxying is no problem. + +Note: Kong verifies the upstream's TLS certificate by default. If the upstream is HTTPS, the service needs `ca_certificates` entries (and, depending on the certificate chain depth, `tls_verify_depth`) or the proxied request will fail with an upstream SSL verify error. Kong's `ca_certificates` entity only accepts one certificate per entity (`POST /ca_certificates` with more than one `-----BEGIN CERTIFICATE-----` block in `cert` fails schema validation with `"please submit only one certificate at a time"`), so each certificate in the trust chain must be uploaded as its own separate `ca_certificates` entity, with every resulting ID listed in the service's `ca_certificates` array. Also note the certificates the upstream sends during the handshake (visible via `openssl s_client -showcerts`) may not be the full trust path Kong needs — a cross-signed intermediate's own issuer (the actual self-signed root CA) can be missing from what the server presents, and that root must be uploaded too or verification still fails with `"unable to get issuer certificate"`. + +See the below example: + +Use decK to sync the following YAML and set up the route and service for the test: + +```yaml + +_format_version: "1.1" +services: +- connect_timeout: 60000 + enabled: true + host: www.dataaccess.com + name: soap-svc + path: /webservicesserver/NumberConversion.wso + port: 443 + protocol: https + read_timeout: 60000 + retries: 5 + ca_certificates: + - + - + - + tls_verify_depth: 5 + routes: + - https_redirect_status_code: 426 + name: soap-rt + path_handling: v0 + paths: + - /soap-test + preserve_host: false + protocols: + - http + - https + regex_priority: 0 + request_buffering: true + response_buffering: true + strip_path: true + write_timeout: 60000 +``` + +``, ``, `` refer to the target's CA certificate chain (intermediate(s) plus the ultimate root CA), each uploaded beforehand as its own separate `ca_certificates` entity in Kong (one certificate per entity) and referenced here by ID. Live-tested against `www.dataaccess.com` on {{site.base_gateway}} 3.14.0.0: uploading only the two non-leaf certificates the server itself presents during the TLS handshake produced a persistent `upstream SSL certificate verify error: (2: unable to get issuer certificate)`; the fix was uploading the actual trusted root CA (in this case Let's Encrypt's `ISRG Root X1`, obtainable from any standard CA bundle) as a third `ca_certificates` entity and adding its ID to the array above — after that, and after allowing a few seconds for the config to sync to the Data Plane, the request succeeded end-to-end. + +This will setup an upstream SOAP service pointing to the upstream: https://www.dataaccess.com/webservicesserver/NumberConversion.wso + +You should then be able to send the following SOAP payload to the upstream via this curl command: + +```bash + +curl --location --request POST 'http://kong-proxy:8000/soap-test' -k \ +--header 'Content-Type: text/xml; charset=utf-8' \ +--data-raw ' + + + +500 + + +' +``` + +Which should result in the following response: + +``` + + + + + + five hundred + + + +``` + +You have successfully proxied SOAP traffic to a SOAP upstream. diff --git a/app/_support/how-to-recreate-kong-admin-user.md b/app/_support/how-to-recreate-kong-admin-user.md new file mode 100644 index 0000000000..0fed3572d0 --- /dev/null +++ b/app/_support/how-to-recreate-kong-admin-user.md @@ -0,0 +1,56 @@ +--- +title: How to recreate `kong_admin` user +content_type: support +description: "How to recreate a deleted `kong_admin` super admin user, either through another existing super admin or by temporarily disabling RBAC." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I recreate the `kong_admin` user after it's been accidentally deleted? + a: | + If another admin already has the default workspace's super admin role, use it to invite a new `kong_admin` and generate a fresh token — no downtime required. If no other super admin exists, temporarily disable RBAC (`KONG_ENFORCE_RBAC`, `KONG_ADMIN_GUI_AUTH`, and `KONG_ADMIN_GUI_SESSION_CONF` must all be unset together) to access Kong manager without authentication, recreate `kong_admin` with the super admin role, then re-enable RBAC. +--- + +## Overview + +The default workspace super admin `kong_admin` might be accidentally deleted. This article tells how to recreate the `kong_admin` user. + +## Steps + +If you have another admin user with the default workspace’s super admin role, you can re-create the super admin by the below method. + +1. Login to Kong manager using another admin user name with the super admin role. +2. Invite admin and grant the super admin role to the user. +3. Login to Kong manager as the new admin user. +4. Re-generate the token and use the new token. + +If you don't have another admin user with the default workspace’s super admin role, please follow the below steps. + +1. Disable RBAC by commenting out the below configurations, and restart Kong. + + Example + + ```bash + + # - KONG_ENFORCE_RBAC=on + # - KONG_ADMIN_GUI_AUTH=basic-auth + # - KONG_ADMIN_GUI_SESSION_CONF={"secret":"secret","storage":"kong","cookie_secure":false} + ``` + + Warning: if `KONG_ADMIN_GUI_AUTH` or `KONG_ADMIN_GUI_SESSION_CONF` are left set while `KONG_ENFORCE_RBAC` is disabled, Kong will fail to start with the explicit error `Error: enforce_rbac must be enabled when admin_gui_auth is enabled` — so all three must be disabled together. (The reverse is not a problem: `KONG_ENFORCE_RBAC=on` by itself, with no `admin_gui_auth`/`admin_gui_session_conf` set, is a normal, valid configuration and starts fine — RBAC-protecting the Admin API doesn't require Kong Manager GUI auth to also be configured.) + +2. Open Kong manager -> invite admin-> to make the user name as `kong_admin`, fill in the email and grant the default workspace’s super admin role to the user. + + If you have an SMTP server configured, you will receive a registration email. Please skip to step 6. + + If there is no SMTP server configured, please continue to do steps 3 and 5. But step 4 is still necessary. + +3. Click the `kong_admin` in the "invited" list -> click 'generate registration URL', and note this URL. +4. Enable RBAC by commenting in the configurations in step 1, and restart Kong. +5. Access the `/`. +6. Reset password. +7. Login Kong manager as `kong_admin`. +8. Reset token and use the new token. diff --git a/app/_support/how-to-recreate-the-default-roles-in-the-default-workspace-after-they-got-deleted-by-mistake.md b/app/_support/how-to-recreate-the-default-roles-in-the-default-workspace-after-they-got-deleted-by-mistake.md new file mode 100644 index 0000000000..c26d974690 --- /dev/null +++ b/app/_support/how-to-recreate-the-default-roles-in-the-default-workspace-after-they-got-deleted-by-mistake.md @@ -0,0 +1,91 @@ +--- +title: How to recreate the default roles in the default workspace if accidentally deleted +content_type: support +description: "How to recreate Kong's three default RBAC roles (`super-admin`, `admin`, `read-only`) in the default workspace after they've been accidentally deleted." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I recreate Kong's default RBAC roles (`super-admin`, `admin`, `read-only`) in the default workspace after they've been deleted? + a: | + If you still have another `super-admin`, use the Admin API to recreate the missing role(s) and their endpoint permissions directly — add `-H kong-admin-token: ` to authenticate. If no `super-admin` remains, you must temporarily disable RBAC (`KONG_ENFORCE_RBAC`, `KONG_ADMIN_GUI_AUTH`, and `KONG_ADMIN_GUI_SESSION_CONF` together) to recreate the roles via the Admin API, then re-enable RBAC. An admin can't delete or reassign their own role — use a different admin account for that step. If you can't turn off RBAC and need to recreate `super-admin` itself, contact Kong Support. +--- + +## Overview + +We have deleted one or more of the three default RBAC roles in the default workspace but still need them. How do I recreate the role? + +## Steps + +Once the `super-admin` role has been deleted from the default workspace or you do not have any Admin user who is a member of that role, you have to turn off RBAC before being able to re-create any of the roles. + +Warning: turning off RBAC alone is not enough if `admin_gui_auth`/`admin_gui_session_conf` are configured. `KONG_ENFORCE_RBAC`, `KONG_ADMIN_GUI_AUTH`, and `KONG_ADMIN_GUI_SESSION_CONF` must all be disabled together — if RBAC is disabled while `admin_gui_auth`/`admin_gui_session_conf` remain set, Kong fails to start with `Error: enforce_rbac must be enabled when admin_gui_auth is enabled` (live-confirmed). Note this is one-directional: `enforce_rbac=on` by itself, with no `admin_gui_auth`/`admin_gui_session_conf` configured at all, boots fine and correctly enforces RBAC on the Admin API — that combination is not a problem. + +Note: if you need to delete an existing role as part of recovering from this (for example, a partially-recreated or duplicate role), an admin cannot delete a role they currently hold themselves — that request returns a 403 (live-confirmed: `DELETE /rbac/roles/` → `403 {"message":"the admin should not delete their own roles"}`, while deleting a role the same admin does not hold succeeds normally). This also extends to modifying your own role assignments: `POST`/`PUT .../admins//roles` for the currently-authenticated admin fails with `403 {"message":"the admin should not update their own roles"}` — live-confirmed. If you're using the "still have a `super-admin` user" path below to re-associate a role, associate it to a *different* admin account, not the one whose credentials you're currently using. You can only delete/reassign a role you are not a member of; use a different admin account or unassign the role first. + +After disabling RBAC (and the two admin GUI auth settings above), you can recreate the three default roles and their permissions with the following API calls: + +**`super-admin` role:** + +1. Create role: + + ```bash + curl -v -X POST /default/rbac/roles -H "content-type: application/json" -d "{\"name\":\"super-admin\",\"comment\":\"Full access to all endpoints, across all workspaces\"}" + ``` + +2. Create permissions: + + ```bash + curl -v -X POST /default/rbac/roles/super-admin/endpoints -H "content-type: application/json" -d "{\"workspace\":\"*\",\"actions\":\"delete,create,update,read\",\"endpoint\":\"*\",\"negative\":false}" + ``` + +3. Associate role to existing Admin user: + + ```bash + curl -v -X POST /default/admins//roles -H "content-type: application/json" -d "{\"roles\":\"super-admin\"}" + ``` + +**`read-only` role:** + +1. Create role: + + ```bash + curl -v -X POST /default/rbac/roles -H "content-type: application/json" -d "{\"name\":\"read-only\",\"comment\":\"Read access to all endpoints, across all workspace\"}" + ``` + +2. Create permissions: + + ```bash + curl -v -X POST /default/rbac/roles/read-only/endpoints -H "content-type: application/json" -d "{\"workspace\":\"*\",\"actions\":\"read\",\"endpoint\":\"*\",\"negative\":false}" + ``` + +**`admin` role:** + +1. Create role: + + ```bash + curl -v -X POST /default/rbac/roles -H "content-type: application/json" -d "{\"name\":\"admin\",\"comment\":\"Full access to all endpoints, across all workspaces—except RBAC Admin API\"}" + ``` + +2. Create permissions: + + ```bash + curl -v -X POST /default/rbac/roles/admin/endpoints -H "content-type: application/json" -d "{\"workspace\":\"*\",\"actions\":\"delete,create,update,read\",\"endpoint\":\"*\",\"negative\":false}" + + curl -v -X /default/rbac/roles/admin/endpoints -H "content-type: application/json" -d "{\"workspace\":\"*\",\"actions\":\"delete,create,update,read\",\"endpoint\":\"rbac/*\",\"negative\":true}" + + curl -v -X POST /default/rbac/roles/admin/endpoints -H "content-type: application/json" -d "{\"workspace\":\"*\",\"actions\":\"delete,create,update,read\",\"endpoint\":\"rbac/*/*\",\"negative\":true}" + + curl -v -X POST /default/rbac/roles/admin/endpoints -H "content-type: application/json" -d "{\"workspace\":\"*\",\"actions\":\"delete,create,update,read\",\"endpoint\":\"rbac/*/*/*\",\"negative\":true}" + + curl -v -X POST /default/rbac/roles/admin/endpoints -H "content-type: application/json" -d "{\"workspace\":\"*\",\"actions\":\"delete,create,update,read\",\"endpoint\":\"rbac/*/*/*/*\",\"negative\":true}" + + curl -v -X POST /default/rbac/roles/admin/endpoints -H "content-type: application/json" -H "Kong-Admin-Token: $3" -d "{\"workspace\":\"*\",\"actions\":\"delete,create,update,read\",\"endpoint\":\"rbac/*/*/*/*/*\",\"negative\":true}" + ``` + +If you still have a `super-admin` user, and just want to recreate one or both of the other default roles, you can use the above API calls with the addition of the kong-admin-token header: `-H kong-admin-token: ` + +If you can not turn off RBAC but need to re-create the `super-admin` role, please contact Kong Support. diff --git a/app/_support/how-to-redirect-routes-based-on-existence-of-a-header.md b/app/_support/how-to-redirect-routes-based-on-existence-of-a-header.md new file mode 100644 index 0000000000..108c75da43 --- /dev/null +++ b/app/_support/how-to-redirect-routes-based-on-existence-of-a-header.md @@ -0,0 +1,158 @@ +--- +title: How to redirect routes based on existence of a header +content_type: support +description: "Use a Request Termination plugin and an Exit Transformer plugin to redirect requests to different routes based on whether a header like `x-vendor` is present." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I redirect a request to a different route based on whether a header exists? + a: | + Use a Request Termination plugin and an Exit Transformer plugin on a shared "main" route: the Exit Transformer's function checks for the header (for example `x-vendor`) with `kong.request.get_header()` and returns a 302 with a `Location` header pointing at one downstream route if the header is missing, or another if it's present. Each downstream route can then have its own plugins (for example Rate Limiting Advanced) applied independently. +--- + +## Overview + +Is it possible to have Kong re-route a request based on Header existence to alter which plugins are triggered? Example: Route 1 - Has the Rate Limiting Advanced plugin. Should trigger if the header `x-vendor` does not exist. Route 2 - Has no plugin. Should trigger if the header `x-vendor` exists. + +## Steps + +This can be accomplished through out-of-the-box plugins with some custom code. + +To start off you will need 3 Routes created all pointing to their respective services (or the same service). For this example, we will use the same service for all 3 Routes. The main service will have a Request Termination plugin and an Exit Transformer plugin. + +Create Service: + +```bash + +curl --request POST \ + --url http://:8001/services/ \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: ' \ + --data '{ + "name": "test", + "host": "mockbin.org", + "path": "/", + "port": 80, + "protocol": "http"}' +``` + +Create Route (main): + +```bash + +curl --request POST \ + --url http://:8001/services/test/routes \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: ' \ + --data '{ + "name": "mainPath", + "paths": ["/main"], + "protocols": ["http", "https"]}' +``` + +Note: Route's `protocols` field defaults to `["https"]` only (not `["http", "https"]`) since Kong 3.x — explicitly setting `protocols` as shown above is required, or the final plain-`http://` curl tests below will fail with `426 {"message":"Please use HTTPS protocol"}` instead of working. + +Create Route (1): + +```bash + +curl --request POST \ + --url http://:8001/services/test/routes \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: ' \ + --data '{ + "name": "route1", + "paths": ["/route1"], + "protocols": ["http", "https"]}' +``` + +Create Route (2): + +```bash + +curl --request POST \ + --url http://:8001/services/test/routes \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: ' \ + --data '{ + "name": "route2", + "paths": ["/route2"], + "protocols": ["http", "https"]}' +``` + +Create Request Termination Plugin (Attached to Main Route): + +```bash + +curl --request POST \ + --url http://:8001/routes/mainPath/plugins/ \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: ' \ + --data '{ + "name": "request-termination", + "config": + { + "status_code": 403}}' +``` + +Create Exit Transformer Plugin (Attached to Main Route): + +```bash + +curl --request POST \ + --url http://:8001/routes/mainPath/plugins/ \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: ' \ + --data '{ + "name": "exit-transformer", + "config": + { + "functions":[ + "return function(status, body, headers)\n status = 302\n local path = kong.request.get_header(\"x-vendor\") \n if not path then\n headers = { [\"Location\"] = \"http://:8000/route1\"}\n body = { message = \"missing a header, x-vendor\" }\n else\n headers = { [\"Location\"] = \"http://:8000/route2\"}\n body = { message = \"redirect is needed\" }\n end\n return status, body, headers\nend\n" + ]}}' +``` + +Create Rate Limiting Advanced plugin (Attached to Route 1). Note that `sync_rate` cannot be set at all when `strategy` is (or defaults to) `"local"` — live-tested on {{site.base_gateway}} 3.14.0.0: including `"sync_rate":0` alongside `"strategy":"local"` fails with `400 schema violation ("sync_rate cannot be configured when using a local strategy")` regardless of whether `strategy` is set explicitly or left to its `"local"` default. `sync_rate` must be omitted entirely (as below) for the local strategy; only set it if `strategy` is `"redis"` or `"cluster"` instead: + +```bash + +curl --request POST \ + --url http://:8001/routes/route1/plugins \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: ' \ + --data '{ + "name": "rate-limiting-advanced", + "tags":["rate1"], + "config": + { + "limit":[2], + "window_size":[60], + "strategy":"local"}}' +``` + +--- + +Now we can run a quick test to confirm. When hitting the Main Route with and without the `x-vendor` header. The Exit Transformer responds with a 302 and a `Location` header, so the verification curl commands need `-L`/`--location` to actually follow the redirect onto Route 1 or Route 2 — without it, curl only sees the 302 from the main route and never reaches either route's plugins. + +With the header `x-vendor`, the request is redirected to Route 2, which has no rate limiting plugin, so it is not counted. + +```bash + +curl --request GET \ + --location \ + --url http://:8000/main \ + --header 'x-vendor: test' +``` + +Without the header `x-vendor`, the request is redirected to Route 1. Call this 3x and you will see the rate limiting advanced plugin be triggered. + +```bash + +curl --request GET \ + --location \ + --url http://:8000/main +``` diff --git a/app/_support/how-to-reference-a-json-or-yaml-config-file-in-a-deck-config-yaml.md b/app/_support/how-to-reference-a-json-or-yaml-config-file-in-a-deck-config-yaml.md new file mode 100644 index 0000000000..989ba72535 --- /dev/null +++ b/app/_support/how-to-reference-a-json-or-yaml-config-file-in-a-deck-config-yaml.md @@ -0,0 +1,92 @@ +--- +title: How to reference a JSON or YAML config file in a deck config YAML +content_type: support +description: As it stands, decK does not offer the ability to reference JSON or YAML files from the filesystem in its config. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: the decK feature of passing env variables in the config + url: /deck/reference/env-variables/ +tldr: + q: How do I reference an external JSON or YAML file in a decK config file? + a: | + decK does not support referencing external JSON or YAML files directly from its config. As a workaround, load the file's contents into an environment variable (for example `export VAR=$(cat file.json)`) and reference it in the plugin config with decK's environment variable substitution, `{% raw %}${{ env "VAR" }}{% endraw %}`. This workaround is limited by your operating system's environment variable size limit. +--- + +## Overview + +There are cases where it would be beneficial to be able to reference a piece of JSON or yaml via a parameter in a deck yaml file, rather than having to copy and paste it directly into the deck config, e.g. when defining the `api_spec` for the OAS validation plugin. Does decK support this requirement? + +## Steps + +As it stands, decK does not offer the ability to reference JSON or YAML files from the filesystem in its config. + +However, a workaround that can work with some limitations, is to use the decK feature of passing env variables in the config. + +Here's an example to show how to implement the solution. + +We can save the json api-spec in a file called `swagger-petstore.json` + +Then it can be loaded in an env variable like so: + +`export DECK_OAS_SPEC_PETSTORE=$(cat swagger-petstore.json)` + +The deck yaml would look like so: + +```yaml + +_format_version: "3.0" +services: +- connect_timeout: 60000 +enabled: true +host: petstore.swagger.io +name: Petstore-Service +path: /v2 +plugins: +- config: +allowed_header_parameters: Host,Content-Type,User-Agent,Accept,Content-Length +api_spec: {% raw %}${{ env "DECK_OAS_SPEC_PETSTORE" }}{% endraw %} +header_parameter_check: false +notify_only_request_validation_failure: false +notify_only_response_body_validation_failure: false +query_parameter_check: false +validate_request_body: true +validate_request_header_params: true +validate_request_query_params: true +validate_request_uri_params: true +validate_response_body: false +verbose_response: true +enabled: true +name: oas-validation +protocols: +- grpc +- grpcs +- http +- https +port: 443 +protocol: https +read_timeout: 60000 +retries: 5 +routes: +- https_redirect_status_code: 426 +name: Petstore-Route +path_handling: v0 +paths: +- /.* +preserve_host: false +protocols: +- http +- https +regex_priority: 0 +request_buffering: true +response_buffering: true +strip_path: true +write_timeout: 60000 +``` + +Please note that the API specification can be either a JSON or YAML based file. If using a YAML file, the spec needs to be URL encoded to preserve the YAML format. However the process of passing it via an env var would be the same as above. + +The limitation of this workaround depends on the size of the JSON or YAML file, since most operating systems would have a hard limit on the size of a environment variable. That means that this workaround will not work for files greater than the env var size limit. diff --git a/app/_support/how-to-reset-password-by-reset-password-link-without-smtp-server.md b/app/_support/how-to-reset-password-by-reset-password-link-without-smtp-server.md new file mode 100644 index 0000000000..b9d94e0b3f --- /dev/null +++ b/app/_support/how-to-reset-password-by-reset-password-link-without-smtp-server.md @@ -0,0 +1,27 @@ +--- +title: "How to reset password by 'Reset Password' link without SMTP server" +content_type: support +description: This article explains how to reset an admin or RBAC user's password using the Reset Password link when no SMTP server is configured. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I reset an admin or RBAC user's password using the Reset Password link when there's no SMTP server configured? + a: | + In Kong Manager, open the invited user and click `Generate registration link` to get a registration link without needing an email from an SMTP server. Copy the link and open it at `{Your kong manager URL: 8002}/{copied registration link in step 3}` to reach the reset password page. +--- + +## Overview + +Sometimes, you'd like to reset the password by clicking the `Reset Password` link. However, there is no SMTP server. This article teaches you how to reset the password by clicking the `Reset Password` link without SMTP server. + +## Steps + +1. Invite an admin user or RBAC user through Kong Manager. +2. Click the invited user. +3. Click `Generate registration link`. +4. Copy the registration link. +5. Access the reset password page at `{Your kong manager URL: 8002}/{copied registration link in step 3}`. diff --git a/app/_support/how-to-reset-rbac-token-for-an-admin-using-the-cli.md b/app/_support/how-to-reset-rbac-token-for-an-admin-using-the-cli.md new file mode 100644 index 0000000000..063e7895fa --- /dev/null +++ b/app/_support/how-to-reset-rbac-token-for-an-admin-using-the-cli.md @@ -0,0 +1,74 @@ +--- +title: How to reset RBAC token for an Admin using the CLI +content_type: support +description: The following script takes a username and password, converts it to a session, which is then used to log in to the `admin-api` and retrieve a new RBAC token for the authenticating admin. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I reset the RBAC token for a Kong Admin using the CLI? + a: | + Run a script that logs in with the admin's username and password to obtain a session cookie, then sends a `PATCH` request to `/admins/self/token` with that session to generate a new RBAC token. Update the `USERNAME`, `PASSWORD`, and `KONG_ADMIN` variables in the script before running it. +--- + +## Overview + +How to reset RBAC token for an Admin using the CLI. + +## Steps + +The following script takes a username and password, converts it to a session, which is then used to log in to the `admin-api` and retrieve a new RBAC token for the authenticating admin. + +Change the `USERNAME`, `PASSWORD`, and `KONG_ADMIN` variables as necessary. + +```bash + +#!/bin/bash + +USERNAME="kong_admin" +PASSWORD="admin" +KONG_ADMIN="https://localhost:8444" + +BASE64_ENCODED_AUTH=$(echo -n $USERNAME:$PASSWORD | base64 ) + +# printf "\n\n$BASE64_ENCODED_AUTH" + +printf "\nAuthorising admin and retrieving session id ..." +GET_SESSION_RESPONSE=$(curl -skv --location --request GET $KONG_ADMIN/auth \ + -H "Authorization: Basic $BASE64_ENCODED_AUTH" \ + -H "Kong-Admin-User: $USERNAME" 2>&1 ) + +SESSION_ID=$(echo "$GET_SESSION_RESPONSE" | grep "Set-Cookie" | cut -d ";" -f 1 | cut -d " " -f 3) + +if [[ $SESSION_ID != "" ]] +then + printf "SUCCESS" + printf "\nSession ID: $SESSION_ID" +else + printf "FAILED\n" + printf "$GET_SESSION_RESPONSE" + exit 1 +fi + +printf "\nGenerating RBAC Token for admin ..." +RBAC_TOKEN_RESPONSE=$(curl -sk --location --request PATCH $KONG_ADMIN/admins/self/token \ + -H 'Content-Type: application/json' \ + -H "Cookie: $SESSION_ID" \ + -H "Kong-Admin-User: $USERNAME" \ + -d '{}') + +NEW_RBAC_TOKEN=$(echo $RBAC_TOKEN_RESPONSE | jq .token -j ) + +if [[ $NEW_RBAC_TOKEN != "" ]] +then + printf "SUCCESS" + printf "\nNew admin RBAC token: $NEW_RBAC_TOKEN" +else + printf "FAILED\n" + printf "$RBAC_TOKEN_RESPONSE" + exit 1 +fi +``` diff --git a/app/_support/how-to-resolve-connection-reset-errors-when-configuring-kong-audit-logs-with-webhooks-and-splunk.md b/app/_support/how-to-resolve-connection-reset-errors-when-configuring-kong-audit-logs-with-webhooks-and-splunk.md new file mode 100644 index 0000000000..51fe04528a --- /dev/null +++ b/app/_support/how-to-resolve-connection-reset-errors-when-configuring-kong-audit-logs-with-webhooks-and-splunk.md @@ -0,0 +1,94 @@ +--- +title: Resolving Connection Reset Errors When Configuring Kong Audit Logs with Webhooks and Splunk +content_type: support +description: "Webhook requests to Splunk can fail with a `\"connection reset by peer\"` error; allowlisting {{site.konnect_product_name}}'s egress IP addresses on the Splunk side typically resolves it." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How to Resolve Connection Reset Errors When Configuring Kong Audit Logs with Webhooks and Splunk? + a: | + {{site.konnect_product_name}}'s egress IP addresses that send audit log webhook traffic may not be allowlisted on the Splunk side, causing Splunk to reset the connection. Retrieve the current egress IPs from `https://ip-addresses.origin.konghq.com/ip-addresses.json`, then allowlist the IPs for your Konnect region plus the US region (Authentication audit logs always originate from the US region) on the Splunk side. +related_resources: [] +--- + +## Problem + +When configuring {{site.konnect_product_name}}'s audit logs with webhooks and Splunk, the webhook request can fail after several attempts with a `"connection reset by peer"` error. + +## Cause + +This problem arises when the connection to Splunk is unexpectedly closed, potentially due to configuration or network issues. + +## Solution + +To address this, follow the steps outlined below: + +1. Verify IP Allowlist on Splunk Side: + + Ensure that all necessary IP addresses from which {{site.konnect_product_name}} sends audit log traffic are allowed on the Splunk side. You can retrieve the current list of {{site.konnect_product_name}} egress IP addresses by accessing the following URL: + + https://ip-addresses.origin.konghq.com/ip-addresses.json + + Please note that you need to enable the `egressIPs` for the region for which you are enabling Audit logging but also ALWAYS need to enable the US region egress IP addresses. + + The reason for having to enable the US egress IP addresses is that at time of creating this KB article, 23 October 2024, Authentication audit logs originate from the US region. + + All the `egressIPs` can be retrieved with the below curl command, filtered using `jq`, which on 23 October 2024 returns the list below: + + ```bash + curl https://ip-addresses.origin.konghq.com/ip-addresses.json|jq -r '.egressIPs' + { + "au": [ + "54.79.153.51", + "13.55.230.239", + "52.65.44.35", + "16.50.18.205", + "16.51.48.148", + "16.50.82.99" + ], + "eu": [ + "35.157.142.129", + "3.68.163.51", + "3.72.189.41", + "54.74.155.219", + "34.254.24.160", + "52.51.153.231" + ], + "us": [ + "18.217.207.159", + "52.15.154.8", + "3.137.43.24", + "52.26.195.109", + "52.13.59.51", + "54.201.160.33" + ] + } + ``` + + Note: the region previously labeled `ap` has been renamed to `au` (the egress IPs are unchanged). Two additional regions, `in` and `sg`, have also been added since this article was originally written — query the endpoint directly to get their current egress IPs rather than relying on a static list. + + If you want to enable Konnect audit log webhooks for the EU region, make sure to allow on Splunk the `eu` and `us` egress IPs which based on the sample output above would be: + + ``` + "35.157.142.129", + "3.68.163.51", + "3.72.189.41", + "54.74.155.219", + "34.254.24.160", + "52.51.153.231" + "18.217.207.159", + "52.15.154.8", + "3.137.43.24", + "52.26.195.109", + "52.13.59.51", + "54.201.160.33" + ``` + +2. Re-enable audit logging, and verify that this continues to work after allowing the list of egress IPs from the US and your Konnect region. + + After applying the above changes, monitor the connection to see if audit logs are successfully received by Splunk without the connection being reset. Test by disabling and then re-enabling the audit webhook in the Konnect UI to verify if the issue recurs or if it has been resolved. + +By following these steps, you should be able to resolve the `"connection reset by peer"` error and ensure a stable connection between {{site.konnect_product_name}} and Splunk for audit log transmission. diff --git a/app/_support/how-to-route-requests-regarding-incoming-request-port-number.md b/app/_support/how-to-route-requests-regarding-incoming-request-port-number.md new file mode 100644 index 0000000000..a74c5c02f7 --- /dev/null +++ b/app/_support/how-to-route-requests-regarding-incoming-request-port-number.md @@ -0,0 +1,110 @@ +--- +title: How to route requests regarding incoming request port number +content_type: support +description: Kong doesn't route requests based on the incoming port natively, so this article uses the `pre-function` plugin to proxy requests to different upstreams by port number. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "Admin API route object reference" + url: "/gateway/entities/route/" +tldr: + q: How do I proxy requests to different upstreams based on the incoming port number in Kong? + a: | + Kong doesn't route requests based on the incoming port by default. Configure multiple ports with `proxy_listen`, then use a `pre-function` plugin whose Lua script checks `kong.request.get_port()` and calls `kong.service.set_target()` to send each port's traffic to a different upstream. This requires setting `untrusted_lua` to `on` or `lax` (not the default `strict`), since `set_target()` is otherwise blocked. +--- + +## Overview + +I would like to proxy requests to different upstreams regarding different incoming ports like below + +``` + +e.g +http://:7000/test will be proxied to upstream1(https://:443/xxx) +http://:8000/test will be proxied to upstream2(https://:443/xxx) +http://:9000/test will be proxied to upstream3(https://:443/xxx) +``` + +How to realize it? + +## Steps + +As Kong does not support routing requests based on ports, we have to use the `pre-function` plugin to realize it. + +Please refer bellow procedure to send requests based on different ports + +(Please modify Lua script/service object/route object depend on your requirements) + +Prerequisite: The pre-function plugin's `kong.service.set_target()` call requires the `untrusted_lua` Kong configuration property to be set to `on` or `lax` (the default is now `strict`, which excludes this call). Set `KONG_UNTRUSTED_LUA=on` and restart Kong before proceeding, otherwise the plugin will fail with a 500 error: `attempt to call field 'set_target' (a nil value)`. + +1. Enable multiple proxy ports for Kong and restart Kong. + + Modify below parameter in kong configuration + + ``` + KONG_PROXY_LISTEN/proxy_listen=0.0.0.0:7000, 0.0.0.0:8000, 0.0.0.0:9000, 0.0.0.0:8443 ssl + ``` + + Restart Kong + + ```bash + kong restart + ``` + +2. Create below service object and route object + + ```yaml + service: + name: test + url: https://:443/xxx + route: + name: test + path: /test + ``` + +3. Write a Lua script like below, let's name it as `redirect.lua` + + ```lua + local port = kong.request.get_port() + if port == 8000 then + kong.service.set_target("", 443) + kong.service.request.set_scheme("https") + kong.service.request.set_path("/xxx") + elseif port == 9000 then + kong.service.set_target("", 443) + kong.service.request.set_scheme("https") + kong.service.request.set_path("/xxx") + end + ``` + +4. Enable the `pre-function` plugin on the route object we created in step 2 + + ```bash + curl -X POST http://:8001/routes/test/plugins \ + -F "name=pre-function" \ + -F "config.access[1]=@/path/to/redirect.lua" + ``` + +5. Testing + + ```bash + curl -i http://:7000/test + > response from upstream1 + + curl -i http://:8000/test + > response from upstream2 + + curl -i http://:9000/test + > response from upstream3 + ``` + +As we could see, + +request from 7000 port has been proxied to upstream1, + +request from 8000 port has been proxied to upstream2, + +request from 9000 port has been proxied to upstream3. diff --git a/app/_support/how-to-send-kong-log-to-splunk-by-using-http-log-plugin.md b/app/_support/how-to-send-kong-log-to-splunk-by-using-http-log-plugin.md new file mode 100644 index 0000000000..43fdfdcf01 --- /dev/null +++ b/app/_support/how-to-send-kong-log-to-splunk-by-using-http-log-plugin.md @@ -0,0 +1,62 @@ +--- +title: How to send Kong log to Splunk by using HTTP Log plugin +content_type: support +description: Configure the HTTP Log plugin to send Kong logs to a Splunk HTTP Event Collector (HEC) raw endpoint using a secure token. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: HTTP Log plugin + url: /plugins/http-log/ + - text: HTTP Log plugin log format + url: /plugins/http-log/#log-format + - text: Splunk HEC examples + url: https://docs.splunk.com/Documentation/Splunk/9.0.2/Data/HECExamples +tldr: + q: How do I send Kong logs to Splunk using the HTTP Log plugin? + a: | + Configure the HTTP Log plugin's `http_endpoint` to point at your Splunk HEC raw endpoint (for example `https://:8088/services/collector/raw`), and set the `Authorization` header to `Splunk `. On Kong versions before 3.0, the `Authorization` header value must be given as a list rather than a plain string. +--- + +## Overview + +How to send Kong log to Splunk by using the HTTP Log plugin? + +## Steps + +Please note the example below is for Splunk 9.0.2. Check the Splunk docs to use the appropriate method if you are using a different version of Splunk. + +The HTTP Log plugin will send the response body in the plugin log format. + +We could follow example 3 and send the raw data to the `/services/collector/raw` endpoint of Splunk. + +Also, the secure token is required by Splunk with this endpoint. + +Assuming our Splunk is running at `https://demo.splunkcloud.com:8088/` and its secure token is `123456`, then we could enable an HTTP Log plugin with the configuration below: + +```yaml +config: + headers: + Authorization: "Splunk 123456" + http_endpoint: https://demo.splunkcloud.com:8088/services/collector/raw + method: POST + timeout: 3000 + retry_count: 1 +``` + +If you are running Kong version less than 3.0, please enable an HTTP Log plugin with the configuration below instead: + +```yaml +config: + headers: + Authorization: + - "Splunk 123456" + http_endpoint: https://demo.splunkcloud.com:8088/services/collector/raw + method: POST + timeout: 3000 + retry_count: 1 +``` + +This HTTP Log plugin will send the log to `https://demo.splunkcloud.com:8088/services/collector/raw` with the secure token, as in example 3 of the Splunk HEC examples. diff --git a/app/_support/how-to-send-metrics-and-logs-to-datadog-in-a-kubernetes-environment.md b/app/_support/how-to-send-metrics-and-logs-to-datadog-in-a-kubernetes-environment.md new file mode 100644 index 0000000000..676b3f9b01 --- /dev/null +++ b/app/_support/how-to-send-metrics-and-logs-to-datadog-in-a-kubernetes-environment.md @@ -0,0 +1,94 @@ +--- +title: How to send metrics and logs to Datadog in a Kubernetes environment +content_type: support +description: Steps to send Kong metrics and logs to Datadog in a Kubernetes environment, using the Datadog Agent, the `prometheus` plugin, and Autodiscovery pod annotations. +products: + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: "Datadog Kong integration documentation" + url: "https://docs.datadoghq.com/integrations/kong/?tab=containerized" + - text: "Datadog site documentation" + url: "https://docs.datadoghq.com/getting_started/site/" + - text: "Datadog API and application keys documentation" + url: "https://docs.datadoghq.com/account_management/api-app-keys/" + - text: "Datadog Helm chart repository" + url: "https://github.com/DataDog/helm-charts" + - text: "Prometheus plugin documentation" + url: "/plugins/prometheus/" + - text: "Kong Helm chart podAnnotations reference" + url: "https://github.com/Kong/charts/blob/main/charts/kong/values.yaml#L1040-L1042" + - text: "Kong Helm chart status endpoint reference" + url: "https://github.com/Kong/charts/blob/main/charts/kong/values.yaml#L235-L241" +tldr: + q: How do I send Kong metrics and logs to Datadog in a Kubernetes environment? + a: | + Create a Datadog account and API key, then install the Datadog Agent via Helm. Enable the `prometheus` plugin in Kong so Datadog can scrape metrics via OpenMetrics, and add `podAnnotations` to the Kong Helm chart so Datadog's Autodiscovery can collect metrics and logs from Kong pods. +--- + +## Overview + +How to send metrics and logs to Datadog in a Kubernetes environment? + +## Steps + +The procedure is described in the Datadog documentation. + +1. Create an account in Datadog in a Datadog site, notice that each Datadog site is independent. + + ```bash + export DD_SITE=datadoghq.eu + ``` + +2. Create an API key on your Datadog account. + + ```bash + export DD_API_KEY=123456789123456789123456789 + ``` + +3. Install the Datadog Agent via Helm chart. + + ```bash + helm install datadog \ + --set datadog.site=$DD_SITE \ + --set datadog.apiKey=$DD_API_KEY \ + --set datadog.logs.enabled=true \ + --set datadog.logs.containerCollectAll=true \ + --set datadog.kubelet.tlsVerify=false \ + datadog/datadog + ``` + + (`datadog.kubelet.tlsVerfiy = false` is required when running tests in minikube.) + +4. Datadog uses OpenMetrics to scrape metrics, we need to run the `prometheus` plugin in Kong. + + ```yaml + apiVersion: configuration.konghq.com/v1 + kind: KongClusterPlugin + metadata: + name: plugin-prometheus + annotations: + kubernetes.io/ingress.class: kong + labels: + global: "true" + plugin: prometheus + ``` + + ```bash + kubectl apply -f prometheus-plugin.yaml + ``` + +5. Add required `podAnnotations` in the Kong Helm chart to allow the Datadog agent to use Autodiscovery to collect metrics and logs from Kong pods: + + ```yaml + podAnnotations: + ad.datadoghq.com/proxy.check_names: '["kong"]' + ad.datadoghq.com/proxy.init_configs: '[{}]' + ad.datadoghq.com/proxy.instances: '[{"openmetrics_endpoint": "http://%%host%%:8100/metrics"}]' + ad.datadoghq.com/proxy.logs: '[{"source": "kong", "service": "kong-proxy"}]' + ad.datadoghq.com/ingress-controller.logs: '[{"source": "kong", "service": "kong-ingress-controller"}]' + ``` + +6. Make sure that the status endpoint is enabled on your Helm chart. diff --git a/app/_support/how-to-set-different-route-host-values-depending-on-the-environment-using-deck.md b/app/_support/how-to-set-different-route-host-values-depending-on-the-environment-using-deck.md new file mode 100644 index 0000000000..91c8e5bff2 --- /dev/null +++ b/app/_support/how-to-set-different-route-host-values-depending-on-the-environment-using-deck.md @@ -0,0 +1,40 @@ +--- +title: "How to set different route \"host\" values depending on the environment using deck" +content_type: support +description: As of version 1.7.0, deck makes it possible to use environment variables to set values inside the deck yaml file. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +published: false +tldr: + q: How do I set different route host values depending on the environment when using decK? + a: | + Since decK 1.7.0, you can reference an environment variable inside the decK YAML file, for example `hosts: [{% raw %}${{ env "DECK_HOSTS" }}{% endraw %}]`, and set that variable to a different value per environment before running `deck sync`. +--- + +## Overview + +When using deck, is it possible to set different route `hosts` values depending on the environment against which deck is run? + +## Steps + +As of version 1.7.0, deck makes it possible to use environment variables to set values inside the deck yaml file. + +For the route `hosts` value, an example would be: + +```yaml + + hosts: [{% raw %}${{ env "DECK_HOSTS" }}{% endraw %}] +``` + +and set the relevant env variable before executing a `deck sync` like this: + +```bash + +export DECK_HOSTS="'my.host1.com', 'my.host2.com'" +``` + +This would be valid for any environment variable to be used as an array. diff --git a/app/_support/how-to-set-healthcheck-in-alb-for-kong-control-plane-when-it-enabled-with-rbac.md b/app/_support/how-to-set-healthcheck-in-alb-for-kong-control-plane-when-it-enabled-with-rbac.md new file mode 100644 index 0000000000..38dc99218c --- /dev/null +++ b/app/_support/how-to-set-healthcheck-in-alb-for-kong-control-plane-when-it-enabled-with-rbac.md @@ -0,0 +1,51 @@ +--- +title: How to set healthcheck in ALB for Kong control plane when it's enabled with RBAC +content_type: support +description: After enabling RBAC for the Kong control plane (CP), its admin port (`8001`/`8444`) will always return `401` if no token has been sent. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: AWS Application Load Balancer introduction + url: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html + - text: Kong hybrid mode deployment topology + url: /gateway/hybrid-mode/ + - text: the `status_listen` configuration reference + url: /gateway/configuration/#status-listen + - text: AWS Load Balancer Controller ingress annotations reference + url: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/ +tldr: + q: How do I configure an ALB health check for the Kong control plane when RBAC is enabled? + a: | + Enable Kong's status port (`KONG_STATUS_LISTEN`, e.g. `0.0.0.0:8100`) since the RBAC-protected admin port always returns `401` without a token. Point the AWS Load Balancer Controller's health check annotations at that status port and the `/status` path instead of the admin port. +--- + +## Overview + +I installed Kong in k8s by using hybrid mode. And I enabled RBAC for the Kong control plane, I have ALB in front of the Kong control plane. How to set healthcheck for the Kong control plane in ALB? + +## Steps + +After enabling RBAC for the Kong control plane (CP), its admin port (`8001`/`8444`) will always return `401` if no token has been sent. + +We have to follow below steps to use status port for healthcheck in ALB. + +1. Enable status port + + Set `KONG_STATUS_LISTEN` env var for CP like below: + + ``` + KONG_STATUS_LISTEN = 0.0.0.0:8100 + ``` + + You could refer to the `status_listen` configuration reference to use a different port and protocol; replace them in step 2 if you use a different port and protocol. + +2. Add below AWS Load Balancer Controller annotations in the ingress for CP: + + ``` + alb.ingress.kubernetes.io/healthcheck-protocol: HTTP + alb.ingress.kubernetes.io/healthcheck-port: 8100 + alb.ingress.kubernetes.io/healthcheck-path: /status + ``` diff --git a/app/_support/how-to-set-ulimiton-kong-server.md b/app/_support/how-to-set-ulimiton-kong-server.md new file mode 100644 index 0000000000..4047237619 --- /dev/null +++ b/app/_support/how-to-set-ulimiton-kong-server.md @@ -0,0 +1,41 @@ +--- +title: How to set `ulimit` on Kong server +content_type: support +description: "`ulimit` is the number of open file descriptors per process." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +published: false +tldr: + q: How do I set the `ulimit` (open file descriptor limit) for the Kong server? + a: | + Kong logs a warning when the open file descriptor limit is too low. Set `ulimit -n` to at least `4096` on the server before starting Kong, ideally during server/node preparation, and restart Kong after changing it. +--- + +## Overview + +You will see the below warning message if the `ulimit` is not enough: + +``` +[warn] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n" +``` + +## Steps + +`Ulimit` is the number of open file descriptors per process. + +So you can use the below Linux command to set `ulimit` on your server. + +The number should be at least 4096. + +```bash + +ulimit -n {number} +``` + +After resetting `ulimit` on the server, please restart Kong. + +The best time to set `ulimit` is in the server/node preparation/installing phase before installing Kong on that server. diff --git a/app/_support/how-to-set-up-kong-to-autostart-kong-with-systemd-with-a-non-root-user.md b/app/_support/how-to-set-up-kong-to-autostart-kong-with-systemd-with-a-non-root-user.md new file mode 100644 index 0000000000..6714599956 --- /dev/null +++ b/app/_support/how-to-set-up-kong-to-autostart-kong-with-systemd-with-a-non-root-user.md @@ -0,0 +1,60 @@ +--- +title: How to set up kong to autostart kong with systemd with a non root user +content_type: support +description: How to configure systemd to run {{site.base_gateway}} as a non-root `kong` user on RHEL and CentOS 8, instead of root. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I set up {{site.base_gateway}} to autostart with systemd as a non-root user? + a: | + Copy Kong's `kong-enterprise-edition.service` unit file to `/home/kong/.config/systemd/user/`, remove the hardcoded `User=` and `Group=` lines, and change `WantedBy=multi-user.target` to `WantedBy=default.target`. Enable lingering for the `kong` user with `loginctl enable-linger $USER` and set `XDG_RUNTIME_DIR` so `systemctl --user` commands can start and enable the service at boot. +--- + +## Overview + +The instructions for controlling {{site.base_gateway}} through systemd require root to be used to start kong as a root user. We would like to use a non-root user, e.g. a "kong" user. + +## Steps + +In RHEL and CentOS 8 you can configure systemd to control the {{site.base_gateway}} as a "kong" user by following these steps: + +1. Log into the VM where you want to configure the Gateway to start as the "kong" user. +2. Create a folder called `/home/kong/.config/systemd/user/`. +3. Copy `/lib/systemd/system/kong-enterprise-edition.service` (this is the file's actual shipped location in the current {{site.ee_product_name}} package/image - it is not found under `/etc/kong/`) to `/home/kong/.config/systemd/user/`. +4. Edit `/home/kong/.config/systemd/user/kong-enterprise-edition.service` and remove the `User=` and `Group=` lines - Kong's shipped unit file hardcodes `User=root`, and a copied user-level unit will fail to start unless those lines are stripped out. +5. In the same file, replace: + + ``` + [Install] + WantedBy=multi-user.target + ``` + + with + + ``` + [Install] + WantedBy=default.target + ``` + +6. Run the following to make sure the {{site.base_gateway}} process can continue to run when the kong user has logged off: + + ```bash + loginctl enable-linger $USER + ``` + +7. Add the following to `.bash_profile`, and source `~/.bash_profile` afterwards: + + ```bash + export XDG_RUNTIME_DIR=/run/user/$(id -u) + ``` + +8. You can now start kong using systemctl, and enable autostart at system boot with these two commands: + + ```bash + systemctl --user start kong-enterprise-edition + systemctl --user enable kong-enterprise-edition + ``` diff --git a/app/_support/how-to-setup-a-pki-hybrid-instance-in-kubernetes-with-certificate-manager.md b/app/_support/how-to-setup-a-pki-hybrid-instance-in-kubernetes-with-certificate-manager.md new file mode 100644 index 0000000000..e9ab3eef11 --- /dev/null +++ b/app/_support/how-to-setup-a-pki-hybrid-instance-in-kubernetes-with-certificate-manager.md @@ -0,0 +1,420 @@ +--- +title: How to setup a PKI Hybrid instance in Kubernetes with certificate manager +content_type: support +description: How to configure {{site.base_gateway}} Hybrid mode to use PKI mutual TLS with certificates issued by cert-manager and a private CA in Kubernetes. +products: + - gateway + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: "cert-manager Kubernetes installation documentation" + url: "https://cert-manager.io/docs/installation/kubectl/" +tldr: + q: How do I configure {{site.base_gateway}} Hybrid mode to use PKI certificates issued by a private CA with cert-manager in Kubernetes? + a: | + Use `cert-manager` to build a certificate chain from a private root CA through two intermediate CAs, then issue separate control plane (`server auth`) and data plane (`client auth`) certificates from the final intermediate issuer. Mount the resulting secrets into the Kong Helm chart values (`cluster_ca_cert`, `cluster_cert`, `cluster_cert_key`), set `cluster_mtls: pki` on both nodes, and set `lua_ssl_verify_depth` high enough to cover the full chain. +--- + +## Overview + +There is a requirement to use cert-manager and a Private CA when setting up Kong in Hybrid mode with PKI certificates. As the certificates are not from a trusted CA, what configuration is needed to allow Kong to use the Private CA certificates? + +## Steps + +### Introduction + +The steps below provide a walkthrough of the necessary configuration. This article assumes that you will be using a Private Root CA and 2 intermediate certificates, and that you have access to the Root CA private key and public certificate. For the purposes of this article, we will be assuming the below certificates are available; + +`rootCA.key` - The Root CA private key + +`rootCA.pem` - The Root CA Public certificate + +It may be that in your environment, you do not have access to this certificate pair and maybe have a `ClusterIssuer` in cert-manager that you can use. You will need to adjust the instructions as appropriate for your environment. + +### Install cert-manager + +Follow the Kubernetes documentation to install and test cert-manager. + +### Create the certificate chain + +#### Create a secret for the RootCA + +Create a `tls` secret that contains the Private Key and Public certificate for the Root CA + +```bash + +kubectl create secret tls rootca-key-pair --key="rootCA-noenc.key" --cert="rootCA.pem" -n kong +``` + +#### Create Root CA Issuer + +```bash + +cat < Intermediate1CA --> Intermediate2CA. This needs to be created as a `generic` secret + +```bash + +kubectl get secret rootca-key-pair -n kong -o jsonpath='{.data.tls\.crt}'|base64 -d > chain.pem +kubectl get secret intermediate-ca1-key-pair -n kong -o jsonpath='{.data.tls\.crt}'|base64 -d >> chain.pem +kubectl get secret intermediate-ca2-key-pair -n kong -o jsonpath='{.data.tls\.crt}'|base64 -d >> chain.pem +kubectl -n kong create secret generic cluster-fullchain --from-file=./chain.pem +``` + +### Create the Kong Cluster certificates + +For PKI mode, the Control plane and Data plane need their own certificates that are signed by the same CA. + +#### Create the Control Plane certificate pair + +Create the cert-manager configuration which creates the control plane certificate. Note, the Control plane needs the "Web Server Authentication" usage; + +```bash + +cat < kong-validations.yaml`, then delete it with `kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io kong-kong-validations` to stop Kubernetes from calling the webhook. Restore it later with `kubectl apply -f kong-validations.yaml`. This only fully disables validation if it's the only matching webhook configuration in the cluster. +--- + +## Overview + +While troubleshooting we might be interested to temporarily disable Kubernetes Ingress Controller Admission Controller. Or in some situations where we get errors connecting the validation service: + +```bash + +kubectl apply -f repro.yaml +Error from server (InternalError): error when creating "repro.yaml": Internal error occurred: failed calling webhook "validations.kong.konghq.com": Post "https://kong-kong-validation-webhook.kong.svc:443/?timeout=10s": service "kong-kong-validation-webhook" not found +``` + +## Steps + +To temporarily disable the Admission Controller, for testing and troubleshooting, you can run these commands in your K8s cluster to: + +1. Get the current `kong-kong-validations` configuration and save it to a file, `kong-validations.yaml`: + + ```bash + kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io kong-kong-validations -o yaml > kong-validations.yaml + ``` + +2. Delete the `kong-kong-validations` webhook configuration: + + ```bash + kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io kong-kong-validations + ``` + +To enable the Admission Controller again you can recover from the file created in previous step: + +```bash + +kubectl apply -f kong-validations.yaml +``` + +Note: this only fully disables admission control if the deleted `ValidatingWebhookConfiguration` is the only Kong-related one in the cluster. Kubernetes invokes every `ValidatingWebhookConfiguration` that matches a given resource, not just the one belonging to a particular KIC release. If there is more than one KIC installation in the cluster, each with its own webhook configuration, you must delete all of the matching webhook configurations to actually let an otherwise-invalid config through - deleting just one release's configuration will not do it, since the others will still intercept the request. + +More information about the Admission Controller can be found in the Admission Controller documentation. diff --git a/app/_support/how-to-update-kong-configuration-that-is-hard-coded-in-usr-local-share-lua-5-1-kong-templates-nginx-kong-lua.md b/app/_support/how-to-update-kong-configuration-that-is-hard-coded-in-usr-local-share-lua-5-1-kong-templates-nginx-kong-lua.md new file mode 100644 index 0000000000..656ca0e3fe --- /dev/null +++ b/app/_support/how-to-update-kong-configuration-that-is-hard-coded-in-usr-local-share-lua-5-1-kong-templates-nginx-kong-lua.md @@ -0,0 +1,57 @@ +--- +title: How to update Kong configuration that is hard coded in `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua` when using Helm charts +content_type: support +description: Use the Helm chart's `userDefinedVolumes` and `userDefinedVolumeMounts` to mount a custom `nginx_kong.lua` template, overriding hard-coded Nginx configuration options that Kong's default template doesn't expose. +products: + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I override Nginx configuration hard-coded in `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua` when using the Kong Helm chart? + a: | + Mount a custom `nginx_kong.lua` file using the Helm chart's `userDefinedVolumes` and + `userDefinedVolumeMounts` settings under `deployment` in `values.yaml`, backed by a `configMap` + created from your edited template. Re-check the template against Kong's default after any + upgrade, since it can change between versions. +--- + +## Overview + +There is one or more nginx configuration options which are hard coded in the `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua` file that propagate to `nginx-kong.conf` in the kong `` directory, and determine the nginx configuration, and we need to override those values. Using a custom nginx template would be a way to address this, but with the Kong Helm charts this does not seem to be straight forward. What is a good way to override the `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua` file when using Kong Helm charts? + +## Steps + +As documented with the Kong Helm charts you can add configMap to a custom volume with a custom mount point using the `userDefinedVolumes`, and `userDefinedVolumeMounts` elements under the deployment property of your `values.yaml` file. + +To replace `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua` you would need to: + +1. Create a configMap using something like the following command: + + ```bash + kubectl -n create configmap nginx-kong-lua --from-file=nginx_kong.lua + ``` + +2. Change the `values.yaml` to use that configMap as in this configuration: + + ```yaml + deployment: + kong: + enabled: true + daemonset: false + userDefinedVolumes: + - name: "nginx-kong-lua" + configMap: + name: nginx-kong-lua + userDefinedVolumeMounts: + - name: "nginx-kong-lua" + mountPath: "/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua" + subPath: "nginx_kong.lua" + + image: + repository: kong/kong-gateway + ... + ``` + +WARNING: Before upgrading Kong (even to a new minor version), you MUST make sure that the `/usr/local/share/lua/5.1/kong/templates/nginx_kong.lua` has not changed or if it has, use the new version with the required changes on your side. diff --git a/app/_support/how-to-upload-pfx-format-certificate-to-kong.md b/app/_support/how-to-upload-pfx-format-certificate-to-kong.md new file mode 100644 index 0000000000..66ec7cae57 --- /dev/null +++ b/app/_support/how-to-upload-pfx-format-certificate-to-kong.md @@ -0,0 +1,79 @@ +--- +title: How to upload a PFX-format certificate to Kong +content_type: support +description: Convert a PFX certificate to separate PEM certificate and key files, then upload them to Kong via Kong Manager or the Admin API. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: Certificate entity - set up a certificate + url: /gateway/entities/certificate/#set-up-a-certificate +tldr: + q: How do I upload a PFX-format certificate to Kong? + a: | + Kong's Certificate entity only accepts PEM-format certificates and keys, so convert the `.pfx` file with `openssl pkcs12` into separate `key.pem` and `cert.pem` files first (stripping any `Bag Attributes` / `Key Attributes` headers). Then upload both files to Kong as a Certificate entity, either through Kong Manager or the Admin API `/certificates` endpoint. +--- + +## Overview + +How to upload a `pfx` format certificate to kong? + +## Steps + +1. Convert a PFX file to separate certificate and private key PEM files: + + ```bash + openssl pkcs12 -in -nocerts -out key.pem -nodes + openssl pkcs12 -in -nokeys -out cert.pem + ``` + +2. Remove `Bag Attributes`, `Key Attributes` and any other attributes from `key.pem` and `cert.pem`. + + Make sure `key.pem` is in the format below: + + ``` + -----BEGIN PRIVATE KEY----- + xxxxxxxxxxxxxxxxxxxx + -----END PRIVATE KEY----- + ``` + + Make sure `cert.pem` is in the format below: + + ``` + -----BEGIN CERTIFICATE----- + yyyyyyyyyyyyyyy + -----END CERTIFICATE----- + ``` + + If you are using intermediate certificates, `cert.pem` should be in the format below instead: + + ``` + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + ... + ``` + +3. Upload `key.pem` and `cert.pem` to Kong via Kong Manager or the Admin API: + + 1. Upload using Kong Manager: + + Access `http://:8002//certificates/create`, + copy the content from `cert.pem` to the "Cert" input box, + copy the content from `key.pem` to the "Key" input box, + then click the "Create" button. + + 2. Upload using the Admin API below: + + ```bash + curl -X POST http://:8001//certificates \ + -F cert=@/path/to/cert.pem \ + -F key=@/path/to/key.pem + ``` + +Check more details in the Certificate entity - set up a certificate documentation. diff --git a/app/_support/how-to-use-custom-environment-variable-in-custom-plugin-or-serverless-functions-plugin-when-installed-kong-via-helm-chart.md b/app/_support/how-to-use-custom-environment-variable-in-custom-plugin-or-serverless-functions-plugin-when-installed-kong-via-helm-chart.md new file mode 100644 index 0000000000..c61867b55a --- /dev/null +++ b/app/_support/how-to-use-custom-environment-variable-in-custom-plugin-or-serverless-functions-plugin-when-installed-kong-via-helm-chart.md @@ -0,0 +1,86 @@ +--- +title: How to use a custom environment variable in a custom plugin or serverless-functions plugin when Kong is installed via a Helm chart +content_type: support +description: "Use the `kong.vault.get()` method to load a custom environment variable set via a Helm chart's `customEnv` values into a custom or serverless-functions plugin." +products: + - gateway + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: "`kong.vault.get()` reference" + url: "/gateway/pdk/reference/kong.vault/#kong-vault-get-reference" +tldr: + q: How do I use a custom environment variable in a custom or serverless-functions plugin when Kong is installed via a Helm chart? + a: | + Use `kong.vault.get()` to read a custom environment variable. Define the variable under `customEnv` in the Helm chart's `values.yaml`, then reference it in the plugin config with `kong.vault.get("{vault://env/}")` — for example in a pre-function plugin's `config.access`. +--- + +## Overview + +We installed Kong via a Helm chart and want to use a custom environment variable in a custom plugin or serverless-functions plugin. How could we implement that? + +## Steps + +It is possible to use the `kong.vault.get()` method to load custom environment variables. + +Please follow the steps below as a reference: + +1. Set the `customEnv` section in `values.yaml` and deploy Kong via the Helm chart: + + ```yaml + # please set it in the root level of values.yaml + customEnv: + test_env: abc + ``` + +2. Create the following service/route objects in Kong for testing purposes: + + ```yaml + service: + name: test + url: https://httpbin.org/anything + route: + name: test + path: /test + ``` + +3. Enable a global pre-function plugin with the following configuration: + + ```yaml + config.access: kong.response.add_header("test-env", kong.vault.get("{vault://env/test_env}")) + ``` + + `kong.vault` is only accessible from within a pre-function/post-function/serverless-functions plugin + when `untrusted_lua` is set to `lax` (or the legacy `sandbox` mode) — the current default, `strict`, + disables it. Set `untrusted_lua: lax` alongside `customEnv` in the Helm chart's `values.yaml` (or + `KONG_UNTRUSTED_LUA=lax` if setting the value via an environment variable directly). + +4. Send a request to the service/route objects created in step 2. Here we assume Kong is running at `localhost:8000`: + + ```bash + curl http://localhost:8000/test -i + ``` + + Response: + + ``` + HTTP/1.1 200 OK + Content-Type: application/json + Content-Length: 485 + Connection: keep-alive + test-env: abc + Server: gunicorn/19.9.0 + Date: Thu, 06 Aug 2026 15:09:40 GMT + Access-Control-Allow-Origin: * + Access-Control-Allow-Credentials: true + X-Kong-Upstream-Latency: 16 + X-Kong-Proxy-Latency: 16 + Via: 1.1 kong/3.14.0.0-enterprise-edition + X-Kong-Request-Id: af1d727ea31ca819464b9f3876fb677c + + ... + ``` + +Here we could see the `test-env` response header has the value of `abc`, showing the pre-function plugin successfully loaded the custom environment variable defined in step 1. diff --git a/app/_support/how-to-use-external-lua-scripts-with-the-pre-function-plugin-in-declarative-config.md b/app/_support/how-to-use-external-lua-scripts-with-the-pre-function-plugin-in-declarative-config.md new file mode 100644 index 0000000000..be5780dc87 --- /dev/null +++ b/app/_support/how-to-use-external-lua-scripts-with-the-pre-function-plugin-in-declarative-config.md @@ -0,0 +1,50 @@ +--- +title: "How to Use External Lua Scripts with the `pre-function` Plugin in declarative config" +content_type: support +description: While Kong DecK does not support referencing external files directly, you can achieve your goal by using environment variables to include the content of your Lua script. +tldr: + q: How do I reference an external Lua script file in the `config.access` field of the `pre-function` plugin with declarative config and decK? + a: | + decK cannot reference external files directly. Load the script into an environment variable (`export DECK_FUNCTION=$(cat function.lua)`) and inject it in the declarative config with `{% raw %}${{ env "DECK_FUNCTION" | indent 8 }}{% endraw %}`. + Multiline indentation support requires decK v1.22.0 or later. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: using environment variables with Kong Deck + url: /deck/reference/env-variables/ + - text: multiline environment variables in Kong Deck pull request + url: https://github.com/Kong/deck/pull/929 +--- + +## Overview + +How can I reference an external Lua script file, such as `function.lua`, in the `config.access` field of the `pre-function` plugin when using Kong with declarative configuration files and Deck CLI? + +## Steps + +While Kong DecK does not support referencing external files directly, you can achieve your goal by using environment variables to include the content of your Lua script. Here's how you can do it: + +1. Create your Lua script and save it in a file, for example, `function.lua`. + +2. Load the content of the Lua script into an environment variable using the following command: + + ```bash + export DECK_FUNCTION=$(cat function.lua) + ``` + +3. Reference the content of the environment variable in your declarative configuration file. Since Kong Deck v1.22.0, you can use indentation to include multiline variable content. Here's an example of how to do this in your configuration: + + ```yaml + plugins: + - config: + access: + - | + {% raw %}${{ env "DECK_FUNCTION" | indent 8 }}{% endraw %} + enabled: true + name: pre-function + ``` + +This approach allows you to maintain the readability and maintainability of your configurations, especially when dealing with complex Lua scripts. diff --git a/app/_support/how-to-use-redis-tls.md b/app/_support/how-to-use-redis-tls.md new file mode 100644 index 0000000000..2562756455 --- /dev/null +++ b/app/_support/how-to-use-redis-tls.md @@ -0,0 +1,235 @@ +--- +title: How to use TLS-enabled Redis as a strategy for the `rate-limiting-advanced` plugin +content_type: support +description: "Set up a TLS-enabled Redis instance and configure it as the strategy for the `rate-limiting-advanced` plugin, including sharing one Redis instance across multiple Kong nodes." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "`lua_ssl_trusted_certificate` reference" + url: "/gateway/configuration/#lua-ssl-trusted-certificate" +tldr: + q: How do I configure a TLS-enabled Redis instance as the strategy for the `rate-limiting-advanced` plugin? + a: | + Build Redis with `BUILD_TLS=yes`, generate a CA and Redis certificate/key pair, and enable `tls-port` in `redis.conf`. Configure the `rate-limiting-advanced` plugin with `config.strategy=redis`, `config.redis.ssl=true`, and `config.redis.ssl_verify=true`, then add the Redis CA certificate to `lua_ssl_trusted_certificate` (or the `KONG_LUA_SSL_TRUSTED_CERTIFICATE` environment variable) and restart Kong. Multiple Kong nodes can safely share the same Redis instance for this plugin. +--- + +## Overview + +1. How to enable TLS for Redis +2. How to use TLS-enabled Redis as a strategy for the `rate-limiting-advanced` plugin +3. Can multiple Kong nodes share the same Redis instance? + +## 1. How to enable TLS for Redis + +Below is an example of installing redis-6.0.5 on CentOS 7. + +### Step 1: Install dependencies + +```bash +#Change to root user +sudo su +#Update CentOS package repository +yum update -y +#Install packages that are needed to compile and install Redis from source +yum install wget -y +yum install tcl -y +yum install gcc -y +yum install centos-release-scl -y +yum install devtoolset-9-gcc devtooset-g-gcc-c++ devtoolset-9-binutils -y +yum install openssl-devel* -y +#Update GCC +scl enable devtoolset-9 bash +echo "source /opt/rh/devtoolset-9/enable" >> /etc/profile +``` + +### Step 2: Install Redis by using make + +```bash +#Create redis user +useradd --system redis +#create directories used by Redis +#Set necessary permissions +mkdir /var/lib/redis +chown redis:redis /var/lib/redis +mkdir /var/log/redis +touch /var/log/redis/redis.log +chmod 660 /var/log/redis +chmod 640 /var/log/redis/redis.log +mkdir /etc/redis +chown -R redis:redis /etc/redis + +#Download redis +cd /tmp/ +mkdir redis +cd redis/ +wget http://download.redis.io/releases/redis-6.0.5.tar.gz +tar -xzvf redis-6.0.5.tar.gz +cd redis-6.0.5 +#Install by make, set BUILD_TLS as yes +make BUILD_TLS=yes install +``` + +### Step 3: Start Redis without TLS + +```bash +#Move redis.conf to the /etc/redis/ +cp redis.conf /etc/redis +chown redis:redis /etc/redis/redis.conf +chmod 640 /etc/redis/redis.conf + +#Modify redis to listen from any IP address +vi /etc/redis/redis.conf +-bind 127.0.0.1 ++bind 0.0.0.0 + +#Start redis without TLS +/usr/local/bin/redis-server /etc/redis/redis.conf & +#Now we can access redis without TLS +/usr/local/bin/redis-cli -h localhost -p 6379 +localhost:6379> ping +PONG +``` + +### Step 4: Create the TLS certificate and key + +```bash +#Move to a new directory for the operation +mkdir /tmp/certs && cd /tmp/certs + +#Generating a key +openssl genrsa -out ca.key 4096 +#Generating a certificate +openssl req -x509 -new -nodes -sha256 -key ca.key -days 365 -subj '/O=Redislabs/CN=Redis Prod CA' -out ca.crt + +#Generating the redis private key +openssl genrsa -out redis.key 2048 +mkdir /etc/ssl/private +#Generating the redis certificate +openssl req -new -sha256 -nodes -key redis.key -subj '/O=Redislabs/CN=Production Redis' | openssl x509 -req -sha256 -CA ca.crt -CAkey ca.key -CAserial /etc/ssl/private/ca.txt -CAcreateserial -days 365 -out redis.crt + +#Move the keys/certs to the appropriate locations +mkdir /usr/local/share/ca-certificates +cp ca.crt /usr/local/share/ca-certificates/ +cp ca.key /etc/ssl/private/ +cp redis.key /etc/ssl/private/ +cp redis.crt /etc/ssl/ +#set necessary permissions +chown redis:redis /usr/local/share/ca-certificates/ca.crt +chmod 644 /usr/local/share/ca-certificates/ca.crt +chown redis:redis /etc/ssl/private/ca.key +chmod 400 /etc/ssl/private/ca.key +chown redis:redis /etc/ssl/private/redis.key +chmod 400 /etc/ssl/private/redis.key +chown redis:redis /etc/ssl/redis.crt +chmod 644 /etc/ssl/redis.crt +``` + +### Step 5: Start Redis with TLS + +```bash +#Modify redis.conf to enable TLS +vi /etc/redis/redis.conf ++port 0 ++tls-port 6379 ++tls-cert-file /etc/ssl/redis.crt ++tls-key-file /etc/ssl/private/redis.key ++tls-ca-cert-file /usr/local/share/ca-certificates/ca.crt ++tls-auth-clients no ++tls-protocols "TLSv1.2" ++tls-ciphersuites TLS_CHACHA20_POLY1305_SHA256 ++tls-prefer-server-ciphers no + +/usr/local/bin/redis-server /etc/redis/redis.conf & + +#Now we can not access redis without TLS +/usr/local/bin/redis-cli -h localhost -p 6379 +localhost:6379> ping +Error +#Now we can access redis with TLS +/usr/local/bin/redis-cli -h localhost -p 6379 --tls --cacert /usr/local/share/ca-certificates/ca.crt +localhost:6379> ping +PONG +``` + +## 2. How to use TLS-enabled Redis as a strategy for the `rate-limiting-advanced` plugin + +### Step 1: Add the `rate-limiting-advanced` plugin with the following configuration + +(This is an example of adding the `rate-limiting-advanced` plugin to a route that only allows 5 requests in 5 minutes.) + +```bash +curl -X POST http://:8001/routes//plugins \ + --data "name=rate-limiting-advanced" \ + --data "config.limit=5" \ + --data "config.window_size=300" \ + --data "config.sync_rate=-1" \ + --data "config.strategy=redis" \ + --data "config.redis.host=" \ + --data "config.redis.port=6379" \ + --data "config.redis.ssl=true" \ + --data "config.redis.ssl_verify=true" +``` + +### Step 2: Add the Redis certificate to `lua_ssl_trusted_certificate` and restart Kong + +Store `/usr/local/share/ca-certificates/ca.crt` from Step 4 above somewhere Kong can access. + +If Kong is not installed in Docker/K8s, please set `lua_ssl_trusted_certificate=/path/to/ca.crt` in the Kong configuration file (e.g., `kong.conf`). + +If Kong is installed in Docker/K8s, please set the environment variable `KONG_LUA_SSL_TRUSTED_CERTIFICATE=/path/to/ca.crt`. + +Please check the `lua_ssl_trusted_certificate` reference for more detail. + +### Step 3: Check the result by making 6 consecutive requests + +```bash +#For the 1st ~ 5th access +curl -i :8000/ +HTTP/1.1 200 OK +Content-Type: application/json +Content-Length: 491 +Connection: keep-alive +RateLimit-Remaining: (This value shows how many times remaining, e.g 4-0) +RateLimit-Limit: 5 +X-RateLimit-Limit-300: 5 +X-RateLimit-Remaining-300: (This value shows how many times remaining, e.g 4-0) +RateLimit-Reset: 61 +Server: gunicorn/19.9.0 +Date: Thu, 06 Aug 2026 14:49:00 GMT +Access-Control-Allow-Origin: * +Access-Control-Allow-Credentials: true +X-Kong-Upstream-Latency: 18 +X-Kong-Proxy-Latency: 90 +Via: 1.1 kong/3.14.0.0-enterprise-edition +X-Kong-Request-Id: c48f99447ab88a09c1259026afa74b6e + +... + +#For the 6th access +curl -i :8000/ +HTTP/1.1 429 Too Many Requests +Date: Thu, 06 Aug 2026 14:49:00 GMT +Content-Type: application/json; charset=utf-8 +Connection: keep-alive +RateLimit-Remaining: 0 +RateLimit-Limit: 5 +X-RateLimit-Limit-300: 5 +X-RateLimit-Remaining-300: 0 +Retry-After: 120 +RateLimit-Reset: 120 +Content-Length: 37 +X-Kong-Response-Latency: 0 +Server: kong/3.14.0.0-enterprise-edition +X-Kong-Request-Id: 691d82238eb0d90064fbad24d7e3bb32 + +{"message":"API rate limit exceeded"} +``` + +## 3. Can multiple Kong nodes share the same Redis instance? + +Yes. + +For example, Kong1 and Kong2 are installed in different environments and use different databases. They can share the same Redis instance for their `rate-limiting-advanced` plugins. No conflict will happen. diff --git a/app/_support/how-to-use-the-deck-with-container.md b/app/_support/how-to-use-the-deck-with-container.md new file mode 100644 index 0000000000..08530205ad --- /dev/null +++ b/app/_support/how-to-use-the-deck-with-container.md @@ -0,0 +1,89 @@ +--- +title: How to use the `decK` docker container +content_type: support +description: Use the `decK` Docker container to dump, sync, reset, and validate Kong declarative configuration against a running Kong Admin API. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I use the `decK` Docker container to manage Kong configuration? + a: | + Run the official `kong/deck` Docker image with `docker run`, mounting a local directory as a volume and pointing `--kong-addr` at your Kong Admin API. From there you can use `dump`, `sync`, `reset`, and `validate` the same as a locally installed `decK` binary. +--- + +## Overview + +How to use the `decK` docker container. + +## Steps + +Assuming Kong is running locally and using the default port, the examples below use `http://host.docker.internal:8001` as the endpoint of the Kong Admin API for the `decK` container. Replace this value with your actual Kong Admin API URL. + +1. Pull the image: + + ```bash + docker pull kong/deck + ``` + +2. Check available commands: + + ```bash + docker run kong/deck --help + ``` + +3. Run `--help` with each command to check available flags: + + ```bash + docker run kong/deck dump --help + docker run kong/deck sync --help + docker run kong/deck reset --help + ... + ``` + +4. Run `deck dump` to export `kong.yaml`. The file is generated in `$(pwd)/kong.yaml`: + + ```bash + docker run -i \ + -v $(pwd):/deck \ + kong/deck --kong-addr http://host.docker.internal:8001 --headers kong-admin-token: -o /deck/kong.yaml dump + ``` + +5. Run `deck dump` to export declarative configuration files for all workspaces. The files are generated in `$(pwd)/`. + + Note: if you run more than one `dump` into the same directory, `decK` will similarly prompt to confirm overwriting existing files, which fails the same way under `docker run -i` without a TTY (`Error: EOF`). Unlike `reset`, `dump` has no `-f`/`--force` flag at all (passing one fails with `Error: unknown shorthand flag: 'f' in -f`) - remove or rename the existing output file(s) before re-running `dump` instead. + + ```bash + docker run -i \ + -v $(pwd):/deck \ + --workdir /deck \ + kong/deck --kong-addr http://host.docker.internal:8001 --headers kong-admin-token: dump --all-workspaces + ``` + +6. Run `deck reset` to reset the Kong objects to their initial state. + + Note: `reset` prompts for an interactive confirmation, which cannot be answered when running under `docker run -i` without a TTY (it fails with `Error: EOF`). Pass `-f`/`--force` to skip the prompt. + + ```bash + docker run -i \ + -v $(pwd):/deck \ + kong/deck --kong-addr http://host.docker.internal:8001 --headers kong-admin-token: reset -f + ``` + +7. Run `deck sync` to import `kong.yaml`. This example assumes `kong.yaml` is in `$(pwd)/kong.yaml`: + + ```bash + docker run -i \ + -v $(pwd):/deck \ + kong/deck --kong-addr http://host.docker.internal:8001 --headers kong-admin-token: -s /deck/kong.yaml sync + ``` + +8. Run `deck validate` to validate `kong.yaml`. This example assumes `kong.yaml` is in `$(pwd)/kong.yaml`: + + ```bash + docker run -i \ + -v $(pwd):/deck \ + kong/deck --kong-addr http://host.docker.internal:8001 --headers kong-admin-token: validate --online -s /deck/kong.yaml + ``` diff --git a/app/_support/how-to-use-the-relevant-paramters-for-refresh-token-authentication-method-in-the-openid-connect-plugin.md b/app/_support/how-to-use-the-relevant-paramters-for-refresh-token-authentication-method-in-the-openid-connect-plugin.md new file mode 100644 index 0000000000..0137803889 --- /dev/null +++ b/app/_support/how-to-use-the-relevant-paramters-for-refresh-token-authentication-method-in-the-openid-connect-plugin.md @@ -0,0 +1,43 @@ +--- +title: How to use the relevant parameters for refresh token authentication method in the `openid-connect` plugin +content_type: support +description: The `refresh_token_param_type` parameter determines where Kong will look for the refresh token in an incoming request. +tldr: + q: How do I configure the `openid-connect` plugin to accept a refresh token in the request body? + a: | + Set `refresh_token_param_type` to `body` so Kong reads the refresh token from the request body (JSON or form-urlencoded), and set `refresh_token_param_name` to the key you send (e.g. `refresh-token`). +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +--- + +## Overview + +We are trying to use the refresh token authentication method with the `openid-connect` plugin but it is not clear how to make this work when setting `refresh_token_param_type` to body. What is an example of how to configure the `openid-connect` plugin, and send the refresh token in a body + +## Steps + +The `refresh_token_param_type` parameter determines where Kong will look for the refresh token in an incoming request. + +If this is set to `body` only then kong expects the refresh token to be sent as part of the request body. The body can be either in a JSON or form url encoded. + +These are two curl examples: + +```bash +curl --request POST \ +--url \ +--header 'Content-Type: application/x-www-form-urlencoded' \ +--data refresh-token= +``` + +```bash +curl --request POST \ +--url \ +--header 'Content-Type: application/json' \ +--data '{"refresh-token": ""}' +``` + +Note that the `refresh_token_param_name` parameter determines what key Kong will be looking for the token for. For the above examples to work, `refresh_token_param_name` needs to be set to `refresh-token` diff --git a/app/_support/how-to-use-the-route-transformer-advanced-plugin-to-modify-only-part-of-the-path.md b/app/_support/how-to-use-the-route-transformer-advanced-plugin-to-modify-only-part-of-the-path.md new file mode 100644 index 0000000000..00bdcf2b4b --- /dev/null +++ b/app/_support/how-to-use-the-route-transformer-advanced-plugin-to-modify-only-part-of-the-path.md @@ -0,0 +1,71 @@ +--- +title: How to use the route-transformer-advanced plugin to modify only part of the path +content_type: support +description: The `route-transformer-advanced` plugin allows you to use any of the current request headers, query parameters, and captured URI groups as templates to populate supported config fields. +tldr: + q: How do I use the route-transformer-advanced plugin to modify only part of the request path? + a: | + Use a captured URI group in the plugin's `path` template to rewrite only a portion of the path — for example capture `~/old_path/(?\S+)` and set `path: /anything/new_path/$(uri_captures['path'])`. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: request-transformer-advanced template as value + url: /plugins/request-transformer-advanced/#templates +--- + +## Overview + +How to use the route-transformer-advanced plugin to modify only some part of the request path? + +## Steps + +The `route-transformer-advanced` plugin allows you to use any of the current request headers, query parameters, and captured URI groups as templates to populate supported config fields. + +These templates are described in the `request-transformer-advanced` plugin. + +This deck example modifies the requests' original route from `/old_path/` to `/anything/new_path/` + +```yaml +_format_version: "3.0" +services: +- host: httpbin.org + name: httpbin.org + port: 443 + protocol: https + routes: + - name: old_path + paths: + - ~/old_path/(?\S+) + plugins: + - config: + path: /anything/new_path/$(uri_captures['path']) + name: route-transformer-advanced +``` + +```bash +curl http://localhost:8000/old_path/asdf/qwerty?myparam=value1 +{ + "args": { + "myparam": "value1" + }, + "data": "", + "files": {}, + "form": {}, + "headers": { + "Accept": "*/*", + "Host": "httpbin.org", + "User-Agent": "curl/7.68.0", + "X-Amzn-Trace-Id": "Root=1-64365e22-2b5844ec206841dc09909874", + "X-Forwarded-Host": "localhost", + "X-Forwarded-Path": "/old_path/asdf/qwerty", + "X-Forwarded-Prefix": "/old_path/asdf/qwerty" + }, + "json": null, + "method": "GET", + "origin": "192.168.16.1, 213.195.110.197", + "url": "https://localhost/anything/new_path/asdf/qwerty?myparam=value1" +} +``` diff --git a/app/_support/how-to-validate-a-custom-header-equal-to-integer.md b/app/_support/how-to-validate-a-custom-header-equal-to-integer.md new file mode 100644 index 0000000000..7a4af555ef --- /dev/null +++ b/app/_support/how-to-validate-a-custom-header-equal-to-integer.md @@ -0,0 +1,76 @@ +--- +title: how to use request validator plugin to validate whether a custom header is equal to an integer +content_type: support +description: "Use the `request-validator` plugin's `parameter_schema` to require a custom header value to match a specific integer, then apply the plugin to a route through the Admin API." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I use the request-validator plugin to check that a custom header equals a specific integer? + a: | + Define a `parameter_schema` entry for the header in the `request-validator` plugin config, using a JSON Schema with `minimum` and `maximum` set to the required integer value. Apply the plugin to the route through the Admin API, then Kong rejects requests where the header doesn't match. +related_resources: [] +--- + +## Overview + +How to use the `request-validator` plugin to validate whether a custom header is equal to an integer? + +## Steps + +1. Write the below JSON file called `header-equal-int.json`. The following config requires that the `x-h1` header value is `1`. Replace `x-h1` and `1` with your actual values: + + ```json + { + "name": "request-validator", + "config": { + "version": "draft4", + "parameter_schema": [ + { + "name": "x-h1", + "in": "header", + "required": true, + "schema": "{\"type\": \"number\", \"minimum\": 1, \"maximum\": 1}", + "style": "simple", + "explode": false + } + ] + } + } + ``` + +2. Enable the request-validator plugin on your route with the JSON file above: + + ```bash + curl -X POST http://{KONG}:8001/routes/{ROUTE}/plugins \ + -H "Content-Type: application/json" \ + --data @header-equal-int.json + ``` + + You could also patch the above JSON file to an existing request-validator plugin: + + ```bash + curl -X PATCH http://{KONG}:8001/plugins/ \ + -H "Content-Type: application/json" \ + --data @header-equal-int.json + ``` + +3. Testing: + + ```bash + ❯ curl :8000/{ROUTE} -H "x-h1:2" + {"message":"request param doesn't conform to schema"}% + + ❯ curl :8000/{ROUTE} -H "x-h1:0" + {"message":"request param doesn't conform to schema"}% + + ❯ curl :8000/{ROUTE} -H "x-h1:a" + {"message":"request param doesn't conform to schema"} + + ❯ curl :8000/{ROUTE} -H "x-h1:1" + 200 response + ``` + + Confirmed Kong will proxy the request to upstreams only when `x-h1` equals `1`. diff --git a/app/_support/http-499-error-responses-when-kong-is-behind-a-cdn-and-using-sse.md b/app/_support/http-499-error-responses-when-kong-is-behind-a-cdn-and-using-sse.md new file mode 100644 index 0000000000..b1ef8eb71f --- /dev/null +++ b/app/_support/http-499-error-responses-when-kong-is-behind-a-cdn-and-using-sse.md @@ -0,0 +1,36 @@ +--- +title: HTTP 499 error responses when Kong is behind a CDN and using SSE +content_type: support +description: HTTP error 499 means that the client closed the connection in the middle of processing the request through the server or before the server answered the request. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does Kong return HTTP 499 error responses when it's behind a CDN and using SSE? + a: | + {{site.base_gateway}} responds with HTTP 200, but the CDN's response to the client is HTTP 499, because the request sent through Kong needs to be shorter than the proxy read timeout or Kong terminates the upstream connection. Disable buffering on the route with `request_buffering = false` and `response_buffering = false`, or use the `X-Accel-Buffering: no` header for SSE connections to disable buffering explicitly. +related_resources: + - text: "X-Accel-Buffering: no header" + url: https://github.com/nginxinc/nginx-wiki/blob/master/source/start/topics/examples/x-accel.rst +--- + +## Problem + +When Kong is behind a CDN and using SSE (Server Sent Events), clients receive HTTP 499 error responses even though Kong itself returns HTTP 200. + +## Cause + +HTTP error 499 means that the client closed the connection in the middle of processing the request through the server or before the server answered the request. This is a common situation when allowing SSE (Server Sent Events) and Kong is behind a CDN: {{site.base_gateway}} responds with HTTP 200, but the CDN responds with HTTP 499. In this scenario, the request sent through Kong needs to be shorter than the proxy read timeout or Kong will terminate the upstream connection. + +## Solution + +To prevent this issue, you can disable buffering in the route setting: + +``` +request_buffering = false +response_buffering = false +``` + +Kong also supports the `X-Accel-Buffering: no` header for SSE connections to disable buffering explicitly. You can find more information in the article: How can I use Kong to allow SSE (Server Sent Events)? diff --git a/app/_support/invalid-credentials-error-in-kong-manager-after-authenticating-user-in-external-idp.md b/app/_support/invalid-credentials-error-in-kong-manager-after-authenticating-user-in-external-idp.md new file mode 100644 index 0000000000..5206dfe7d5 --- /dev/null +++ b/app/_support/invalid-credentials-error-in-kong-manager-after-authenticating-user-in-external-idp.md @@ -0,0 +1,25 @@ +--- +title: "\"Invalid credentials\" error in Kong Manager after authenticating user in external IDP" +content_type: support +description: An incorrect `admin_claim` configuration — not `consumer_claim` — causes Kong Manager to reject OIDC logins even after the external IdP authenticates successfully. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does Kong Manager show an "Invalid credentials" error when authentication is successful in the external IdP? + a: | + Kong Manager's OIDC admin login always forces `consumer_optional=true`, so `consumer_claim` has no effect on admin auth — the field that actually controls it is `admin_claim` (default `email`). Confirm `admin_claim` is configured correctly and that the corresponding claim is present in the token returned by the external IdP. +--- + +## Problem + +When enabling OIDC for authentication in Kong Manager, it redirects to the external IDP and you can verify the authentication is successful. But when it redirects back to Kong Manager, it shows an "Invalid credentials" error, even though authentication was successful in the external IDP. + +## Solution + +This behavior is caused by an incorrect configuration of the `admin_claim` attribute, not `consumer_claim`. Kong Manager's OIDC admin authentication always forces `consumer_optional=true`, so `consumer_claim` has no effect on admin login. The field that actually controls admin authentication is `admin_claim`, which defaults to `email`. Check that `admin_claim` is correctly configured, for example: `"admin_claim": "email"`, and that the corresponding claim is present in the token returned by the external IdP. + +Note that an OIDC authentication failure in Kong Manager returns a generic 401 response, not the literal "Invalid credentials" string — that string is reserved for non-OIDC (basic/key/etc.) admin authentication failures. diff --git a/app/_support/is-it-possible-for-kong-to-notify-when-an-session-is-created.md b/app/_support/is-it-possible-for-kong-to-notify-when-an-session-is-created.md new file mode 100644 index 0000000000..ecb277182a --- /dev/null +++ b/app/_support/is-it-possible-for-kong-to-notify-when-an-session-is-created.md @@ -0,0 +1,123 @@ +--- +title: Sending notifications when a Kong session is created using Event Hooks +content_type: support +description: You can use Kong Event Hooks to send notifications to a webhook endpoint. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Is it possible for Kong to notify when a session is created? + a: | + Yes. Use Kong Event Hooks to send a notification to a webhook endpoint on a CRUD event for the `sessions` entity. Check the available event hook sources and the `sessions` schema through the Admin API, then create an event hook with `source: crud`, `event: sessions`, and a `webhook-custom` handler pointing at your endpoint. +related_resources: + - text: Kong Event Hooks documentation + url: /gateway/entities/event-hook/ +--- + +## Sending notifications when a Kong session is created + +Can Kong be configured to send a notification when an authorized session is created? For example, when logging in to Kong Manager? + +You can use Kong Event Hooks to send notifications to a webhook endpoint. There is further detail on Kong Event Hooks at the documentation link below. + +For our requirement, we can check to see what event hook sources are available. Specifically, we are interested in CRUD events for the `sessions` entity: + +```bash +curl -sk -X GET 'https://api.kong.lan:8444/event-hooks/sources' -H 'Kong-Admin-Token: password' | jq '.data.crud.sessions' +{ + "fields": [ + "operation", + "entity", + "old_entity", + "schema" + ] +} +``` + +From the result above, we can see that the source of the event is `crud` and the event is for `sessions`. We can also see the available fields for the webhook notification are `operation`, `entity`, `old_entity`, and `schema`. + +But what parameters does the `sessions` entity have? For this information, we can make a request to the `/schemas` endpoint for the `sessions` entity: + +```bash +curl -sk -X GET 'https://api.kong.lan:8444/schemas/sessions' -H 'Kong-Admin-Token: password' | jq +{ + "fields": [ + { + "id": { + "uuid": true, + "auto": true, + "len_min": 1, + "type": "string", + "description": "A string representing a UUID (universally unique identifier)." + } + }, + { + "session_id": { + "required": true, + "len_min": 1, + "unique": true, + "type": "string" + } + }, + { + "expires": { + "type": "integer" + } + }, + { + "data": { + "type": "string", + "len_min": 1 + } + }, + { + "created_at": { + "type": "integer", + "timestamp": true, + "auto": true, + "description": "An integer representing an automatic Unix timestamp in seconds." + } + }, + { + "ttl": { + "type": "number", + "between": [ + 0, + 100000000 + ], + "description": "Time-to-live value for data" + } + } + ], + "entity_checks": [] +} +``` + +Now we have all the information needed for the event hook, which can be created with an Admin API call like this: + +```bash +curl -s -X POST 'https://api.kong.lan:8444/event-hooks' \ +-H 'content-type: application/json' \ +-H 'Kong-Admin-Token: password' \ +--data-raw '{ + "source": "crud", + "event": "sessions", + "handler": "webhook-custom", + "config": { + "body_format": true, + "method": "POST", + "payload_format": true, + "headers_format": false, + "payload": { + {% raw %}"text": "webhook-custom for sessions : operation {{ operation }}, entity.session_id {{ entity.session_id }}, entity.expires {{ entity.expires }}, entity.data {{ entity.data }}, entity.created_at {{ entity.created_at }} schema {{ schema }}"{% endraw %} + }, + "url": "", + "headers": { + "content-type": "application/json" + }, + "ssl_verify": false + } +}' +``` diff --git a/app/_support/is-it-possible-to-rename-a-workspace.md b/app/_support/is-it-possible-to-rename-a-workspace.md new file mode 100644 index 0000000000..aab6a1de4b --- /dev/null +++ b/app/_support/is-it-possible-to-rename-a-workspace.md @@ -0,0 +1,38 @@ +--- +title: Renaming a workspace in {{site.base_gateway}} +content_type: support +published: false +description: "Renaming a workspace is not supported at the moment because it is associated with other database entities, such as RBAC roles." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Is it possible to rename a workspace? + a: | + No — renaming a workspace isn't supported because the workspace name is referenced by other database entities, such as RBAC roles. To rename one, create a new workspace with the desired name, use decK to migrate the Kong entities into it, and then delete the old workspace. +--- + +## Renaming a workspace + +Can a workspace be renamed? + +Renaming a workspace is not supported at the moment because it is associated with other database entities. + +One of the entities that are using workspaces entities is RBAC roles. If we check RBAC roles in the database, we can see the workspace's name is associated with it. + +```sql +kong=# select * from rbac_roles; + id | name | comment | created_at | is_default +--------------------------------------+---------------------+---------------------------------------------------------------------------+------------------------+------------ + 343b0991-0f74-4e6e-8258-4b3efb47499d | default:read-only | Read access to all endpoints, across all workspaces | 2026-06-11 08:59:32+00 | f + 64f4b79e-58bc-4538-bbea-e9b17fb72ab0 | default:admin | Full access to all endpoints, across all workspaces—except RBAC Admin API | 2026-06-11 08:59:32+00 | f + 9df65539-f068-4c96-a3cf-8f09357cbfa2 | default:super-admin | Full access to all endpoints, across all workspaces | 2026-06-11 08:59:32+00 | f + 89dcba7b-e803-462f-b3ff-73f876d3abfa | default:kong_admin | Default user role generated for kong_admin | 2026-06-11 08:59:32+00 | t +``` + +Our recommendation is to create a new workspace with the new name, migrate the Kong entities to the new workspace and then delete the old workspace. + +The best way to perform this migration is to use decK. This will allow you to backup the configuration into declarative config format, edit the config to use the new workspace name, sync the configuration to the new workspace and delete the old workspace. diff --git a/app/_support/keeping-kong-functional-during-db-down-times.md b/app/_support/keeping-kong-functional-during-db-down-times.md new file mode 100644 index 0000000000..4149a3674e --- /dev/null +++ b/app/_support/keeping-kong-functional-during-db-down-times.md @@ -0,0 +1,61 @@ +--- +title: Keeping Kong Functional During DB Down Times +content_type: support +description: Configure database cache TTLs, warmup entities, and memory settings so {{site.base_gateway}} keeps proxying during database maintenance or downtime. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: the related pull request + url: https://github.com/Kong/kong/pull/4565 +tldr: + q: How can {{site.base_gateway}} stay functional when the database is down? + a: | + Configure `db_cache_ttl`, `db_resurrect_ttl`, `db_update_frequency`, and `db_cache_warmup_entities` to keep the cache resilient and minimize database dependency during downtime. Increase `mem_cache_size` if warming additional entities triggers cache-size warnings in the logs. +--- + +## Problem + +When the database is down—due to maintenance or other reasons—{{site.base_gateway}} needs additional configuration to keep functioning without interruption. + +## Solution + +During periods when the database is down—due to maintenance or other factors—you can configure several settings to minimize impact and keep the Gateway operational. Assuming you use the DB strategy, the only limitation is that vitals may not be written to the database during downtime. + +Key configuration properties to manage cache behavior and provide resilience when the database is unavailable are: + +- `db_cache_ttl` +- `db_resurrect_ttl` +- `db_update_frequency` + +The `db_cache_warmup_entities` setting can be configured to prevent database access by warming up specific entities in the cache if they haven't been used recently. By default, only `services` are warmed up by this setting — `plugins` are pre-warmed separately and are explicitly ignored by `db_cache_warmup_entities`. You can include other entities, such as: + +- `acls` +- `acme_storage` +- `basicauth_credentials` +- `ca_certificates` +- `certificates` +- `clustering_data_planes` +- `consumers` +- `hmacauth_credentials` +- `jwt_secrets` +- `keyauth_credentials` +- `oauth2_authorization_codes` +- `oauth2_credentials` +- `oauth2_tokens` +- `parameters` +- `ratelimiting_metrics` +- `rbac_roles` +- `rbac_users` +- `services` +- `sessions` +- `snis` +- `targets` +- `upstreams` +- `workspaces` + +Adjust `mem_cache_size` if warnings about cache size appear in the logs due to warming multiple entities. + +Note: The `routes` entity does not require warming since it is cached in the in-memory router object. See the related pull request for reference. diff --git a/app/_support/kong-ai-gateway-error-expected-512-dimensions-not-1024-when-using-ai-semantic-cache.md b/app/_support/kong-ai-gateway-error-expected-512-dimensions-not-1024-when-using-ai-semantic-cache.md new file mode 100644 index 0000000000..74001e785e --- /dev/null +++ b/app/_support/kong-ai-gateway-error-expected-512-dimensions-not-1024-when-using-ai-semantic-cache.md @@ -0,0 +1,29 @@ +--- +title: "{{site.ai_gateway_name}}: \"ERROR: expected 512 dimensions, not 1024\" when using `ai-semantic-cache`" +content_type: support +description: Switching embedding models to a different dimension size after the `ai-semantic-cache` plugin's vector table was created causes a Postgres dimension-mismatch error in the background cache-store logs. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: "Why does {{site.ai_gateway_name}} log \"ERROR: expected 512 dimensions, not 1024\" when using `ai-semantic-cache`?" + a: | + The `ai-semantic-cache` plugin's vector table locks in the embedding dimension size the first time it's used; switching to a model with a different dimension count doesn't update the table, so `config.vectordb.dimensions` no longer matches. Delete and re-create the plugin to regenerate the table with the new dimensions. +--- + +## Problem + +The following error appears in the Gateway logs when using the `ai-semantic-cache` plugin: + +`"Unable to store response in the cache: failed to insert key: ERROR: expected 512 dimensions, not 1024, context: ngx.timer"` + +## Solution + +This error occurs when the plugin was initially configured with a specific embedding dimension (e.g., 512), and later switched to an embedding model with a different dimension size (e.g., 1024). The underlying issue is that the embedding field in the vector database table is initialized using the dimensions set during the first use of the cache. Changing the model does not automatically update the table structure. + +Note that if `config.vectordb.dimensions` isn't updated alongside the new model, Kong now intercepts the mismatch earlier and returns a synchronous 500 error to the client, rather than letting the request through. The `ERROR: expected 512 dimensions, not 1024` Postgres error above only ever appears in the Gateway's server logs — it comes from a background timer write (the cache-store step) and is never returned to the client. + +To resolve this, you need to delete and re-create the plugin. Doing so generates a new table in the vector database with the correct embedding dimensions. diff --git a/app/_support/kong-deck-when-deploying-multiple-files-at-once-global-select-tags-are-applied-to-all-resources.md b/app/_support/kong-deck-when-deploying-multiple-files-at-once-global-select-tags-are-applied-to-all-resources.md new file mode 100644 index 0000000000..0a35214167 --- /dev/null +++ b/app/_support/kong-deck-when-deploying-multiple-files-at-once-global-select-tags-are-applied-to-all-resources.md @@ -0,0 +1,95 @@ +--- +title: "Kong decK: When deploying multiple files at once, global `select_tags` are applied to all resources" +content_type: support +description: Both resources are applying the global tags values as they are being applied at the same time. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why do decK's global `select_tags` apply to all resources when deploying multiple files at once? + a: | + When deploying multiple files together, the `select_tags` values from each file are all applied globally at the same time, so every resource ends up with the combined tags instead of just the ones in its own file. To keep tags scoped per resource, set specific resource-level `tags` instead of relying on `select_tags`. +related_resources: [] +--- + +## Problem + +We are trying to deploy multiple files at once to {{site.base_gateway}} using decK, all with different tags. However, we noticed that all resources created contain the same tags even though the file specifies different tags. + +For example: + +File 1: + +```yaml +_format_version: "3.0" +_info: + defaults: {} + select_tags: + - global + - global-auth +``` + +File 2: + +```yaml +_format_version: "3.0" +_info: + defaults: {} + select_tags: + - global + - global-no-auth +``` + +Sample command: + +```bash +deck gateway sync --kong-addr 'http://localhost:8001/' --headers 'kong-admin-token:token' plugin1.yaml plugin2.yaml +``` + +How can we resolve this issue? + +## Solution + +Both resources are applying the global tags values as they are being applied at the same time. To resolve this, we need to move from `select-tags` to specific resource tags. + +Example: + +```yaml +_format_version: "3.0" +_info: + defaults: {} + select_tags: + - global +_workspace: testworkspace +plugins: +- config: + ... + enabled: true + name: openid-connect + tags: + - global-oidc-auth +``` + +```yaml +_format_version: "3.0" +_info: + defaults: {} + select_tags: + - global +_workspace: testworkspace +plugins: +- config: + ... + enabled: true + name: pre-function + tags: + - global-no-auth +``` + +Now if we run the sync command - both resources get created with the specific tags. + +```bash +deck gateway sync --kong-addr 'http://localhost:8001/' --headers 'kong-admin-token:token' plugin1.yaml plugin2.yaml +``` diff --git a/app/_support/kong-fails-to-start-with-the-error-password-authentication-failed-for-user.md b/app/_support/kong-fails-to-start-with-the-error-password-authentication-failed-for-user.md new file mode 100644 index 0000000000..675f3dc12a --- /dev/null +++ b/app/_support/kong-fails-to-start-with-the-error-password-authentication-failed-for-user.md @@ -0,0 +1,49 @@ +--- +title: "Kong fails to start with the error \"password authentication failed for user\"" +content_type: support +description: A hash mark (`#`) in a Postgres password set via `kong.conf` gets truncated as a comment, causing Kong to fail to start with a password authentication error. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does Kong fail to start with a "password authentication failed for user" error? + a: | + A hash mark (`#`) in a `kong.conf`-configured Postgres password is treated as a comment, truncating the password. Escape it as `\#` in `kong.conf`, or set the password via the `KONG_PG_PASSWORD` environment variable instead, which isn't affected. +--- + +## Problem + +After installing and trying to start Kong, the below error is thrown: + +``` + +Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:156: +nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:462: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: FATAL: password authentication failed for user "root" +stack traceback: + [C]: in function 'assert' + /usr/local/share/lua/5.1/kong/init.lua:462: in function 'init' + init_by_lua:3: in main chunk +a:3: in main chunk + + Run with --v (verbose) or --vv (debug) for more details +waiting for db +``` + +You will notice the migrations jobs have run, however Kong fails to start using the same db account. A review of the Postgres logs will show a similar error + +``` + +2025-12-15 21:00:00 UTC:10.10.100.190(26680):root@kong:[10839]:FATAL: password authentication failed for user "root" +2025-12-15 21:00:00 UTC:10.10.100.190(26680):root@kong:[10839]:DETAIL: Password does not match for user "root". +``` + +## Solution + +The issue stems from the handling of special characters in the Postgres password. In particular, if the password contains a hash mark (#), Kong treats everything after it as a comment when parsing `kong.conf`, which truncates the password and produces this error. + +This only affects password values set in the `kong.conf` file. If the password is instead supplied via the environment variable `KONG_PG_PASSWORD`, the hash mark is not interpreted as a comment and the value is passed through correctly, so no change is needed in that case. + +For passwords set in `kong.conf`, rather than changing the password to remove the hash mark, escape it as `\#` in the file. This preserves the real password (including the `#` character) while preventing Kong from truncating it at parse time. diff --git a/app/_support/kong-gateway-401-http-response-and-unable-to-verify-digest-error-seen-after-updating-signing-certificate-in-idp.md b/app/_support/kong-gateway-401-http-response-and-unable-to-verify-digest-error-seen-after-updating-signing-certificate-in-idp.md new file mode 100644 index 0000000000..bbc9a1d580 --- /dev/null +++ b/app/_support/kong-gateway-401-http-response-and-unable-to-verify-digest-error-seen-after-updating-signing-certificate-in-idp.md @@ -0,0 +1,35 @@ +--- +title: "{{site.base_gateway}}: 401 HTTP response and \"unable to verify digest\" error seen after updating signing certificate in IdP" +content_type: support +description: "When the Kong OIDC plugin throws a 401 error following an update to the IDP signer certificate, it is typically due to the plugin's cache not being updated with the new certificate information." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does the Kong OIDC plugin return an HTTP 401 with an "unable to verify digest" error after the IdP's signing certificate is updated? + a: | + The plugin's cache isn't refreshed because the JWKS `kid` (Key ID) didn't change even though the signing certificate did, so the cache has no signal to recrawl. Short-term, disable and re-enable the plugin (or clear the cache via the `/openid-connect/issuers` endpoint) to force a recrawl. Long-term, use dynamic JWKS `kid` values instead of static ones so client-side caches refresh automatically. +related_resources: [] +--- + +## Problem + +We are using the Kong OpenID Connect (openid-connect / OIDC) plugin. We use extra JWKS URLs and recently the IdPs signing certificate was updated. We are now observing the following error in our logs with failed requests (HTTP 401 responses): + +``` + +2023/09/19 04:59:59 [notice] 26170#0: *7857901 [lua] responses.lua:21: [openid-connect] unable to RSA SHA512 verify digest, client: {ipAddress}, server: kong, request: "POST {URI} HTTP/1.1", host: "{hostname}" +``` + +What does this error mean and why are we seeing this, how can we resolve this issue? + +## Solution + +When the Kong OIDC plugin throws a 401 error following an update to the IDP signer certificate, it is typically due to the plugin's cache not being updated with the new certificate information. This can happen when the `kid` (Key ID) in the JWKS (JSON Web Key Set) does not change even though the signing certificate has been updated. The OIDC plugin's cache expects either a new `kid` or a new issuer to trigger a recrawl of the JWKS endpoint. + +There is a short-term and long-term solution to this issue below: + +1. The short-term solution is to disable and re-enable the plugin, this will force a cache rebuild and allow the {{site.base_gateway}} to recrawl for the latest JWKS. If there are many instances of this plugin affected, then the quickest method will be to delete the cache system-wide with either a restart of the node or a cURL command to the `/openid-connect/issuers` API endpoint. An example request to that endpoint: `curl -X DELETE http://:8001/openid-connect/issuers` +2. The long-term solution is to discontinue use of static key IDs. Although employing static `kid` values in the JSON Web Key Set (JWKS) is allowed, it introduces challenges and is not aligned with industry-recommended practices. The preferred strategy is using dynamic `kid` values as they are automatically rotated to a different value when the signing certificate or other relevant details are updated. This approach not only enhances security but also ensures automatic refreshment of client-side caching without manual intervention. If you continue to use static `kid` values, then you will need to incorporate the cache clearing steps above in step 1 into your process whenever the JWKS are modified or signing certificates are changed. diff --git a/app/_support/kong-gateway-ai-proxy-content-type-header-does-not-match-request-body.md b/app/_support/kong-gateway-ai-proxy-content-type-header-does-not-match-request-body.md new file mode 100644 index 0000000000..c84c56ba62 --- /dev/null +++ b/app/_support/kong-gateway-ai-proxy-content-type-header-does-not-match-request-body.md @@ -0,0 +1,56 @@ +--- +title: "{{site.base_gateway}}: AI Proxy \"content-type header does not match request body\"" +content_type: support +description: "Explains why the AI Proxy plugin's `content-type header does not match request body` error is dead code in current {{site.base_gateway}} releases, and how to resolve the `request body doesn't contain valid inputs` error that replaced it." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does the AI Proxy plugin log `content-type header does not match request body`, and how do I resolve it? + a: | + This literal error message is dead code in current {{site.base_gateway}} releases and no longer appears — the same underlying issue now surfaces as `request body doesn't contain valid inputs`. Confirm the content type is `application/json` and that the request body matches the shape your configured LLM provider expects. +--- + +## Problem + +When attempting to use the AI-Proxy plugin we receive the below error: + +Response Body: + +```json + +{ + "error": { + "message": "content-type header does not match request body" + } +} +``` + +Kong Error log: + +``` +2024/05/27 13:09:33 [warn] 2160906#0: *2936 [kong] handler.lua:23 [ai-proxy] content-type header does not match request body, client: 192.168.10.10, server: kong, request: "POST /ai HTTP/1.1", host: "kong", request_id: "cbfefa2a2e15db3d91ce0224dea8fb1d" +``` + +What causes this and how can it be resolved? + +## Cause + +The exact error message shown above (`content-type header does not match request body`) is now unreachable dead code in current {{site.base_gateway}} releases — you will not see this literal message anymore. The underlying situation it used to describe (a genuine mismatch between the content-type header and the request body, or a request body the AI Proxy plugin cannot parse) still occurs, but now instead surfaces as a different error: + +```json +{ + "error": { + "message": "request body doesn't contain valid inputs" + } +} +``` + +## Solution + +If you encounter this error, check that the content type is `application/json` and use a validator to confirm the JSON payload is properly structured and matches the shape expected by the configured LLM provider format. + +Note: an undersized `nginx_http_client_body_buffer_size` (causing the request body to be buffered to disk) does not, on its own, reproduce this error. Buffered-to-disk log lines near the time of the error are not the root cause and increasing this value is not a fix for this issue. diff --git a/app/_support/kong-gateway-auth0-idp-returns-invalid-bearer-token-when-using-with-consumer-claim.md b/app/_support/kong-gateway-auth0-idp-returns-invalid-bearer-token-when-using-with-consumer-claim.md new file mode 100644 index 0000000000..2217731f28 --- /dev/null +++ b/app/_support/kong-gateway-auth0-idp-returns-invalid-bearer-token-when-using-with-consumer-claim.md @@ -0,0 +1,52 @@ +--- +title: "{{site.base_gateway}}: Auth0 IDP returns invalid bearer token when using with consumer claim" +content_type: support +description: This happens specifically with Auth0 when there is no audience specified with the OIDC configuration. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does Auth0 return an invalid bearer token that fails the OIDC plugin's consumer claim mapping? + a: | + Auth0 omits the expected claim (e.g. `azp`) from the token when no `audience` is specified in the OIDC configuration. Set the `audience` field in the OIDC plugin to the application's API identifier from Auth0 so the IdP issues a token with the claim needed for consumer mapping. +related_resources: [] +--- + +## Problem + +OIDC logs show this error: + +``` + +2024/02/29 23:43:05 [notice] 2397#0: *128795 [lua] responses.lua:24: [openid-connect] kong consumer was not found (claim (azp) was not found for consumer mapping), client: 172.17.0.1, server: kong, request: "GET /new?code=fNoCoi60sMzlUGq6ISsavJoZzA0-QQUKlHPuYqzeQWgLk&state=vSCOSePIItmEK-qSdefWeMQ9 HTTP/1.1", host: "localhost:8000", request_id: "ed621d4370bad6319cf058330812184e" +``` + +The bearer token returned by the IDP is invalid when decoded: + +``` + +eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiaXNzIjoiaHR0cHM6Ly9kZXYtanhvZm9ydjdqY2FnMWt3by51cy5hdXRoMC5jb20vIn0..QvcoFJl3EGkMvAJr.x2mQPwSQHg7aJWRqYu8oydMKyfLaT0NZ6jKEGp8HIB0x0amIM4OqbNiviF_LR2MYX5e0uWYPtVyTs5THFP-8bvBRV98lUHNBqoTPNsMo3b9uLWOOzByr7_mrZHlXTEHY-tprio56igFtmHE_WJ0LnHlIfBx855Ro1ul6OvY5-LOSefYU8lTF9tcnIG49q6WWiUy65heZrX8tOOrGiejd2fdCv37-Pn4H2kynsLo-l_eWI19qIkd4y7vVoouxQU60VvD0urSHXw1VWXxv882k1NvHaELmm8tLsvl1OOUXshRIAAV7v3g5S5pyp2NAvD5pXCt13g.yo6tr-ftGFZlRjBjobLsew +``` + +Decoding the token + +## Solution + +This happens specifically with Auth0 when there is no audience specified with the OIDC configuration. Audience value is taken from the IDP. + +Navigate to the application in the IDP and click on the API tab. We will find the identifier defined for this application. The identifier has to be added to the OIDC plugin under the `Audience` field. + +IDP: + +Kong OIDC Plugin config: + +Once this is added, we can see that the IDP returns proper token as shown below and the error with consumer claims is resolved as well. + +Valid token: + +``` + +eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IlFianJNOXhUa2NSLU9EMTdrdFhjRSJ9.eyJpc3MiOiJodHRwczovL2Rldi1qeG9mb3J2N2pjYWcxa3dvLnVzLmF1dGgwLmNvbS8iLCJzdWIiOiJhdXRoMHw2NWQzYmI4ZDFmNDk3OTNlYzlkMWViOGIiLCJhdWQiOlsiaHR0cDovL2tvbmcuZXhhbXBsZS5jb20iLCJodHRwczovL2Rldi1qeG9mb3J2N2pjYWcxa3dvLnVzLmF1dGgwLmNvbS91c2VyaW5mbyJdLCJpYXQiOjE3MDkyNTA0OTUsImV4cCI6MTcwOTMzNjg5NSwiYXpwIjoiRjZYU3dLa0RkaWtZamZuT1BQaWhieDA1dzdLYWkzWTEiLCJzY29wZSI6Im9wZW5pZCJ9.XmBE2G1tn9QBWR-Qan9POhznoBfI1U-BLDkOtbAKc7cEbYcAytd5WzMXnZcEZ15HOClf-mfG-oplo8E4Znfo0xSDCvj-t_KoinFQcaIHOq1vg0WEMnEm2ahwnc0oFsIsGEtyxcvYY7k6pmjD2LMnIW5pUuTZZg-srM30dNIaRpGoc6v7OIuRi1UqkA0VuFUXNtU8UqbXoLG2LD5mS9qyAerQVjE6vsvyLenWgJPRBg5b2oT3ReFnaRXFXUJqf8Rq63q9zGVPkmofEEn2JRTgwto4tJelfPpBEJWNSvm1mBSwht2oozV3fw3Y_cNAL0tj1g0mhMjwof0EJnVwGJ6wgg +``` diff --git a/app/_support/kong-gateway-connections-intended-for-websocket-route-are-routing-to-http-routes.md b/app/_support/kong-gateway-connections-intended-for-websocket-route-are-routing-to-http-routes.md new file mode 100644 index 0000000000..4aa88fcb2b --- /dev/null +++ b/app/_support/kong-gateway-connections-intended-for-websocket-route-are-routing-to-http-routes.md @@ -0,0 +1,31 @@ +--- +title: "{{site.base_gateway}}: Connections intended for websocket route are routing to HTTP routes" +content_type: support +published: false +description: To force the request to use the websocket route we need to add some differentiator to the route. +tldr: + q: Why do connections intended for my websocket route get routed to an equivalent HTTP route instead? + a: | + When two routes are otherwise identical, Kong's route priority matching can send the request to the HTTP route. + Add a differentiator such as the `Upgrade: websocket` header to the websocket route so it is prioritized. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: priority matching + url: /gateway/entities/route/#priority-matching +--- + +## Problem + +We have 2 identical routes configured. The only difference is one uses WSS and the other uses HTTP. When we are calling out the websocket route we have noticed that the equivalent HTTP route is being called in its place. If I reach out to the websocket endpoint directly the request is processed properly through the WSS protocol. How can we resolve this and force the connection to go to the websocket route. + +## Solution + +To force the request to use the websocket route we need to add some differentiator to the route. + +One option is to add the header `Upgrade: websocket` to the route. + +When curling the 2 routes now it will prioritize the request that contains the header `upgrade: websocket` to the WSS route. This behavior is governed by Kong's route priority matching. diff --git a/app/_support/kong-gateway-error-data-plane-lua-365-clustering-unable-to-update-running-config-no-memory-dp-won-t-sync.md b/app/_support/kong-gateway-error-data-plane-lua-365-clustering-unable-to-update-running-config-no-memory-dp-won-t-sync.md new file mode 100644 index 0000000000..8681e84f39 --- /dev/null +++ b/app/_support/kong-gateway-error-data-plane-lua-365-clustering-unable-to-update-running-config-no-memory-dp-won-t-sync.md @@ -0,0 +1,50 @@ +--- +title: "{{site.base_gateway}}: Error: \"data_plane.lua:365: [clustering] unable to update running config: no memory\" DP won't sync" +content_type: support +description: "When the dataplane fails to sync it will throw a message of \"unable to update running config: xyz\"." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: "Why does {{site.base_gateway}} show \"unable to update running config: no memory\" when the data plane won't sync?" + a: | + This means there's no memory left for the `db_cache`, not system memory. Check the `/status` endpoint for the `kong_core_db_cache` values, then increase `mem_cache_size` to accommodate your config's growth. +related_resources: + - text: "`mem_cache_size` configuration reference" + url: /gateway/configuration/#mem-cache-size +--- + +## Problem + +We are running into an issue where we are receiving 404s on our routes and our config.cache.json isn't being generated. + +Our logs are displaying the following error: + +``` + +data_plane.lua:365: [clustering] unable to update running config: no memory +``` + +However our system memory looks to be alright and nothing is maxing out. What can be done to resolve this error? + +## Solution + +When the dataplane fails to sync it will throw a message of "unable to update running config: `xyz`". In this case, the problem is that there is no memory left for the `db_cache`. + +To validate this we can call the `/status` endpoint and verify the `db_cache` values. + +In this scenario we would see something like this: + +```json + +"kong_core_db_cache": { +"allocated_slabs": "128.00 MiB", +"capacity": "128.00 MiB" +}, +``` + +To resolve this, we can increase our `mem_cache_size` value. + +This value will need to be increased based on your config growth and should be self monitored. diff --git a/app/_support/kong-gateway-error-postgresql-error-failed-to-retrieve-postgresql-server-version-num-authentication-exchange-unsuccessful.md b/app/_support/kong-gateway-error-postgresql-error-failed-to-retrieve-postgresql-server-version-num-authentication-exchange-unsuccessful.md new file mode 100644 index 0000000000..e41c5918c1 --- /dev/null +++ b/app/_support/kong-gateway-error-postgresql-error-failed-to-retrieve-postgresql-server-version-num-authentication-exchange-unsuccessful.md @@ -0,0 +1,40 @@ +--- +title: "{{site.base_gateway}}: \"Error: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: authentication exchange unsuccessful\"" +content_type: support +description: This error means your `PG_PASSWORD` parameter does not match the password set on the database. +products: + - gateway +works_on: + - on-prem + - konnect +published: false +tldr: + q: "Why does {{site.base_gateway}} show \"failed to retrieve PostgreSQL server_version_num: authentication exchange unsuccessful\" on startup?" + a: | + Your `PG_PASSWORD` doesn't match the password set on the PostgreSQL database. The messaging differs by version: PostgreSQL 14 reports this generic exchange error, while PostgreSQL 13 reports the more explicit `FATAL: password authentication failed for user "kong"`. +related_resources: [] +--- + +## Problem + +When starting up our new instance of {{site.base_gateway}}, we see the following error: + +``` + +Error: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: authentication exchange unsuccessful +``` + +We are using PostgreSQL 14.X + +## Solution + +This error means your `PG_PASSWORD` parameter does not match the password set on the database. + +The messaging has changed between PostgreSQL 13 and PostgreSQL 14. + +Example of PostgreSQL 13 password messaging: + +``` + +[PostgreSQL error] failed to retrieve PostgreSQL server_version_num: FATAL: password authentication failed for user "kong" +``` diff --git a/app/_support/kong-gateway-error-with-environment-variables-vault-could-not-find-cached-values.md b/app/_support/kong-gateway-error-with-environment-variables-vault-could-not-find-cached-values.md new file mode 100644 index 0000000000..8dcdfc25ee --- /dev/null +++ b/app/_support/kong-gateway-error-with-environment-variables-vault-could-not-find-cached-values.md @@ -0,0 +1,79 @@ +--- +title: "{{site.base_gateway}}: \"could not find cached values\" error with the environment variables vault backend" +content_type: support +description: "The \"could not find cached values\" error appears when a vault reference for the environment variables (`env`) vault backend can't resolve, often due to a mismatched `KONG_VAULT_ENV_PREFIX` or a lowercase environment variable name." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: Store secrets as environment variables + url: /gateway/entities/vault/#store-secrets-as-environment-variables + - text: Secrets management referenceable fields + url: /gateway/secrets-management/ + - text: How do I reference secrets stored in a vault? + url: /gateway/entities/vault/#how-do-i-reference-secrets-stored-in-a-vault +tldr: + q: How do I fix the "could not find cached values" error when using the environment variables vault backend? + a: | + The `env` vault backend requires the referenced environment variable name to be uppercase, and if `KONG_VAULT_ENV_PREFIX` is set, the vault's `config.prefix` must match it exactly. A lowercase variable name or a mismatched prefix causes {{site.base_gateway}} to log a `could not find cached values` error instead of resolving the secret. Use `kong vault get /` from inside the Kong container to verify the reference resolves. +--- + +## Problem + +When referencing a secret from the environment variables vault backend with a configured prefix, {{site.base_gateway}} logs a `could not find cached values` error instead of resolving the value. + +``` + +2024/03/07 16:47:52 [notice] 2307#0: *69496 [kong] vault.lua:708 error updating secret reference {vault://env/clientsecret}: could not find cached values, client: 172.28.0.1, server: kong, request: "GET /check HTTP/1.1", host: "localhost:48000" +``` + +## Solution + +There are different kinds of vault backends, one of which is environment variables. + +When using this, we would reference the variable to be used, as an environment variable in the Kong configuration file and then the reference will be able to pick it up when executing the plugin. + +Things to check: + +1. The environment variable in the docker file should have these added: The variables that would be referenced with the vault and if you are using an environment variable prefix, then `KONG_VAULT_ENV_PREFIX` should be added. In my example below, my `KONG_VAULT_ENV_PREFIX=MY_` and hence all the variables that I want this vault to reference will also have this prefix: + +```bash +-e "MY_CLIENTID=" \ + +-e "KONG_VAULT_ENV_PREFIX=MY_" \ + +-e "MY_CLIENTSECRET=" \ +``` + +2. Note that these environment variables should be specified in upper case. If specified in lower case, then they are not valid. More on this can be found here. + +3. Vault configuration: The name of the vault is: `my-env-vault`, type: `env`, `config.prefix="MY_"`. The `config.prefix` advanced setting should match the env variable: `KONG_VAULT_ENV_PREFIX` + +4. Check from inside Kong container if the reference is right. For this exec into the Kong container and execute the following command to view env variables and make sure the variables you created are showing up: + +```bash +printenv | sort +``` + +Execute the below command to know if from Kong container, we are able to reference these variables through vault parameters: + +```bash +kong vault get /environment_variable +Eg: kong vault get my-env-vault/clientsecret +``` + +Note that the prefix (`MY_`) is not added here to the env variable name. If we add them (`my_clientid`) and check, we will get the error found here: + +5. You could then reference as shown below in any of the supported referenceable fields in the plugin. + +Here I am referencing these in the OIDC plugin `clientid` and `clientsecret` fields. + +Syntax: + +``` +{vault://} +``` + +I am getting redirected to the IDP since the client ID and secret is valid: diff --git a/app/_support/kong-gateway-file-log-failed-to-open-the-file-permission-denied-while-logging-request-dev-stdout.md b/app/_support/kong-gateway-file-log-failed-to-open-the-file-permission-denied-while-logging-request-dev-stdout.md new file mode 100644 index 0000000000..d3375b07c4 --- /dev/null +++ b/app/_support/kong-gateway-file-log-failed-to-open-the-file-permission-denied-while-logging-request-dev-stdout.md @@ -0,0 +1,51 @@ +--- +title: "\"[file-log] failed to open the file: Permission denied\" error when logging to /dev/stdout as a non-root user" +content_type: support +description: "Explains why the file-log plugin returns a Permission denied error when logging to `/dev/stdout`, and how to fix it by keeping the container running as the `kong` user instead of root." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: file-log plugin documentation + url: /plugins/file-log/ +tldr: + q: Why does the file-log plugin return a "Permission denied" error when logging to /dev/stdout? + a: | + `/dev/stdout` is owned by the `kong` user with `700` permissions, so a {{site.base_gateway}} process running as root can't write to it. This typically happens when a Dockerfile switches to `USER root` for setup steps and never switches back. Add `USER kong` back to the Dockerfile before Kong starts, rebuild the image, and redeploy. +--- + +## Problem + +We are trying to deploy the file-log plugin with the path defined as `/dev/stdout`. However, we are running into the following error: + +``` + +[file-log] failed to open the file: Permission denied while logging request +``` + +When checking the permissions we don't see anything that is lacking. + +```bash +ls -al /dev/stdout +lrwxrwxrwx 1 root root 15 Mar 13 13:22 /dev/stdout -> /proc/self/fd/1 +``` + +The file-log plugin documentation shows that logging to `/dev/stdout` is supported, but running the plugin this way produces a permissions error. + +## Solution + +It is possible to configure the file-log plugin to `/dev/stdout`. The issue here is that the worker permissions are locked down. + +`/dev/stdout` is owned by kong with 700 permissions so root cannot write to that dir. + +This will happen if the user is switched to root in a Dockerfile. The way to resolve this is by changing the user back to kong. In your Dockerfile we need to add the following: + +```dockerfile +USER kong +``` + +After this we need to re-create the image and then re-deploy kong with the updated image. + +Rerun the file-log plugin now, and the issue will be resolved. diff --git a/app/_support/kong-gateway-graceful-shutdown-of-kong-in-ecs-not-working.md b/app/_support/kong-gateway-graceful-shutdown-of-kong-in-ecs-not-working.md new file mode 100644 index 0000000000..47fa64ac0e --- /dev/null +++ b/app/_support/kong-gateway-graceful-shutdown-of-kong-in-ecs-not-working.md @@ -0,0 +1,82 @@ +--- +title: "{{site.base_gateway}}: Graceful Shutdown of Kong in ECS not working" +content_type: support +description: "Kong's master process runs as PID 1 in ECS tasks, so ECS sends SIGTERM directly to it instead of honoring the container's `STOPSIGNAL`, causing in-flight requests to fail. Using an init process such as `dumb-init` enables a graceful shutdown." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: dumb-init + url: https://github.com/Yelp/dumb-init +tldr: + q: Why doesn't {{site.base_gateway}} shut down gracefully when ECS scales down a task? + a: | + In ECS, Kong's master process runs as `PID 1`, so ECS sends `SIGTERM` directly to it instead of honoring the container's `STOPSIGNAL`, causing in-flight requests to fail. Running Kong under an init process such as `dumb-init` moves the master process off `PID 1` and lets you translate the incoming `SIGTERM` into a `SIGQUIT` for a graceful shutdown. +--- + +## Problem + +When running our Kong in ECS, when tasks scale down we see that Kong is not shutting down gracefully and thus in-flight requests are failing. Based on further investigating we see that ECS appears to not be honoring the `STOPSIGNAL` defined in the docker entrypoint. + +## Solution + +It is a known limitation of ECS that it does not honor `STOPSIGNAL` definitions. + +The primary issue is that Kong's master process is running as `PID 1` in the container. When ECS attempts to stop/scale down a task, it sends a `SIGTERM` signal directly to the Kong master process and thus Kong shuts down immediately. + +To remedy the situation, you need the Kong master process to NOT be running as `PID 1`. To achieve this, you need to use a tool such as `dumb-init`. This tool will replace `PID 1` with a 'dummy' process thus moving the Kong master process to some other PID. + +From here on out, our dummy process will receive the `SIGTERM` from ECS and, with a change to our `ENTRYPOINT`, we will forward a `SIGQUIT` to the Kong master process allowing for a graceful shutdown: + +```dockerfile + +ENTRYPOINT ["/usr/local/bin/dumb-init", "--rewrite", "15:3", "--"] +``` + +Below is a sample dockerfile for a Kong image that includes dumb-init: + +```dockerfile + +# load kong-ubuntu:latest +FROM kong/kong-gateway:latest-ubuntu + +USER root + +# add dumb-init +ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.5/dumb-init_1.2.5_x86_64 /usr/local/bin/dumb-init +RUN chmod +x /usr/local/bin/dumb-init + +COPY tini-wrapper.sh /tini-wrapper.sh + +USER kong + +# set tini as entrypoint +ENTRYPOINT ["/usr/local/bin/dumb-init", "--rewrite", "15:3", "--"] + +CMD ["/tini-wrapper.sh"] +``` + +And the corresponding tini-wrapper.sh: + +```bash + +#!/usr/bin/env bash +set -Eeo pipefail + +# translate SIGTERM to SIGQUIT for graceful shutdown +graceful_shutdown() { + echo "SIGTERM received, sending SIGQUIT to Kong..." + # get PID of Kong's master process + local kong_pid=$(pgrep -f "/usr/local/openresty/nginx/sbin/nginx") + # send SIGQUIT to master and all workers + kill -SIGQUIT "$kong_pid" +} + +trap 'graceful_shutdown' SIGTERM + +exec /entrypoint.sh kong docker-start +``` + +Attached (tini-warp.zip) is a full example of how you might achieve this diff --git a/app/_support/kong-gateway-host-header-case-sensitive-on-routes-for-route-matching-causing-http-404-errors.md b/app/_support/kong-gateway-host-header-case-sensitive-on-routes-for-route-matching-causing-http-404-errors.md new file mode 100644 index 0000000000..36035a205d --- /dev/null +++ b/app/_support/kong-gateway-host-header-case-sensitive-on-routes-for-route-matching-causing-http-404-errors.md @@ -0,0 +1,63 @@ +--- +title: "{{site.base_gateway}}: Host header case-sensitive on Routes for route matching, causing HTTP 404 errors" +content_type: support +description: "Host header matching on {{site.base_gateway}} Routes is case-sensitive, per RFC, even though header names are case-insensitive, so requests with a different-case Host header (e.g. `Example.com` vs. `example.com`) return a 404." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does a Route with a Host value of example.com return a 404 for requests with a different-case Host header, like Example.com? + a: | + {{site.base_gateway}}'s Route matching is case-sensitive for header values (though header names themselves are case-insensitive) per RFC, so `Example.com` won't match a Route configured with `example.com`, producing an HTTP 404. Work around it by normalizing the Host header at the client or load balancer, listing all expected case variations on the Route, or using a global pre-function plugin to lowercase the header before routing. +--- + +## Problem + +We are setting the Host parameter on a Route for matching-purposes, however we are finding some requests are not being matched properly to the Route. The pattern we've seen is the Host header in the request has a different case structure than what we've set on the Route. For example, our Route has a Host value of "example.com" but the request is coming in with a Host header value of "Example.com" and thus is not matched. These are causing 404 HTTP errors and we want to understand how best to address this problem. + +## Solution + +In this case, the {{site.base_gateway}} is working as designed/expected as header values are case-sensitive per RFC. While header names are case-insensitive, their values are case-sensitive. This is why "Example.com" is different from "example.com" and generates a 404. + +There are a few ways to resolve this situation: + +Method #1: Simply ensure that your client apps are sending the Host header correctly. If you have control over the application sending the requests for example, ensure it's sending the expected Host header to match what has been set in the {{site.base_gateway}}. For those apps which you do not have control over, make it known perhaps in the API documentation that they should be sending the Host header as all lower-case for example. + +Method #2: Some load balancers (LBs) actually have a function which can manipulate the headers for consistency so that they can all be lower-case values for example. If this is an option in your environment, we recommend this be considered. + +Method #3: You can set multiple values in the Host parameter on the Route. So in the event that you only see a few different variations (i.e. example.com, Example.com, EXAMPLE.COM), then it may be simpler to add those variations to the Host setting on the Route so that all commonly seen variations are matched correctly. + +Method #4: If the first two options are not possible in your environment, you may be able to use the Serverless Plugin (pre-function) with {{site.base_gateway}} to manipulate the headers prior to it reaching the Route matching phase. An example is included 'as-is' below. Please understand that Serverless function code is outside the scope of Kong Support. You may wish to contact your Account Executive to hire our Field Engineering / Professional Services team who can write the code to meet your specific use-case. + +To explain the JSON config from a Pre-Function plugin below from Method #4: the code to add is `ngx.req.set_header("Host", string.lower(ngx.req.get_headers()["Host"]))` to the rewrite phase and assign it at the global scope. Limitations: This does not work scoped to a particular Service or Route, it must be Global. This also limits the ability to add more Pre-Function plugins as only one Pre-Function plugin can be added to a particular scope at a time, meaning in this example another Pre-Function plugin could not be applied globally. + +```json + +{ + "service": null, + "config": { + "log": [], + "ws_handshake": [], + "certificate": [], + "ws_upstream_frame": [], + "rewrite": ["ngx.req.set_header(\"Host\", string.lower(ngx.req.get_headers()[\"Host\"]))"], + "access": [], + "ws_client_frame": [], + "header_filter": [], + "body_filter": [], + "ws_close": [] + }, + "consumer": null, + "name": "pre-function", + "created_at": 1679436267, + "id": "c0aa4d2d-941c-4225-9b70-bba140bd11a4", + "route": null, + "protocols": ["grpc", "grpcs", "http", "https"], + "ordering": null, + "tags": null, + "enabled": true +} +``` diff --git a/app/_support/kong-gateway-how-can-a-specific-json-key-be-replaced-in-the-response-sent-to-clients.md b/app/_support/kong-gateway-how-can-a-specific-json-key-be-replaced-in-the-response-sent-to-clients.md new file mode 100644 index 0000000000..1fa8b47962 --- /dev/null +++ b/app/_support/kong-gateway-how-can-a-specific-json-key-be-replaced-in-the-response-sent-to-clients.md @@ -0,0 +1,53 @@ +--- +title: "{{site.base_gateway}}: Replace a specific JSON key in the response sent to clients" +content_type: support +published: false +description: Use the transform functions of the Response Transformer Advanced plugin to rename a specific JSON key in the response sent to clients. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I replace a specific JSON key in the response sent to clients? + a: | + Use the `transform` functions of the Response Transformer Advanced plugin. Supply a Lua function that copies the value of the old key to the new key and then removes the old key from the response body. +related_resources: + - text: Response Transformer Advanced plugin examples + url: /plugins/response-transformer-advanced/examples/ +--- + +## Problem + +We have a JSON structure returned from our upstream server, for example: + +```json +{ + "name": "gruber", + "location": "nakatomi plaza", + "internal_id": "12fjsd9123" +} +``` + +We would like to replace the key name `internal_id` with `group_id` while retaining the value. How can this be done? + +## Solution + +You can achieve this using the transform functions of the Response Transformer Advanced plugin. + +For example, this checks that `internal_id` isn't nil/missing and sets `group_id` to the value before removing it. + +```lua +return function (data) + if type(data) ~= "table" then + return data +end + +if data["internal_id"] ~= nil then + data["group_id"] = data["internal_id"] + data["internal_id"] = nil +end + +return data +end +``` diff --git a/app/_support/kong-gateway-how-can-i-see-the-request-kong-receives-on-demand.md b/app/_support/kong-gateway-how-can-i-see-the-request-kong-receives-on-demand.md new file mode 100644 index 0000000000..33240dc154 --- /dev/null +++ b/app/_support/kong-gateway-how-can-i-see-the-request-kong-receives-on-demand.md @@ -0,0 +1,65 @@ +--- +title: View the request {{site.base_gateway}} receives on demand using the Request Termination plugin +content_type: support +description: "Use the Request Termination plugin's `config.echo` and `config.trigger` settings to echo back the full request — headers, body, and matched route/service — on demand for a single request." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "Request Termination plugin reference" + url: "/plugins/request-termination/" +tldr: + q: How can I see the request Kong receives on-demand? + a: | + Enable `config.echo` on the Request Termination plugin to echo back the request Kong receives — headers, body, and matched route/service. Set `config.trigger` to an arbitrary string so only requests carrying that string as a header or query parameter trigger the echo, keeping it non-intrusive for normal traffic. +--- + +## Problem + +When troubleshooting an issue, you may need to see the full request Kong receives — including headers and request body — without exposing that data for every request. + +## Solution + +The Request Termination plugin allows for this behavior. + +By setting `config.echo` to true the Gateway will echo back the following data: + +- `node_id`: Node ID of the Kong node that handled the request +- `worker_pid`: nginx worker PID that handled the request +- `hostname`: The hostname of the machine that handled the request +- request scheme: The scheme component of the request's URL. +- `request_host`: The host component of the request's URL, or the value of the "Host" header +- request port: The port component of the request's URL +- `request_headers`: The request headers received by Kong +- `request_query`: The query arguments obtained from the query string +- `request_body`: The request body received by Kong +- `request_method`: The HTTP method of the request. +- `request_path`: The normalized path component of the request's URL +- `matched_route`: The Kong route that matched the request (including route properties such as `path`, `protocols`, `strip_path`, etc.) +- `matched_service`: The Kong service that matched the request (including service properties such as connect/read/write timeouts, etc.) + +To set this on-demand, configure the `config.trigger` setting to an arbitrary string. Doing so allows only requests containing this string as a header or query parameter to activate the plugin and echo back the request. + +For example: `config.trigger=x-gruber` + +```bash +curl -H "x-gruber:1" localhost:8000/echo --data "name":"value" +(output shortened for readability) + +"kong": {..."hostname": "kong-node1}, + "message": "Service unavailable", + "matched_service": {..."read_timeout": 60000,"path": "/anything"}, + "matched_route": {..."path_handling": "v0","paths": ["/echo"],"strip_path": true}, + "request": { + "path": "/echo", + "headers": {..."host": "localhost:8000","x-gruber": "1" }, + "port": 8000, + "host": "localhost", + "method": "POST", + "scheme": "http", + "raw_body": "name:value" + } +} +``` diff --git a/app/_support/kong-gateway-how-can-i-use-the-request-transformer-advanced-plugin-to-add-headers-stored-as-a-secret.md b/app/_support/kong-gateway-how-can-i-use-the-request-transformer-advanced-plugin-to-add-headers-stored-as-a-secret.md new file mode 100644 index 0000000000..3bb74625e4 --- /dev/null +++ b/app/_support/kong-gateway-how-can-i-use-the-request-transformer-advanced-plugin-to-add-headers-stored-as-a-secret.md @@ -0,0 +1,85 @@ +--- +title: Adding headers stored as a secret with the Request Transformer Advanced plugin +content_type: support +description: "Store a header's `value` (not the whole `headerName:headerValue` pair) as the secret when using Secrets Management with the Request Transformer Advanced plugin, to avoid the header name getting URL-encoded." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How can I use the Request Transformer Advanced plugin to add headers stored as a secret? + a: | + Store only the header's `value` as the secret, not the whole `headerName:headerValue` pair — otherwise the header name gets mangled and URL-encoded. Reference it with `config.add.header: {vault://env/}` for environment variables, or `{vault://aws/}` (or `{vault://aws//}` for multi-value secrets) for AWS Secrets Manager. +--- + +## Problem + +When testing the Kong Secrets Management with the Request Transformer Advanced plugin, you notice that the formatting is incorrect. + +Example: + +```json +{ + "headers": { + "%7B%22Name%22": "\"gruber\"}", + "Accept": "*/*", + "Host": "httpbin.org", +} +``` + +## Cause + +Failure to format this properly will result in extra characters being added to the header name, which in turn, get URL encoded as seen above. + +## Solution + +To use a secret as a header, the correct format is to store the `value` as seen below. + +``` +headerName:headerValue +``` + +Examples: + +### Environment variable + +For environment variables, including Docker and Kubernetes: + +```bash +NEWHEADER="addMe:addValue" +``` + +Referenced as: `config.add.header: {vault://env/newheader}` + +```json +{ + "headers": { + "Accept": "*/*", + "Addme": "addValue", + "Host": "httpbin.org", +} +``` + +### AWS Secrets Manager + +AWS Secrets Manager defaults to storing secrets as key/value pairs. To store headers in AWS Secrets Manager, there are a few options available. + +Option 1: + +Secret type: other type of secret + +Key/value pairs: Choose plaintext, delete the default JSON formatting and replace it with the Header Name and Header Value separated by a colon. + +Name: This can be whatever you like and will be used later when we reference the secret in the Request Transformer Plugin. For the example here I will name the secret `header`. + +`addMe:addValue` + +Referenced as: `config.add.header: {vault://aws/secretname}` + +Option 2: + +If the requirement is to store multiple Key/Value pairs inside one AWS Secret, then the Headers can be stored as a Key/Value pair inside the Secret Value. The Request Transformer Advanced plugin can then retrieve any specific Secret Value by referencing the Secret Key and add it as a header. + +Referenced as: `config.add.header: {vault://aws/secretname/header}` diff --git a/app/_support/kong-gateway-how-can-you-log-the-upstream-hostname-in-a-logging-plugin.md b/app/_support/kong-gateway-how-can-you-log-the-upstream-hostname-in-a-logging-plugin.md new file mode 100644 index 0000000000..7b4e5f670b --- /dev/null +++ b/app/_support/kong-gateway-how-can-you-log-the-upstream-hostname-in-a-logging-plugin.md @@ -0,0 +1,53 @@ +--- +title: "{{site.base_gateway}}: Log the upstream/target hostname in a logging plugin" +content_type: support +published: false +description: "`balancer_data` returns the service host, not the upstream target host; use `ngx.var.upstream_host` to log the upstream/target hostname in a logging plugin." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How can I log the upstream/target hostname in a logging plugin? + a: | + `ngx.ctx.balancer_data.host` returns the service host configured in Kong, not the resolved upstream target. + Use `ngx.var.upstream_host` instead to log the actual upstream/target hostname. +--- + +## Problem + +We are looking to grab the upstream/target hostname in the logging plugin. However we are noticing that when we grab the value of `ctx.balancer_data.host` it is returning the value of the upstream resource inside Kong. + +Example: + +pre-function: + +Log phase: + +```lua +kong.log.err(ngx.ctx.balancer_data.host) +``` + +Returns: + +``` +[error] 2193#0: *5947 [kong] [string "kong.log.err(ngx.ctx.balancer_data.host)"]:1 [pre-function] SampleUpstreamName while logging request +``` + +How can we get the upstream/target host returned instead? + +## Cause + +`balancer_data` returns the host of the service that is being utilized. In this case, the service host is `SampleUpstreamName`. So this is reflected when `ngx.ctx.balancer_data.host` is utilized. + +## Solution + +To return the upstream/target host you can use the variable `ngx.var.upstream_host` + +Sample output: + +``` +2025/08/29 18:16:12 [error] 2193#0: *6163 [kong] [string "kong.log.err(ngx.var.upstream_host)"]:1 [pre-function] mockbin.org while logging request, client: 123.34.45.567, server: kong, request: "GET /upstream HTTP/1.1", upstream: "http://123.34.45.567:80/request", host: "localhost:8000" +``` diff --git a/app/_support/kong-gateway-how-to-configure-aws-lambda-plugin-when-doing-cross-account-function-invocation.md b/app/_support/kong-gateway-how-to-configure-aws-lambda-plugin-when-doing-cross-account-function-invocation.md new file mode 100644 index 0000000000..8595450da2 --- /dev/null +++ b/app/_support/kong-gateway-how-to-configure-aws-lambda-plugin-when-doing-cross-account-function-invocation.md @@ -0,0 +1,93 @@ +--- +title: "{{site.base_gateway}}: How to configure AWS Lambda plugin when doing cross-account function invocation" +content_type: support +description: "Covers the AWS IAM role and `aws-lambda` plugin misconfigurations that cause cross-account AWS Lambda invocation errors in {{site.base_gateway}}, and how to correctly set up role assumption using the `aws_assume_role_arn` field." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: aws-lambda plugin configuration + url: /plugins/aws-lambda/ +tldr: + q: Why does the aws-lambda plugin fail to invoke a Lambda function in another AWS account? + a: | + Cross-account Lambda invocation with the `aws-lambda` plugin requires an IAM role in the target account that trusts the calling account, with the calling account's IAM user or role allowed to assume it. In {{site.base_gateway}}, set the ARN of that role in the plugin's `aws_assume_role_arn` field — omitting it, or granting permissions directly to the IAM user instead of using role assumption, produces `not authorized to perform: lambda:InvokeFunction` or `Cross-account log access is not allowed` errors. +--- + +## Problem + +I configured IAM permission policies to allow one AWS IAM user to invoke a lambda function which is owned by another AWS account, after applied the AWS Lambda plugin in {{site.base_gateway}}, I see errors like below: + +``` + +[aws-lambda] User: arn:aws:iam::851725256956:user/sk2 is not authorized to pe +rform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-2:851725256956:function:sgao_test_func because no identity-based policy allows the lambda:InvokeFunction action, client: 10.0.0.1, server: kong, request: "GET /lambda HTTP/1.1", host: "localhost:8000", request_id: "f472d3038ca6f260129d74215c5cdd16" +``` + +Note: this happens when correct AWS IAM permission policies and assumeRole permissions are in place, but missing the `Aws Assume Role Arn` value in the AWS Lambda plugin + +Or + +``` + +2024/03/21 14:30:44 [error] 2188#0: *1477941 [kong] init.lua:351 [aws-lambda] {"Message":"Cross-account log access is not allowed"}, client: 10.51.210.234, server: kong, request: "GET /platform/v1/download?id=3535345345 HTTP/1.1", host: "api.us.dev.contoso.cloud"} +``` + +Note: this could happen when grant permissions directly to the IAM user in the other AWS account to invoke lambda functions in the target AWS account. However, this approach may increase the complexity of managing permissions across accounts, and it's generally considered a better practice to use role assumption for cross-account access whenever possible. + +## Solution + +The core issue revolves around the AWS IAM role/permission policy and the `aws-lambda` plugin configuration in {{site.base_gateway}}, specifically when attempting to invoke an AWS Lambda function across different AWS accounts. + +The above errors are most likely related to misconfiguration on AWS and Kong's `aws-lambda` plugin. + +To resolve this issue, follow these steps: + +1. **Cross-Account Lambda Function Invocation Setup:** + +In the AWS account owning the Lambda function: + +- Create an IAM Role that grants the necessary permissions for invoking Lambda functions. This role will be assumed by IAM users from another AWS account. + +- Define a trust relationship with the other AWS account, allowing IAM users from that account to assume the role. + +- Attach policies to the IAM role that grant permissions to invoke Lambda functions. + +In the other AWS account (the one used by the `aws-lambda` plugin): + +- Create an IAM user who will be invoking Lambda functions in the above AWS account. + +- Attach a policy to the IAM user that allows assuming the IAM role created in the above account. + +2. **{{site.base_gateway}} `aws-lambda` Plugin Configuration:** + +Ensure that the role ARN is added into the `aws-lambda` plugin configuration. This is crucial for cross-account Lambda function invocation. The relevant field in the plugin configuration is `aws_assume_role_arn`. + +Here is an example snippet of the `aws-lambda` plugin configuration with the `aws_assume_role_arn` field: + +```json + +{ +"config": { +"function_name": "arn:aws:lambda:us-east-1:123456789012:function:your-function-name", +"aws_region": "us-east-1", +"aws_key": "your-aws-access-key", +"aws_secret": "your-aws-secret-key", +"aws_assume_role_arn": "arn:aws:iam::123456789012:role/your-assume-role", +"invocation_type": "RequestResponse", +"log_type": "Tail", +"timeout": 60000, +"forward_request_headers": true +} +} +``` + +Ensure that the `aws_assume_role_arn` field is correctly set with the ARN of the IAM role created for cross-account access. + +3. **Additional Tips:** + +- For logging purposes, if you encounter issues with logs not appearing when using the `aws-lambda` plugin, consider adjusting the `log_type` configuration based on your requirements for security monitoring and troubleshooting. + +By following these steps, you should be able to configure the `aws-lambda` plugin in {{site.base_gateway}} for successful cross-account Lambda function invocation. diff --git a/app/_support/kong-gateway-how-to-get-correlation-id-inside-the-access-phase-when-using-correlation-id-plugin.md b/app/_support/kong-gateway-how-to-get-correlation-id-inside-the-access-phase-when-using-correlation-id-plugin.md new file mode 100644 index 0000000000..64fcacd02a --- /dev/null +++ b/app/_support/kong-gateway-how-to-get-correlation-id-inside-the-access-phase-when-using-correlation-id-plugin.md @@ -0,0 +1,54 @@ +--- +title: "{{site.base_gateway}}: How to get Correlation ID inside the Access Phase when using Correlation ID Plugin" +content_type: support +description: "Whether a custom or pre-function plugin can read the Correlation ID generated by the Correlation ID plugin during the Access phase depends on plugin execution priority — in {{site.ee_product_name}}, use a `post-function` plugin, which runs after the Correlation ID plugin." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: Plugins execution order + url: /custom-plugins/handler.lua/#plugins-execution-order +tldr: + q: How do you get the Correlation ID inside the Access phase when using the Correlation ID plugin? + a: | + Whether the Correlation ID is available in the Access phase of another plugin depends on plugin execution priority. If your custom or pre-function plugin runs before the `correlation-id` plugin (priority `100001`), the ID won't exist yet. In {{site.ee_product_name}}, use a `post-function` plugin instead — its priority (`-1000`) is lower, so it always runs after `correlation-id` and can read the generated ID via `kong.request.get_header("kong-request-id")`. +--- + +## Problem + +How do you get the Correlation ID inside the access phase of a custom plugin or a pre-function plugin? During our testing we noticed that if the Correlation ID was manually added as part of the request package prior, it will be picked up successfully by the pre-function plugin inside the Access Phase. However, if we execute the Correlation ID plugin to generate the Correlation ID we then notice that the Correlation ID is not accessible inside the Access Phase. + +## Solution + +This depends on the plugin execution order. If the Correlation ID plugin runs after your pre-function plugin then it will not be able to capture the Correlation ID being added by the Correlation ID plugin. + +The Correlation ID plugin priority changes depending on which type of {{site.base_gateway}} is being utilized open-source, free mode or enterprise. + +To grab the Correlation ID in {{site.ee_product_name}}, you need to use the `post-function` plugin as the priority is lower than the Correlation ID. + +Plugins with higher priority will be executed first. + +``` + +PLUGIN PRIORITY +correlation-id 100001 +post-function -1000 +``` + +Post-function plugin configuration: + +```yaml + +- config: + access: + - kong.log.err(kong.request.get_header("kong-request-id")) +``` + +Now inside the {{site.base_gateway}} logs, you will see the Correlation ID logged: + +``` + +2023/04/03 13:49:21 [error] 2106#0: *2164920 [kong] [string "kong.log.err(kong.request.get_header("kong-re..."]:1 [post-function] e940eb22-5a83-4821-824a-b8faec93ded9, client: 192.168.0.1, server: kong, request: "POST /test HTTP/1.1", host: "localhost:8000" +``` diff --git a/app/_support/kong-gateway-how-to-get-opentelemetry-span-id-and-trace-id-inside-the-file-log-plugin.md b/app/_support/kong-gateway-how-to-get-opentelemetry-span-id-and-trace-id-inside-the-file-log-plugin.md new file mode 100644 index 0000000000..eae9a52631 --- /dev/null +++ b/app/_support/kong-gateway-how-to-get-opentelemetry-span-id-and-trace-id-inside-the-file-log-plugin.md @@ -0,0 +1,53 @@ +--- +title: "{{site.base_gateway}}: How to get OpenTelemetry Span ID and Trace ID inside the file log plugin" +content_type: support +published: false +description: The OpenTelemetry plugin automatically appends the headers for whichever tracing utility is being utilized. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I get the OpenTelemetry Span ID and Trace ID inside the file-log plugin? + a: | + The OpenTelemetry plugin appends a `traceparent` header to the request. Its value has the form `00---01`, so the first segment is the Trace ID and the second is the Span ID. + The file-log plugin records request headers, so the `traceparent` header (and therefore both IDs) appears in the logged request headers. +--- + +## Logging the OpenTelemetry Span ID and Trace ID in the file-log plugin + +We are utilizing the OpenTelemetry plugin and the file log plugin. We have a requirement where we need the Span ID and Trace ID inside the file log. How can we log these 2 fields? + +The OpenTelemetry plugin automatically appends the headers for whichever tracing utility is being utilized. + +For example if Zipkin is being utilized, then you will notice a `traceparent` header being appended. + +```json +"traceparent": "00-72a6c1be825f5339cacd16c58254835c-2b1f74d3b0939afd-01" +``` + +This contains both your TraceID and your SpanID. + +The first larger string is your TraceID and the smaller string is your SpanID. + +``` +traceid = 72a6c1be825f5339cacd16c58254835c +spanid = 2b1f74d3b0939afd +``` + +Now when we use the file log plugin these headers will be appended to the Request Headers section. + +```json + "request": { + "method": "GET", + "size": 118, + "headers": { + "traceparent": "00-72a6c1be825f5339cacd16c58254835c-2b1f74d3b0939afd-01", + "accept": "*/*", + "host": "localhost:8000", + "user-agent": "insomnia/2023.1.0", + "kong-debug": "1" + } +``` diff --git a/app/_support/kong-gateway-how-to-rate-limit-oauth2-token-endpoint.md b/app/_support/kong-gateway-how-to-rate-limit-oauth2-token-endpoint.md new file mode 100644 index 0000000000..25e5e87fb0 --- /dev/null +++ b/app/_support/kong-gateway-how-to-rate-limit-oauth2-token-endpoint.md @@ -0,0 +1,143 @@ +--- +title: "{{site.base_gateway}}: How to rate limit OAuth2 token endpoint" +content_type: support +description: Out of the box it is not a configurable option to limit token creation through the OAuth2 plugin. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I rate limit the OAuth2 token endpoint in {{site.base_gateway}}? + a: | + The OAuth2 plugin has no built-in option to limit token creation. As a workaround, create a second service whose path points at the `/oauth2/token` endpoint, expose it through a new route, and apply the Rate Limiting Advanced plugin to that service so token generation through the proxy is rate limited. + Direct access to the original token endpoint is still possible and must be blocked by external means if required. +--- + +## Rate limiting the OAuth2 token endpoint + +We setup the OAuth2 plugin and can confirm it is working as expected. However, when we try to apply a rate limiting advanced (RLA) plugin with the OAuth2 plugin, we noticed that the token endpoint is not being rate limited. For example if we setup the RLA plugin with `config.limits` as 3 and `config.window_size` as 30 globally. We can endlessly call the `/oauth2/token` endpoint. Is there a way to rate limit token generation? + +Out of the box it is not a configurable option to limit token creation through the OAuth2 plugin. + +However, there is a workaround where if we proxy the `/oauth2/token` endpoint through the proxy. We can then rate limit the service this way. + +## Steps + +Steps to test: + +1) Create workspace + +```bash +curl --request POST \ + --url http://localhost:8001/workspaces \ + --header 'Content-Type: multipart/form-data' \ + --header 'kong-admin-token: ' \ + --form name=teamA +``` +2) Create Service + +```bash +curl --request POST \ + --url http://localhost:8001/teamA/services \ + --header 'Content-Type: multipart/form-data' \ + --header 'kong-admin-token: ' \ + --form name=mockbin \ + --form url=http://mockbin.org/request +``` +3) Create Route + +```bash +curl --request POST \ + --url http://localhost:8001/teamA/services/mockbin/routes \ + --header 'Content-Type: multipart/form-data' \ + --header 'kong-admin-token: ' \ + --form name=mockbin \ + --form paths=/mockbin +``` +4) Create OAuth2 plugin + +```bash +curl --request POST \ + --url http://localhost:8001/teamA/services/mockbin/plugins \ + --header 'Content-Type: multipart/form-data' \ + --header 'kong-admin-token: admin' \ + --form name=oauth2 \ + --form config.scopes=email \ + --form config.enable_authorization_code=true \ + --form config.enable_client_credentials=true +``` +5) Create consumer + +```bash +curl --request POST \ + --url http://localhost:8001/teamA/consumers \ + --header 'Content-Type: multipart/form-data' \ + --header 'kong-admin-token: admin' \ + --form username=user1 +``` +At this point we can generate tokens successfully. + +Example call: + +```bash +curl --request POST -k \ + --url 'https://localhost:8443/mockbin/oauth2/token' \ + --header 'Content-Type: multipart/form-data' \ + --header 'kong-admin-token: admin' \ + --form client_id=456 \ + --form client_secret=789 \ + --form grant_type=client_credentials +``` +To begin Rate Limiting them we need to do the following: + +6) Create a service with the `/oauth2/token` endpoint. + +```bash +curl --request POST \ + --url http://localhost:8001/teamA/services \ + --header 'Content-Type: application/json' \ + --header 'Kong-Admin-Token: admin' \ + --data '{ + "name": "Oauth", + "retries": 0, + "host": "localhost", + "path": "/mockbin/oauth2/token", + "port": 8443, + "protocol":"https" +}' +``` +7) Create a route for the previous service + +```bash +curl --request POST \ + --url http://localhost:8001/teamA/services/Oauth/routes \ + --header 'Content-Type: multipart/form-data' \ + --header 'kong-admin-token: admin' \ + --form name=oauthRoute \ + --form paths=/oauth +``` +At this point we can call the new service/route to generate a token. + +8) Add a Rate Limiting Advanced (RLA) plugin. + +```bash +curl --request POST \ + --url http://localhost:8001/teamA/services/mockbin/plugins \ + --header 'Content-Type: application/json' \ + --header 'kong-admin-token: admin' \ + --data '{ + "name": "rate-limiting-advanced", + "config": + { + "limit":[3], + "window_size":[30], + "sync_rate": -1, + "strategy": "local" +} +}' +``` +We can now proxy the token endpoint for exactly 3 times in 30 seconds and then it will rate limit the token creation. + +**Direct access to the token endpoint is still possible and should be blocked using external means if this is a requirement.** diff --git a/app/_support/kong-gateway-how-to-remove-portion-of-path-leaving-and-sending-remainder-to-backend.md b/app/_support/kong-gateway-how-to-remove-portion-of-path-leaving-and-sending-remainder-to-backend.md new file mode 100644 index 0000000000..1d5d06e935 --- /dev/null +++ b/app/_support/kong-gateway-how-to-remove-portion-of-path-leaving-and-sending-remainder-to-backend.md @@ -0,0 +1,53 @@ +--- +title: "{{site.base_gateway}}: How to remove portion of path leaving and sending remainder to backend" +content_type: support +published: false +description: "To accomplish this we can use capture groups and then the request transformer advanced plugin to trip the \"/test/api\"." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I remove a portion of the request path in {{site.base_gateway}} and send only the remainder to the backend? + a: | + Use a capture group in the route's path regex to capture the segment you want to keep, for example `~/test/api/(?v2/validtest2)`. Then set `config.replace.uri` on the Request Transformer Advanced plugin to `/$(uri_captures['validtest'])` so only the captured portion of the path is sent upstream. +--- + +## {{site.base_gateway}}: How to remove portion of path leaving and sending remainder to backend + +We are looking to use 1 service and 2 routes. Each route will point to the same service and have different paths. However, our backend will handle each route differently. + +We are currently using the following paths: + +``` +/test/api/validtest1 +/test/api//v2/validtest2 +``` +We only need "validtest1" or "validtest2" sent to the backend. + +How can we remove "test/api" from the request without distorting the routes? + +To accomplish this we can use capture groups and then the request transformer advanced plugin to trip the "/test/api". + +Example: + +Route 1: + +``` +~/test/api/(?/validtest1) +``` +Route 2: + +``` +~/test/api/(?v2/validtest2) +``` +On the Request Transformer Advanced plugin we can set the following: + +`config.replace.uri`: + +``` +/$(uri_captures['validtest']) +``` +Now when you proxy requests to either of these 2 routes, the data entered into the capture group will only be sent to the upstream. diff --git a/app/_support/kong-gateway-is-it-possible-to-use-a-custom-log-format-without-using-a-custom-nginx-template.md b/app/_support/kong-gateway-is-it-possible-to-use-a-custom-log-format-without-using-a-custom-nginx-template.md new file mode 100644 index 0000000000..8725d80d08 --- /dev/null +++ b/app/_support/kong-gateway-is-it-possible-to-use-a-custom-log-format-without-using-a-custom-nginx-template.md @@ -0,0 +1,48 @@ +--- +title: "{{site.base_gateway}}: Custom log format without a custom nginx template" +content_type: support +published: false +description: "You can use the direct injection of nginx parameters to specify a custom log format without maintaining a custom nginx template." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: "{{site.base_gateway}}: Is it possible to use a custom log format without using a custom nginx template?" + a: | + Yes. Set the `nginx_http_log_format` parameter in `kong.conf` (or `KONG_NGINX_HTTP_LOG_FORMAT` in Docker) to define the custom format directly, then point `proxy_access_log` (or `KONG_PROXY_ACCESS_LOG` in Docker) at it. No custom nginx template is required. +related_resources: + - text: "{{site.base_gateway}} nginx directives" + url: /gateway/nginx-directives/ +--- + +## {{site.base_gateway}}: Is it possible to use a custom log format without using a custom nginx template + +The documentation shows an example of defining a custom nginx log format which uses a custom nginx template. This is an overhead to maintain between versions, is there a way to specify a custom format without a custom nginx file and the associated maintenance overhead? + +You can use the direct injection of nginx parameters to specify a log format. + +For example, define the desired log format using the `nginx_http_log_format` parameter in the `kong.conf`; + +```nginx +nginx_http_log_format=show_everything '\$time_iso8601 - \$bytes_sent - \$request - \$status - \$remote_addr' +``` + +In Docker, you would use environment variables as per the usual Docker configuration; + +```bash +-e "KONG_NGINX_HTTP_LOG_FORMAT=show_everything '\$time_iso8601 - \$bytes_sent - \$request - \$status - \$remote_addr'" +``` + +and then specify the log file to use the format in the `kong.conf`; + +```nginx +proxy_access_log=/dev/stdout show_everything +``` + +or via environment variables for Docker; + +```bash +-e "KONG_PROXY_ACCESS_LOG=/dev/stdout show_everything" +``` diff --git a/app/_support/kong-gateway-is-unable-to-create-a-session-after-successfully-authenticating-a-user.md b/app/_support/kong-gateway-is-unable-to-create-a-session-after-successfully-authenticating-a-user.md new file mode 100644 index 0000000000..b00be4860a --- /dev/null +++ b/app/_support/kong-gateway-is-unable-to-create-a-session-after-successfully-authenticating-a-user.md @@ -0,0 +1,40 @@ +--- +title: "{{site.base_gateway}} is unable to create a session after successfully authenticating a user" +content_type: support +description: "Kong Manager or the Dev Portal can fail to create a session, with the browser rejecting the session cookie for an invalid domain, when the host domain is on the Mozilla Public Suffix List. Use a more narrowly scoped domain or private DNS to work around it." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: Public Suffix List + url: https://publicsuffix.org/list/public_suffix_list.dat +tldr: + q: Why is Kong Manager or the Developer Portal rejecting the session cookie after a successful login? + a: | + The browser rejects the session cookie with an invalid-domain error when the host domain (or a suffix of it) is listed on the Mozilla Public Suffix List — a list used to prevent cookies from being scoped to shared, high-level domain suffixes like `*.com` or `com.au`. Work around it by using a more narrowly scoped domain where feasible, or by using private DNS. +--- + +## Problem + +Attempts to login to Kong Manager/Portal are failing, however no message is displayed in the GUI indicating an error. The user ID and password have been confirmed to be valid and only when inspecting the messages in the browser developer tools can a problem be seen. + +Firefox (console tab): You can see that the session cookie is being rejected with the message: `Cookie "" has been rejected for invalid domain` + +Chrome/Edge (network tab): Indicated on the response header by the yellow icon. `This attempt to set a cookie via a Set-Cookie header was blocked because its Domain attribute was invalid with regards to the current host URL.` + +## Solution + +This can occur when you are trying to set a cookie using a domain listed on the Mozilla Public Suffix List. This list is used to limit the scope of the cookie being set to avoid issues like setting a "supercookie" for high level domain suffixes. + +For example, you cannot set a cookie for `*.com` as it would apply to any website hosted on this TLD. + +To correct this issue for second-level domains (i.e., `com.au`) in a unified way, Mozilla created the Public Suffix List. + +This list is used to determine where cookies may and may not be set. Domains on the list are restricted and will result in these errors. + +To resolve this you would need to either + +- Use a more narrowly scoped domain, where feasible +- Use private DNS diff --git a/app/_support/kong-gateway-key-auth-on-consumer-isn-t-working-with-vault-secrets.md b/app/_support/kong-gateway-key-auth-on-consumer-isn-t-working-with-vault-secrets.md new file mode 100644 index 0000000000..8755461a37 --- /dev/null +++ b/app/_support/kong-gateway-key-auth-on-consumer-isn-t-working-with-vault-secrets.md @@ -0,0 +1,32 @@ +--- +title: "{{site.base_gateway}}: Key-Auth on Consumer isn't working with Vault secrets" +content_type: support +description: This is expected behavior at this time as there is no support yet for referenceable fields in the `key-auth` plugin. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why doesn't referencing a Vault secret work for the `key-auth` plugin's `key` field on a Consumer? + a: | + Referenceable fields aren't yet supported in the `key-auth` plugin (tracked as GTWY-I-549), so a Vault reference is taken as a literal string instead of being resolved. + Check whether a field is referenceable by looking for "referenceable" in its attributes, as with the `client_id` field in the `openid-connect` plugin. +--- + +## Problem + +We are trying to reference a Vault secret for the Key Authentication (`key-auth`) plugin field `key` on a Consumer, but it's not working at all and appears to be taking the value as a literal string instead. + +An error may be seen when running a decK sync command for example as below: + +```bash +> deck sync --konnect-runtime-group-name development +Error: reading file: validating file content: 1 errors occurred: +consumers.0.keyauth_credentials.0.key: Invalid type. Expected: string, given: object +``` + +## Solution + +This is expected behavior at this time as there is no support yet for referenceable fields in the `key-auth` plugin. This is a feature request filed as GTWY-I-549 for inclusion in the future. To know which values are referenceable by Vault, the configuration property must include "referenceable" in the attributes of it such as it does on this field in the OpenID Connect (`openid-connect`) plugin for the `client_id` field. We also have a dedicated documentation page which lists all the fields that are referenceable at a glance. diff --git a/app/_support/kong-gateway-kong-manager-login-not-working-in-openshift.md b/app/_support/kong-gateway-kong-manager-login-not-working-in-openshift.md new file mode 100644 index 0000000000..b527be0893 --- /dev/null +++ b/app/_support/kong-gateway-kong-manager-login-not-working-in-openshift.md @@ -0,0 +1,53 @@ +--- +title: "{{site.base_gateway}}: Kong Manager Login Not Working in OpenShift" +content_type: support +description: "In OpenShift, using HTTP/2 with passthrough Routes that share the same SSL certificate across the Kong Manager and Admin API routes causes connection coalescing, which crisscrosses requests between them and breaks Kong Manager login." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: Configuring ingress cluster traffic - Using HTTP/2 (Red Hat OpenShift documentation) + url: https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.5/html/networking/nw-http2-haproxy_configuring-ingress +tldr: + q: Why does Kong Manager login fail with OIDC when running {{site.base_gateway}} in OpenShift? + a: | + In OpenShift, using HTTP/2 on passthrough Routes that share the same SSL certificate for the Kong Manager and Admin API causes HAProxy to coalesce the connections, crisscrossing requests between them. This breaks Manager login, content-type handling, cookies, CORS, and `DELETE` requests. Fix it by giving each Route its own SSL certificate, or by disabling HTTP/2 on passthrough Routes that share a certificate. +--- + +## Problem + +We've recently installed {{site.base_gateway}} with Manager and OIDC Auth enabled in OpenShift, here is what we are seeing during login: + +1) Manager login fails + +2) Unexpected content types being returned from the underlying Admin API requests (`text/html` instead of `application/json`) + +3) Session cookies not being set appropriately + +4) CORS errors + +5) Certain request method types appear to be failing (`DELETE` specifically) + +This same setup works in our non-OpenShift environment. + +## Solution + +Assuming `admin_gui_session_conf` is set appropriately and we are confident in the OIDC configuration (worked elsewhere): + +The issue stems from making use of HTTP/2 with passthrough OpenShift Routes and having the same SSL certificate bound to those routes. + +In OpenShift, if you are doing the above, it will attempt to coalesce the connections and you will effectively see it criss-crossing (sending requests meant for the Admin API to the Manager, and vice versa). + +This leads to the issues stated above and logging in will not function. + +Details of this can be found here: + +"To enable the use of HTTP/2 for the connection from the client to HAProxy, a route must specify a custom certificate. A route that uses the default certificate cannot use HTTP/2. This restriction is necessary to avoid problems from connection coalescing, where the client re-uses a connection for different routes that use the same certificate." + +To remedy the situation, you can take one of two actions: + +1) Ensure that your OpenShift Routes all make use of different SSL certificates + +2) Turn off HTTP/2 when using passthrough OpenShift Routes with the same SSL certificate diff --git a/app/_support/kong-gateway-nginx-configuration-is-invalid.md b/app/_support/kong-gateway-nginx-configuration-is-invalid.md new file mode 100644 index 0000000000..62baf0bec2 --- /dev/null +++ b/app/_support/kong-gateway-nginx-configuration-is-invalid.md @@ -0,0 +1,43 @@ +--- +title: "\"nginx configuration is invalid\" error when starting Kong with a custom template built for a different Kong version" +content_type: support +description: "The `nginx configuration is invalid` error occurs when starting Kong with a custom Nginx template built from a `nginx_kong.lua` of a different Kong version — keep custom templates in sync with the Kong version you're running." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "Custom Nginx templates reference" + url: "/gateway/nginx-directives/#custom-nginx-templates" +tldr: + q: Why do I see an "nginx configuration is invalid" error when starting Kong with a custom template? + a: | + This happens when a custom Nginx template (or its underlying `nginx_kong.lua`) was built for a different Kong version than the one you're running. Keep custom templates in sync with your Kong version when upgrading or downgrading to avoid this error. +--- + +## Problem + +When attempting to start Kong using a custom template the below error is observed and Kong fails to start. + +``` +Error: nginx configuration is invalid (exit code 1): +nginx: [emerg] "return" directive is not allowed here in /usr/local/kong/nginx.conf:18 +nginx: configuration file /usr/local/kong/nginx.conf test failed +``` + +These errors can take similar forms, but may include slightly different wording such as: + +``` +Error: nginx configuration is invalid (exit code 1): +nginx: [emerg] unknown directive "--" in /usr/local/kong/nginx.conf:26 +nginx: configuration file /usr/local/kong/nginx.conf test failed +``` + +## Cause + +This issue can be caused by starting Kong with a custom template that is based on a different version of Kong than the one you are running. For example, you are running {{site.base_gateway}} 3.14.0.0 and using the `nginx_kong.lua` from a 3.4 install to build your custom template. + +## Solution + +As there can be changes between releases, it is important when moving between versions that you ensure your custom templates are up to date. For notes on creating a custom template, see the custom Nginx templates reference. diff --git a/app/_support/kong-gateway-no-matches-for-kind-horizontalpodautoscaler-in-version-autoscaling-v2beta2.md b/app/_support/kong-gateway-no-matches-for-kind-horizontalpodautoscaler-in-version-autoscaling-v2beta2.md new file mode 100644 index 0000000000..5720a9d35a --- /dev/null +++ b/app/_support/kong-gateway-no-matches-for-kind-horizontalpodautoscaler-in-version-autoscaling-v2beta2.md @@ -0,0 +1,70 @@ +--- +title: "`no matches for kind \"HorizontalPodAutoscaler\" in version \"autoscaling/v2beta2\"` error during a Helm upgrade after upgrading Kubernetes to 1.26" +content_type: support +description: "A Helm upgrade of the Kong deployment fails with `no matches for kind \"HorizontalPodAutoscaler\" in version \"autoscaling/v2beta2\"` after upgrading Kubernetes to 1.26, because that API version was removed. Fix it by editing the stored Helm release Secret or ConfigMap to reference `autoscaling/v2` instead." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: "Why does a Helm upgrade of the Kong deployment fail with \"no matches for kind 'HorizontalPodAutoscaler' in version 'autoscaling/v2beta2'\" after upgrading Kubernetes?" + a: | + Kubernetes 1.26 removed the `autoscaling/v2beta2` API, so a Helm release created with an older Kong chart still references it in the stored release Secret or ConfigMap, causing subsequent Helm upgrades to fail. Fix it by decoding the stored release object, replacing `v2beta2` with `v2`, re-encoding it, and patching it back onto the Secret or ConfigMap before retrying the Helm upgrade. +--- + +## Problem + +We have recently upgraded our Kubernetes cluster to 1.26. + +We are now receiving the following error when attempting helm upgrades on the Kong deployment and the upgrade fails: + +``` + +Error: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "kong" namespace: "kong" from "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2" +``` + +## Solution + +This issue occurs because `autoscaling/v2beta2` has been removed as of Kubernetes version 1.26. Possibly due to using an outdated Helm chart to install Kong, your helm release secret/configmap (where the current configuration from your helm deployment is stored) still has an entry for `autoscaling/v2beta2`. + +In order to remedy this we need to take the following steps: + +- Get the name of the Secret or Configmap associated with the latest deployed release: + - Secrets backend: `kubectl get secret -l owner=helm,status=deployed,name= --namespace | awk '{print $1}' | grep -v NAME` + - ConfigMap backend: `kubectl get configmap -l owner=helm,status=deployed,name= --namespace | awk '{print $1}' | grep -v NAME` +- Get latest deployed release details: + - Secrets backend: `kubectl get secret -n -o yaml > release.yaml` + - ConfigMap backend: `kubectl get configmap -n -o yaml > release.yaml` +- Backup the release in case you need to restore if something goes wrong: + - `cp release.yaml release.bak` + - In case of emergency, restore: `kubectl apply -f release.bak -n ` +- Decode the release object: + - Secrets backend: `cat release.yaml | grep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d > release.data.decoded` + - ConfigMap backend: `cat release.yaml | grep -oP '(?<=release: ).*' | base64 -d | gzip -d > release.data.decoded` +- Open the decoded object in a text editor (we recommend Visual Studio Code), CTRL-F for `v2beta2` and change this to `v2` +- Encode the edited release object: + - Secrets backend: `cat release.data.decoded | gzip | base64 | base64` + - ConfigMap backend: `cat release.data.decoded | gzip | base64` +- Create a patch.json file locally with the following fields: + +```json + +{ + "data": { + "release": "YOUR_NEW_ENCODED_RELEASE_DATA" + } +} +``` + +Ensure that there are no whitespaces or line breaks in the encoded release object (many text editors/linux shells will add line breaks) + +- Apply the patch to your secret with the following command: + +```bash + +kubectl patch secret -n --patch-file=patch.json +``` + +You should now be able to helm upgrade normally. diff --git a/app/_support/kong-gateway-oidc-plugin-is-returning-error-invalid-iat-claim-on-login.md b/app/_support/kong-gateway-oidc-plugin-is-returning-error-invalid-iat-claim-on-login.md new file mode 100644 index 0000000000..54fc8923ad --- /dev/null +++ b/app/_support/kong-gateway-oidc-plugin-is-returning-error-invalid-iat-claim-on-login.md @@ -0,0 +1,61 @@ +--- +title: "{{site.base_gateway}}: OIDC plugin is returning error \"invalid iat claim\" on login" +content_type: support +description: This error occurs because the time of the host machine and the IDP are out of sync. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "OpenID Connect plugin reference" + url: "/plugins/openid-connect/" + - text: "Kong Manager OpenID Connect group mapping reference" + url: "/gateway/kong-manager/openid-connect/#set-up-authenticated-group-mapping" +tldr: + q: Why is the OIDC plugin returning an "invalid iat claim" error on login? + a: | + The host machine's clock and the identity provider's clock are out of sync, so the token's issued-at (`iat`) time looks invalid. Add a `leeway` value (for example, `60`) to the OpenID Connect plugin configuration to tolerate the clock difference. +--- + +## Problem + +We have setup OIDC for Kong Manager and are running into the following error: + +``` +2023/04/25 14:46:00 [notice] 2174#0: *2908 [lua] responses.lua:21: [openid-connect] invalid iat claim (1682433961) was specified for id token, client: 172.19.0.1, server: kong, request: "GET /mockbin?code=l5eX_1cw5pSf6COqxsLXUpNO5vmCJBHPJaLdibtQWv4&state=itcXU_mgY2a901EzLnELHANv HTTP/1.1", host: "localhost:8000" +``` + +How can we resolve this? + +## Cause + +This error occurs because the time of the host machine and the IDP are out of sync. Typically, the `id_token` should be issued before it reaches your host machine. + +## Solution + +To allow for this minor time difference between host machine and IDP, we can add a `leeway` field to the OIDC configuration. + +Setting this value to `60` will allow for a difference of 60 seconds and will allow you to resolve the original error. + +Sample configuration: + +```json +{ + "issuer": "{YOUR_IDP_URL}", + "admin_claim": "email", + "client_id": ["{CLIENT_ID}"], + "client_secret": ["{CLIENT_SECRET}"], + "authenticated_groups_claim": ["{CLAIM_NAME}"], + "ssl_verify": false, + "leeway": 60, + "redirect_uri": ["{YOUR_REDIRECT_URI}"], + "login_redirect_uri": ["{YOUR_LOGIN_REDIRECT_URI}"], + "logout_methods": ["GET", "DELETE"], + "logout_query_arg": "logout", + "logout_redirect_uri": ["{YOUR_LOGOUT_REDIRECT_URI}"], + "scopes": ["openid","profile","email","offline_access"], + "auth_methods": ["authorization_code"] + } +``` + diff --git a/app/_support/kong-gateway-oidc-plugin-troubleshooting-high-redis-memory-usage.md b/app/_support/kong-gateway-oidc-plugin-troubleshooting-high-redis-memory-usage.md new file mode 100644 index 0000000000..599b79495f --- /dev/null +++ b/app/_support/kong-gateway-oidc-plugin-troubleshooting-high-redis-memory-usage.md @@ -0,0 +1,55 @@ +--- +title: "{{site.base_gateway}}: OIDC plugin - Troubleshooting High Redis Memory Usage" +content_type: support +description: "Explains why Redis memory usage grows unbounded when the OIDC plugin's bearer and session auth methods create redundant sessions, and how to fix it by tuning `session_rolling_timeout` and disabling sessions for bearer authentication." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does Redis memory usage keep growing when the OIDC plugin has auth code, bearer, and session auth methods enabled together? + a: | + Redundant sessions created by validation requests are the primary cause. Reduce `session_rolling_timeout` to limit how long sessions stay active, and disable sessions for bearer authentication (either by dropping `session` from `auth_methods`, or more surgically by setting `config.disable_session` for the methods that don't need it) to stop the unbounded memory growth. +--- + +## Problem + +I have "auth code", "bearer" and "session" auth methods enabled in the openid-connect plugin, the redis instance backing session storage for the OIDC experienced increasing memory usage until it could not operate. What could be the reasons causing this issue? + +## Cause + +This was primarily due to redundant sessions being created by validation requests. + +## Solution + +To mitigate this issue, the following steps were taken by modifying the OIDC plugin settings: + +1. Adjust the `session_rolling_timeout` parameter. + + Initially, adjusting the `session_rolling_timeout` parameter to a smaller value was recommended to help manage the session lifetime and potentially reduce memory usage. This parameter dictates the duration a session remains active, and reducing its value can help in limiting the number of active sessions stored in Redis. The default value for this setting is 3600 seconds. + + ``` + session_rolling_timeout = 300 # Sets the timeout to 300 seconds (5 minutes) + ``` + +2. Disable sessions for bearer authentication. + + The root cause of the memory issue was identified as redundant sessions created by validation requests. By disabling `session` in `auth_methods`, sessions for bearer token authentication were disabled, effectively eliminating the unbounded memory growth. This solution is particularly useful if sessions are not required for bearer token authentication in your implementation. + + ``` + auth_methods = ["bearer"] + ``` + + A more surgical alternative to fully removing `session` from `auth_methods` is to set `config.disable_session` for the auth methods that don't need it (e.g. `disable_session = ["bearer"]`). This stops the redundant session cookie from being created for those methods while leaving `session` available in `auth_methods` for the methods that do need it. + +3. Disable the `session_store_metadata` setting if it is enabled and not required. + + ``` + session_store_metadata = false + ``` + +It's important to consider the potential consequences of disabling sessions for bearer authentication. Disabling sessions could lead to a loss of session-based functionality if any part of the system relies on session data (e.g., user preferences, shopping carts). Additionally, relying solely on the bearer token for authorization increases the risk if the bearer token is compromised, as an attacker could potentially use it to impersonate the authorized user until the token expires. + +These steps were effective in addressing the issue of increasing memory usage in Redis due to session storage with the OIDC plugin. By adjusting `session_rolling_timeout` and disabling sessions for bearer authentication, the unbounded memory growth was eliminated, allowing the system to operate efficiently. diff --git a/app/_support/kong-gateway-oidc-plugin-with-redis-session-storage-calls-auth-endpoint-irregularly.md b/app/_support/kong-gateway-oidc-plugin-with-redis-session-storage-calls-auth-endpoint-irregularly.md new file mode 100644 index 0000000000..4888ff79cf --- /dev/null +++ b/app/_support/kong-gateway-oidc-plugin-with-redis-session-storage-calls-auth-endpoint-irregularly.md @@ -0,0 +1,27 @@ +--- +title: "{{site.base_gateway}}: OIDC plugin with Redis session storage calls auth endpoint irregularly" +content_type: support +description: "Explains why the OIDC plugin's Redis-backed sessions re-authorize earlier than the Access Token TTL, caused by having both `session_idling_timeout` and `session_rolling_timeout` configured together, and how removing `session_idling_timeout` fixes it." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does the OIDC plugin with Redis session storage call the auth endpoint before the Access Token TTL expires? + a: | + Having both `session_idling_timeout` and `session_rolling_timeout` configured together causes premature re-authorization even though the Access Token TTL hasn't expired. Remove `session_idling_timeout` so only the Access Token TTL governs re-authorization. +--- + +## Problem + +We have {{site.base_gateway}} deployed with OIDC (OpenID Connect) auth and we are using Redis as the session storage type. However, we have noticed that the auth endpoint is repeatedly called without the tokens actually expiring. Our Access Token TTL (time to live) is configured for 5 minutes however the auth requests occur every 2~ minutes. Each call to the test endpoint occurs every 30 seconds. How can we prevent this from occurring so frequently? + +## Cause + +If the Access Token TTL is configured for 5 minutes and its re-authorizing earlier than anticipated then there has to be another configuration in place causing it to re-authorize. The session module's timeout behavior is controlled by three separate settings: `session_idling_timeout` (formerly referred to as "Session Cookie Idletime"), `session_rolling_timeout` (formerly "Session Cookie Lifetime"), and `session_absolute_timeout`, which defaults to 24 hours and caps the overall session lifetime regardless of activity. For example having both `session_idling_timeout` and `session_rolling_timeout` configured will result in this issue. + +## Solution + +To resolve this, we need to remove the `session_idling_timeout` setting and retest. In this scenario after removing `session_idling_timeout` we can see the Access Token TTL is being utilized. Each request to the test endpoint occurs every 30 seconds. Now after 5 minutes a 401 occurs and a reauthorize is needed as expected. diff --git a/app/_support/kong-gateway-openid-plugin-with-okta-receiving-cannot-request-openid-scopes-using-client-credentials.md b/app/_support/kong-gateway-openid-plugin-with-okta-receiving-cannot-request-openid-scopes-using-client-credentials.md new file mode 100644 index 0000000000..d5952e9099 --- /dev/null +++ b/app/_support/kong-gateway-openid-plugin-with-okta-receiving-cannot-request-openid-scopes-using-client-credentials.md @@ -0,0 +1,56 @@ +--- +title: "{{site.base_gateway}}: OpenID plugin with Okta receiving \"Cannot request 'openid' scopes using client credentials.\"" +content_type: support +description: "Some IDPs, such as Okta, don't allow the `openid` scope with the `client_credentials` grant type, causing the OpenID Connect plugin to return an `invalid_scope` error. Fix it by creating a custom scope in the IDP and updating the plugin's scopes to use it instead of `openid`." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does the OpenID Connect plugin return "Cannot request 'openid' scopes using client credentials" with Okta? + a: | + Some IDPs, including Okta, don't allow the `openid` scope with the `client_credentials` grant type, so the OpenID Connect plugin returns an `invalid_scope` error and the request gets an Unauthorized response. Create a custom scope in the IDP and update the plugin's scopes configuration to use that scope instead of `openid`. +--- + +## Problem + +When setting up Client Credentials inside {{site.base_gateway}}'s OpenID Connect plugin we are receiving the following debug statement: + +``` + +[debug] 2149#0: *3771675 [lua] debug.lua:28: debug(): {"error":"invalid_scope","error_description":"Cannot request 'openid' scopes using client credentials."} +``` + +Our curl command results in: + +``` + +{"message":"Unauthorized"} +``` + +Our IDP is Okta in this test case and we can confirm that Authorization flow works as expected so we know the connectivity is successful. + +## Solution + +The issue here is that the scope `openid` is not applicable when using the grant type `Client_Credentials`. Some IDPs may not allow `openid` as a scope in this scenario resulting in the error above. + +The way around this is to create a custom scope inside Okta and update the OpenID Connect plugin to reflect this change as well. + +The default scope configured on the OpenID Connect plugin for `Config.Scopes` is `openid`. For this, we should update the plugin to accommodate the custom scope that was created. + +Once done, we can retest and now should be able to receive 200s. + +Sample Command: + +```bash + +curl --request POST \ + --url http://localhost:8000/test \ + --header 'accept: application/json' \ + --header 'authorization: Basic ' \ + --header 'cache-control: no-cache' \ + --header 'content-type: application/x-www-form-urlencoded' \ + --data 'grant_type=client_credentials&scope=testScope' +``` diff --git a/app/_support/kong-gateway-receiving-error-entity-tags-missing-fields-error-no-namespace-name-openid-connect.md b/app/_support/kong-gateway-receiving-error-entity-tags-missing-fields-error-no-namespace-name-openid-connect.md new file mode 100644 index 0000000000..9c363461db --- /dev/null +++ b/app/_support/kong-gateway-receiving-error-entity-tags-missing-fields-error-no-namespace-name-openid-connect.md @@ -0,0 +1,48 @@ +--- +title: "{{site.base_gateway}}: Receiving error \"entity tags missing fields\" error=\"no namespace\" name=openid-connect" +content_type: support +description: This error occurs typically due to not recognizing the license that has been created. +products: + - gateway + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: "{{site.kic_product_name}} Enterprise license deployment steps" + url: /kubernetes-ingress-controller/license/#applying-a-static-license +tldr: + q: Why does the {{site.kic_product_name}} fail to apply the `openid-connect` plugin with an "enterprise only plugin" error? + a: | + The `openid-connect` plugin requires a valid {{site.ee_product_name}} license, and this error means no valid license is currently loaded on the Gateway/data plane. Check the proxy pod's startup logs for licensing messages, then redeploy the Enterprise license to resolve it. +--- + +## Problem + +We're currently deployed in DB-less on kubernetes. While installing {{site.base_gateway}} with {{site.kic_product_name}} (KIC), we noticed that our ingress-controller is failing to start up properly. + +When reviewing the logs, or the Kubernetes events for the affected resource, we can see a schema violation for the `openid-connect` plugin stating it is an enterprise-only plugin, surfaced through one of two paths: + +As an admission-webhook rejection when applying the resource: + +``` + +Error from server: error when creating "plugin.yaml": admission webhook "validations.kong.konghq.com" denied the request: openid-connect is an enterprise only plugin +``` + +Or as a `KongConfigurationApplyFailed` Warning Event on the resource if the webhook path is bypassed: + +``` + +Warning KongConfigurationApplyFailed ingress-controller openid-connect is an enterprise only plugin +``` + +How can we resolve this? + +## Cause + +This occurs because the `openid-connect` plugin is only available with a valid {{site.ee_product_name}} license, and no license (or no valid license) is currently loaded on the Gateway/data plane. We can verify this by checking the startup logs for the proxy pod for licensing-related messages. + +## Solution + +To resolve this, please redeploy the license and verify the steps. diff --git a/app/_support/kong-gateway-receiving-semver-errors-when-installing-gateway-3-4-0-0-with-helm.md b/app/_support/kong-gateway-receiving-semver-errors-when-installing-gateway-3-4-0-0-with-helm.md new file mode 100644 index 0000000000..19051c1176 --- /dev/null +++ b/app/_support/kong-gateway-receiving-semver-errors-when-installing-gateway-3-4-0-0-with-helm.md @@ -0,0 +1,54 @@ +--- +title: "\"Invalid Semantic Version\" SemVer error when installing or upgrading {{site.base_gateway}} with Helm using an invalid `image.tag` value" +content_type: support +description: "Explains the Helm SemVer validation error (`Invalid Semantic Version`) that occurs when the chart's `image.tag` value doesn't match the required version format for Enterprise or open source {{site.base_gateway}} images." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why do I get a SemVer validation error (`Invalid Semantic Version`) when installing or upgrading {{site.base_gateway}} with Helm? + a: | + The Helm chart validates the `image.tag` value in your values file against Kong's version format — `X.X.X.X` for Enterprise, `X.X.X` for open source. Any `image.tag` outside that format triggers the SemVer error; this isn't specific to any one Gateway version. Set `image.tag` to a correctly formatted version to resolve it. +--- + +## Problem + +When we attempt to install/upgrade {{site.base_gateway}} with an invalid SemVer image tag, we see a SemVer validation error similar to the following: + +``` + +Error: template: kong/templates/deployment.yaml:273:3: executing "kong/templates/deployment.yaml" at : error calling include: template: kong/templates/_helpers.tpl:1637:12: executing "kong.proxy.compatibleReadiness" at : error calling semverCompare: Invalid Semantic Version +``` + +This is not specific to {{site.base_gateway}} 3.4.0.0 — it occurs with any invalid-SemVer `image.tag` value on any current chart version. Note that on current chart versions (e.g. 3.4.1) the failure now typically surfaces much earlier in the template rendering, from the `kong.metaLabels` helper's bare `semver` call (invoked by nearly every templated resource, including the proxy Service) rather than from `kong.proxy.compatibleReadiness`'s `semverCompare` call — so the exact template path, line numbers, and even which Sprig function (`semver` vs `semverCompare`) is named in the error you see may differ from the example above, though the root cause (an `image.tag` that fails Helm's SemVer parsing, reported as `Invalid Semantic Version`) is the same. + +## Cause + +Our helm chart performs several versioning checks against the provided `image.tag` in your values file. Those checks require the following version formats: + +Enterprise Kong: `X.X.X.X` + +Open Source Kong: `X.X.X` + +If your `image.tag` field has anything outside of the above formatting, you will receive the above error. + +## Solution + +Working example: + +```yaml +image: + repository: kong/kong-gateway + tag: 3.4.0.0 +``` + +Broken example: + +```yaml +image: + repository: kong/kong-gateway + tag: randomletters +``` diff --git a/app/_support/kong-gateway-sharing-oidc-session-across-plugin-instances.md b/app/_support/kong-gateway-sharing-oidc-session-across-plugin-instances.md new file mode 100644 index 0000000000..9233d74810 --- /dev/null +++ b/app/_support/kong-gateway-sharing-oidc-session-across-plugin-instances.md @@ -0,0 +1,36 @@ +--- +title: "{{site.base_gateway}}: Sharing OIDC session across plugin instances" +content_type: support +description: This can be done by using the same session secret across multiple plugin instances. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How do I share an OIDC session across multiple scoped plugin instances? + a: | + Configure each scoped OIDC plugin instance with the same issuer and the same `Config.Session Secret`, and enable session management on each. + A session created against one instance is then accepted by the others. Use with caution, as a shared secret can have unintended consequences if misconfigured. +--- + +## Problem + +We have a use case for allowing access to several services using the same session generated by our OIDC plugin. With this being a subset of services using a global plugin is not ideal. How can this be achieved? + +## Solution + +This can be done by using the same session secret across multiple plugin instances. This should only be done with extreme caution and this can lead to unintended consequences if improperly configured. + +To configure this: + +1. Create your first scoped OIDC plugin, i.e: service level. Configure the plugin as needed for your desired auth methods and additionally enable: + - Session Management + - `Config.Session Secret` +2. Configure a new instance of the plugin for the 2nd entity you wish to protect (service/routes/etc): + - Configure the same issuer as for step 1 + - Enable session management, no other auth methods are required + - Configure the same `Config.Session Secret` as done for the first instance. + +You can now consume the first service using the desired auth method and a new session will be created. Upon accessing the second service the same cookie will be passed back to the server and validated allowing access. diff --git a/app/_support/kong-gateway-unsafe-legacy-renegotiation-disabled-error-after-3-4-0-0-upgrade.md b/app/_support/kong-gateway-unsafe-legacy-renegotiation-disabled-error-after-3-4-0-0-upgrade.md new file mode 100644 index 0000000000..2942b2eb82 --- /dev/null +++ b/app/_support/kong-gateway-unsafe-legacy-renegotiation-disabled-error-after-3-4-0-0-upgrade.md @@ -0,0 +1,56 @@ +--- +title: "{{site.base_gateway}}: unsafe legacy renegotiation disabled error after 3.4.0.0 upgrade" +content_type: support +description: In {{site.base_gateway}} 3.4.0.0+ we have bumped OpenSSL to 3.1.1+. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why do I get an "unsafe legacy renegotiation disabled" error after upgrading to {{site.base_gateway}} 3.4.0.0+? + a: | + {{site.base_gateway}} 3.4.0.0+ bumped OpenSSL to 3.1.1+, which rejects TLS renegotiation with upstreams running outdated SSL libraries. + Fix it by updating the upstream's SSL libraries, or as a last resort re-enable legacy renegotiation with `nginx_proxy_proxy_ssl_conf_command = Options +UnsafeLegacyRenegotiation` (this weakens security). +--- + +## Problem + +We recently upgraded to {{site.base_gateway}} 3.4.0.0+, we are now having trouble connecting to certain backends via Kong and we get the following error in our error.log: + +``` +226218 SSL_do_handshake() failed (SSL: error:0A000152:SSL routines::unsafe legacy renegotiation disabled) while SSL handshaking to upstream +``` + +## Cause + +In {{site.base_gateway}} 3.4.0.0+ we have bumped OpenSSL to 3.1.1+. Receiving the above error indicates that your upstream server/backend is utilizing outdated SSL libraries. + +## Solution + +If you are receiving this error there are two ways to remedy the situation: + +1. Upgrading the SSL libraries utilized by the upstream/backend. This will more than likely require a conversation with the application team who owns the backend. + +2. You can reenable legacy renegotiation on Kong using the following environment variable setting. + + Via `kong.conf`/Helm values file env section: + + ```bash + nginx_proxy_proxy_ssl_conf_command = Options +UnsafeLegacyRenegotiation + ``` + + Via docker run (Konnect): + + ```bash + -e "KONG_NGINX_PROXY_PROXY_SSL_CONF_COMMAND=Options +UnsafeLegacyRenegotiation" \ + ``` + + Via docker compose environment section: + + ```bash + KONG_NGINX_PROXY_PROXY_SSL_CONF_COMMAND=Options +UnsafeLegacyRenegotiation + ``` + +Please note, this is not the optimal way of fixing this and can lead to security vulnerabilities. diff --git a/app/_support/kong-gateway-user-roles-are-being-intermittently-removed-updated.md b/app/_support/kong-gateway-user-roles-are-being-intermittently-removed-updated.md new file mode 100644 index 0000000000..4b7668f42f --- /dev/null +++ b/app/_support/kong-gateway-user-roles-are-being-intermittently-removed-updated.md @@ -0,0 +1,25 @@ +--- +title: "{{site.base_gateway}}: User roles are being intermittently removed/updated" +content_type: support +description: This can occur when you manually assign roles, either through Manager or the Admin API, but have also defined an `authenticated_groups_claim` in your `admin_gui_auth_conf`. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why do Kong Manager role assignments get intermittently removed or reset? + a: | + If you manually assign roles through Kong Manager or the Admin API but also configure `authenticated_groups_claim` in `admin_gui_auth_conf`, the group claim mapping takes precedence on each login and overwrites the manual role assignment. Either manage roles entirely through group mapping, or remove `authenticated_groups_claim` to stop the conflict. +related_resources: + - text: Reference + url: /gateway/kong-manager/openid-connect/#set-up-authenticated-group-mapping +--- + +## Problem + +When using OIDC for Kong Manager login, it was noticed that the roles assigned are being intermittently removed/updated. After resetting the role via Manger/Admin API, the problem is fixed but will resurface shortly after. What is causing this issue? + +## Solution + +This can occur when you manually assign roles, either through Manager or the Admin API, but have also defined an `authenticated_groups_claim` in your `admin_gui_auth_conf`. When a user authenticates the group claim will be used to map a role within Kong. If this maps to a non-existent group or has a role with different permissions assigned, it will take precedence over the manual role updates. If you wish to manage roles based on the `authenticated_groups_claim`, you should properly define your roles and refrain from managing the permissions through the Manager/API. If you do not wish to do role mapping, the claim can simply be removed to avoid the conflict. diff --git a/app/_support/kong-ingress-controller-dataplane-synchronizer-could-not-update-kong-admin-creating-consumer-failed-due-existing-acls-entity-references-this-consumers-entity.md b/app/_support/kong-ingress-controller-dataplane-synchronizer-could-not-update-kong-admin-creating-consumer-failed-due-existing-acls-entity-references-this-consumers-entity.md new file mode 100644 index 0000000000..a06f6e9c4e --- /dev/null +++ b/app/_support/kong-ingress-controller-dataplane-synchronizer-could-not-update-kong-admin-creating-consumer-failed-due-existing-acls-entity-references-this-consumers-entity.md @@ -0,0 +1,50 @@ +--- +title: "\"dataplane-synchronizer Could not update kong admin - creating consumer failed due existing 'acls' entity references this 'consumers' entity\" error when deploying the same consumer across multiple workspaces with {{site.kic_product_name}}" +content_type: support +description: "Explains why deploying the same consumer with ACL groups across multiple {{site.kic_product_name}} workspaces fails with a `creating consumer failed due existing 'acls' entity references this 'consumers' entity` error, caused by the `FillIDs` feature gate, and how to disable it." +products: + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: "{{site.kic_product_name}} Feature Gates" + url: /kubernetes-ingress-controller/reference/feature-gates/ + - text: Kubernetes Ingress Controller CHANGELOG + url: https://github.com/Kong/kubernetes-ingress-controller/blob/main/CHANGELOG.md#300 +tldr: + q: Why does creating a consumer with ACL groups in a second {{site.kic_product_name}} workspace fail with `creating consumer failed due existing 'acls' entity references this 'consumers' entity`? + a: | + KIC's `FillIDs` feature gate (enabled by default since KIC 3.0.0) generates UUIDs for entities from their attribute values, so identical consumers in different workspaces get the same UUID and collide when added to ACL groups. Disable both `FillIDs` and `KongCustomEntity` together via the `feature_gates` / `CONTROLLER_FEATURE_GATES` environment variable — `KongCustomEntity` depends on `FillIDs`, so disabling only one causes KIC to crash-loop. +--- + +## Problem + +When deploying the same consumer with ACL groups in multiple workspaces with {{site.kic_product_name}} (KIC), creating the consumer fails with the error `dataplane-synchronizer Could not update kong admin - creating consumer failed due existing 'acls' entity references this 'consumers' entity`. + +## Cause + +The issue you are encountering is due to a feature gate parameter `FillIDs` that was enabled by default starting in KIC 3.0.0. This feature causes KIC to generate UUIDs for entities based on their attribute values. When you attempt to create identical consumers across different workspaces, the same UUID is generated, leading to a conflict when adding the consumer to ACL groups in the second workspace as the UUID is already associated with an ACL group from the first workspace. + +## Solution + +To resolve this issue, you need to disable the `FillIDs` feature by adding an environment variable to your KIC configurations. Note that `FillIDs` must be disabled together with the `KongCustomEntity` feature gate — disabling `FillIDs` alone causes KIC to crash-loop with an explicit dependency error, since `KongCustomEntity` depends on `FillIDs` being enabled. Here are the steps to do so: + +1. Modify your KIC deployment configuration to include the `feature_gates` environment variable with both `FillIDs` and `KongCustomEntity` set to `false`. + +2. Apply the updated configuration to your Kubernetes cluster. + +Here is an example of how to set the environment variable in your KIC configuration: + +```yaml + +env: + - name: CONTROLLER_FEATURE_GATES + value: "FillIDs=false,KongCustomEntity=false" +``` + +After making this change, redeploy your KIC and test the deployment of your consumers across the workspaces again. You should find that KIC now starts and syncs cleanly, and is able to create the consumers and their ACL grouping without encountering the previous error. + +For your reference, more information about feature gates is available in the {{site.kic_product_name}} documentation. + +Additionally, the change log that introduced this feature is available in the GitHub repository for the Kong Kubernetes Ingress Controller. diff --git a/app/_support/kong-ingress-controller-deploying-the-kic-with-an-expired-license.md b/app/_support/kong-ingress-controller-deploying-the-kic-with-an-expired-license.md new file mode 100644 index 0000000000..981195b846 --- /dev/null +++ b/app/_support/kong-ingress-controller-deploying-the-kic-with-an-expired-license.md @@ -0,0 +1,52 @@ +--- +title: "{{site.kic_product_name}}: Deploying the KIC with no Enterprise license" +content_type: support +description: This error occurs when no Enterprise license (not an expired one) is present, so enterprise-only plugins are rejected. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why do enterprise-only plugins like `openid-connect` fail to apply when deploying {{site.kic_product_name}}? + a: | + Enterprise-only plugins are rejected because no Enterprise license is loaded on the Gateway/data plane — this is different from an expired license, which does not block plugin creation. + Redeploy the Enterprise license secret and restart the affected pods to resolve it. +related_resources: + - text: "{{site.ee_product_name}} license secret deployment steps" + url: /kubernetes-ingress-controller/license/#applying-a-static-license +--- + +## Problem + +While installing {{site.base_gateway}} with {{site.kic_product_name}} (KIC) and no Enterprise license present, we noticed that enterprise-only plugins (such as `openid-connect`) fail to apply. + +When reviewing the logs, or the Kubernetes events for the affected resource, we can see a schema violation stating the plugin is enterprise-only, surfaced through one of two paths: + +As an admission-webhook rejection when applying the resource: + +``` + +Error from server: error when creating "plugin.yaml": admission webhook "validations.kong.konghq.com" denied the request: openid-connect is an enterprise only plugin +``` + +Or as a `KongConfigurationApplyFailed` Warning Event on the resource if the webhook path is bypassed: + +``` + +Warning KongConfigurationApplyFailed ingress-controller openid-connect is an enterprise only plugin +``` + +Note: this is specifically an absent-license symptom. An expired (but present) Enterprise license behaves differently — expired licenses do not block enterprise-only plugin creation the way an absent license does. + +## Cause + +This occurs because no Enterprise license is loaded on the Gateway/data plane, so enterprise-only plugins are rejected. + +## Solution + +We can first verify this by checking the startup logs for the proxy pod for licensing-related messages. + +To resolve this, please redeploy the license and verify the steps. + +Note: You will need to restart/delete/deploy the pods following the new license secret to see the effect. diff --git a/app/_support/kong-ingress-controller-ingress-with-a-particular-route-host-name-fails-due-to-invalid-value.md b/app/_support/kong-ingress-controller-ingress-with-a-particular-route-host-name-fails-due-to-invalid-value.md new file mode 100644 index 0000000000..3a5df5a400 --- /dev/null +++ b/app/_support/kong-ingress-controller-ingress-with-a-particular-route-host-name-fails-due-to-invalid-value.md @@ -0,0 +1,64 @@ +--- +title: "{{site.kic_product_name}}: Ingress with a particular route host name fails due to invalid value" +content_type: support +description: "{{site.kic_product_name}} rejects an Ingress `host` value containing an underscore because Kubernetes enforces RFC 1123 hostname rules." +products: + - gateway + - kic +works_on: + - on-prem + - konnect +tldr: + q: "Why does {{site.kic_product_name}} reject an Ingress with a \"spec.rules[0].host: Invalid value\" error?" + a: | + Kubernetes enforces RFC 1123 hostname rules on the Ingress `host` field, which don't allow underscores. Replace any underscore in the hostname with a hyphen so the value is a valid FQDN, and the Ingress will be accepted. +related_resources: [] +--- + +## Problem + +I am using {{site.base_gateway}} on Kubernetes with the {{site.kic_product_name}}. However when I try to add a new route / ingress, I receive the following error: + +``` + +The Ingress "{ingressName}" is invalid: spec.rules[0].host: Invalid value: "{hostName}": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*') +``` + +I'm using the following Ingress values in my deployment when I get the error above: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: example + namespace: example-app + annotations: + konghq.com/protocols: https + konghq.com/https-redirect-status-code: '308' + konghq.com/strip-path: 'true' + konghq.com/preserve-host: 'false' +spec: + ingressClassName: kong + rules: + - host: hostname_api.example.com + paths: + - path: /examplePath + pathType: ImplementationSpecific + backend: + service: + name: example-svc + port: + number: 8080 +``` + +## Cause + +This is expected behavior. The root cause of the error message using the example Ingress declaration is due to the use of an underscore in the `Host` field. It's important to note that this is not a specific error from Kong; rather, it's a validation error provided by the Kubernetes platform as it scrutinizes the Ingress data during deployment. Many platforms beyond Kubernetes adhere to these rules too, as DNS itself does not allow underscores in subdomain host names. + +Per the documented Kubernetes Ingress Spec Rules, the "host" is considered to be a fully-qualified domain name (FQDN), and therefore must comply to FQDN naming rules which includes a specific set of characters that are allowed in a host name and rejecting all other characters from the host name which includes underscores. Included below is a snippet of the relevant RFC for convenience: + +A host identified by a registered name is a sequence of characters usually intended for lookup within a locally defined host or service name registry, though the URI's scheme-specific semantics may require that a specific registry (or fixed name table) be used instead. The most common name registry mechanism is the Domain Name System (DNS). A registered name intended for lookup in the DNS uses the syntax defined in Section 3.5 of [RFC1034] and Section 2.1 of [RFC1123]. Such a name consists of a sequence of domain labels separated by ".", each domain label starting and ending with an alphanumeric character and possibly also containing "-" characters. + +## Solution + +The `Host` value must comply with the RFC as required by the Kubernetes platform. In the example provided, simply change the underscore to a hyphen instead and this will then meet the requirements of RFC 1123. Ultimately, the `Host` value must only consist of lowercase alphanumeric characters, hyphens, and dots per the error message, and anything else will not meet the RFC requirements and will thus be rejected by the Kubernetes platform. diff --git a/app/_support/kong-ingress-controller-is-not-affected-by-ingressnightmare.md b/app/_support/kong-ingress-controller-is-not-affected-by-ingressnightmare.md new file mode 100644 index 0000000000..bba57d667e --- /dev/null +++ b/app/_support/kong-ingress-controller-is-not-affected-by-ingressnightmare.md @@ -0,0 +1,39 @@ +--- +title: "{{site.kic_product_name}} is not affected by IngressNightmare" +content_type: support +description: "{{site.kic_product_name}} (KIC) is not impacted or affected by the vulnerability \"IngressNightmare\"." +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Is {{site.kic_product_name}} affected by the IngressNightmare vulnerability? + a: | + No. IngressNightmare (CVE-2025-24513, CVE-2025-24514, CVE-2025-1097, CVE-2025-1098, CVE-2025-1974) affects Ingress-nginx configurations that construct file paths, which KIC does not do, so KIC is not impacted. +related_resources: + - text: Reference + url: https://kubernetes.io/blog/2025/03/24/ingress-nginx-cve-2025-1974/ +--- + +## Problem + +{{site.kic_product_name}} users need to know whether they are affected by the vulnerability codenamed "IngressNightmare". + +## Solution + +{{site.kic_product_name}} (KIC) is not impacted or affected by the vulnerability "IngressNightmare". + +In Ingress-nginx, some configurations rely on constructing file paths, but KIC does not. As a result, KIC remains unaffected by this vulnerability. + +Vulnerabilities included in IngressNightmare: + +CVE-2025-24513 + +CVE-2025-24514 + +CVE-2025-1097 + +CVE-2025-1098 + +CVE-2025-1974 diff --git a/app/_support/kong-ingress-controller-reports-failed-to-connect-to-reporting-server.md b/app/_support/kong-ingress-controller-reports-failed-to-connect-to-reporting-server.md new file mode 100644 index 0000000000..9b651ae054 --- /dev/null +++ b/app/_support/kong-ingress-controller-reports-failed-to-connect-to-reporting-server.md @@ -0,0 +1,36 @@ +--- +title: "{{site.kic_product_name}} reports \"failed to connect to reporting server\"" +content_type: support +description: "{{site.kic_product_name}}, by default, sends anonymous usage data to help improve Kong." +products: + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does {{site.kic_product_name}} log "failed to connect to reporting server"? + a: | + {{site.kic_product_name}} sends anonymous usage data to `kong-hf.konghq.com` by default, and a firewall blocking that outbound connection produces this error. Set `CONTROLLER_ANONYMOUS_REPORTS` to `false` in the controller configuration to disable the reports and stop the error. +--- + +## Problem + +{{site.kic_product_name}} reports this error in the logs: + +``` +time="2026-03-04T13:14:00Z" level=error msg="failed to connect to reporting server: dial tcp 34.233.69.182:61833: i/o timeout" +``` + +{{site.kic_product_name}}, by default, sends anonymous usage data to help improve Kong, and a firewall may be blocking it from communicating this reporting data to `kong-hf.konghq.com` on port 61833. + +## Solution + +You can disable sending this reporting data by setting `-anonymous-reports` to `false` in the {{site.kic_product_name}} configuration: + +```yaml +- name: CONTROLLER_ANONYMOUS_REPORTS + value: "false" +``` + +More information is available in the {{site.kic_product_name}} environment variables documentation. diff --git a/app/_support/kong-konnect-how-can-we-utilize-a-service-account-no-email-with-oidc-for-konnect-developer-portal.md b/app/_support/kong-konnect-how-can-we-utilize-a-service-account-no-email-with-oidc-for-konnect-developer-portal.md new file mode 100644 index 0000000000..01aa8585f8 --- /dev/null +++ b/app/_support/kong-konnect-how-can-we-utilize-a-service-account-no-email-with-oidc-for-konnect-developer-portal.md @@ -0,0 +1,34 @@ +--- +title: "{{site.konnect_product_name}}: Using a Service Account without an email address with OIDC for the Developer Portal" +content_type: support +description: Remove the `email` scope from the OIDC identity provider configuration on the Konnect Developer Portal so Service Accounts without an email address can log in. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How can we log in to the Konnect Developer Portal with a Service Account (no email) using OIDC? + a: | + Remove the `email` scope from the identity provider configuration under Developer Portal > Settings > Identity > edit Provider > Advanced Settings. This lets accounts without an email address complete OIDC login. +related_resources: [] +--- + +## Problem + +We are utilizing OIDC with a Konnect Developer portal. + +We need to setup a Service Account (an account with no email address). However we're getting the following error when trying to log in. + +``` + +"loginError=oidc+callback+error" +``` + +If we log in with our standard user account (an account with an email), it logs in successfully using OIDC. + +How can we log in with a Service Account? + +## Solution + +In this specific instance when trying to setup a Service Account (no email) we need to make sure that the `email` scope is removed from the Identity tab. To do this we can go to the Developer portal tab -> Settings -> Identity -> "edit Provider". Once there, we can see under Advanced Settings and remove the scope `email` under scopes. Lastly, we need to verify that the user accounts and email accounts can log in successfully. If you continue to receive this error message after this, please reach out to Kong Support and we can look further into it. diff --git a/app/_support/kong-konnect-how-do-i-upload-a-custom-plugin-schema-to-a-konnect-ingress-controller-control-plane.md b/app/_support/kong-konnect-how-do-i-upload-a-custom-plugin-schema-to-a-konnect-ingress-controller-control-plane.md new file mode 100644 index 0000000000..b355501c25 --- /dev/null +++ b/app/_support/kong-konnect-how-do-i-upload-a-custom-plugin-schema-to-a-konnect-ingress-controller-control-plane.md @@ -0,0 +1,43 @@ +--- +title: "{{site.konnect_product_name}}: Uploading a custom plugin schema to a Konnect Ingress Controller Control Plane" +content_type: support +description: Currently the only way to do this is via the Konnect Admin API as the KIC Control Plane GUI is read-only. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: "{{site.konnect_product_name}}: How do I upload a custom plugin schema to a Konnect Ingress Controller Control Plane?" + a: | + The Konnect Ingress Controller Control Plane GUI is read-only, so upload a custom plugin schema through the Konnect Admin API's plugin-schemas endpoint instead. If the plugin is written in a language other than Lua, convert its schema to Lua first, since that's the only format Konnect accepts. +related_resources: + - text: "Konnect API reference: create-plugin-schemas" + url: /api/konnect/control-planes-config/v2/#/Custom%20Plugin%20Schemas/create-plugin-schemas +--- + +## Problem + +When I use a standard Hybrid Control Plane in Konnect, I have the ability to upload my custom plugin schema in the GUI. This functionality appears to not be available when I am using a {{site.kic_product_name}} Control Plane. How do I upload my `schema.lua` file to a Konnect Ingress Controller Control Plane? + +## Solution + +Currently the only way to do this is via the Konnect Admin API as the KIC Control Plane GUI is read-only. In the future this may change for a better user experience. + +Here is a sample POST request to achieve this: + +```bash + +curl --request POST \ + --url https://us.api.konghq.com/v2/control-planes//core-entities/plugin-schemas \ + --header 'Authorization: Bearer ' \ + --header 'Content-Type: application/json' \ + --data '{ + "lua_schema": "return {\n name = \"py-hello\",\n fields = {\n { config = {\n type = \"record\",\n fields = {\n { message = { type = \"string\", required = true } }\n },\n },\n },\n },\n}" +} +' +``` + +Additional considerations: + +If your custom plugin is coded in a language other than LUA (GO, Python, etc), you will need to convert your schema file to LUA for it to be accepted by Konnect diff --git a/app/_support/kong-konnect-how-to-publish-unpublish-an-api-product-using-the-admin-api.md b/app/_support/kong-konnect-how-to-publish-unpublish-an-api-product-using-the-admin-api.md new file mode 100644 index 0000000000..daa7df2c04 --- /dev/null +++ b/app/_support/kong-konnect-how-to-publish-unpublish-an-api-product-using-the-admin-api.md @@ -0,0 +1,55 @@ +--- +title: "{{site.konnect_product_name}}: How to publish/unpublish an API Product using the Admin API" +content_type: support +description: Publish or unpublish an API Product without specifying a version by using the Konnect Admin API's v2 API Products endpoint. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I publish or unpublish an API Product using the Admin API instead of the UI? + a: | + Use the Admin API's `v2/api-products/` endpoint (`PATCH /v2/api-products/`) instead of the UI. + Send `portal_ids` with a portal ID to publish an API Product to the Developer Portal, or send an empty `portal_ids` array to unpublish it. + The v2 endpoint is now Legacy — new integrations should use the Konnect API Catalog's v3 publications endpoint instead. +related_resources: + - text: API Products update-api-product reference + url: /api/konnect/api-products/v2/#/API%20Products/update-api-product +--- + +## Problem + +We are looking for a way to publish/unpublish an API Product without specifying a version. This is possible from the UI, but there doesn't seem to be a way to do it via the Admin API. For example, how can we automate this process using the Admin API? + +## Solution + +It is possible to do from the Admin API as well. To do this we can use the `v2/api-products/` endpoint. + +Note: the v2 API Products endpoint still works but is now marked Legacy, superseded by the Konnect API Catalog's v3 publications endpoint. New integrations should target the v3 API Catalog publications endpoint instead of v2 API Products. + +To add the API Product object to Developer Portal we can use the following command: + +```bash + +curl --request PATCH \ +--url 'https://us.api.konghq.com/v2/api-products/' \ +--header 'Content-Type: application/json' \ +--header 'accept: application/json' \ +--header 'Authorization: Bearer kpat...' \ +--data '{"portal_ids":[""]}' +``` + +To remove it, we just need to remove the portal IDs from the command above. + +Ex: + +```bash + +curl --request PATCH \ +--url 'https://us.api.konghq.com/v2/api-products/' \ +--header 'Content-Type: application/json' \ +--header 'accept: application/json' \ +--header 'Authorization: Bearer kpat...' \ +--data '{"portal_ids":[]}' +``` diff --git a/app/_support/kong-konnect-kic-control-plane-shows-not-fully-operational-after-creating-ingress-objects.md b/app/_support/kong-konnect-kic-control-plane-shows-not-fully-operational-after-creating-ingress-objects.md new file mode 100644 index 0000000000..dad90d48f3 --- /dev/null +++ b/app/_support/kong-konnect-kic-control-plane-shows-not-fully-operational-after-creating-ingress-objects.md @@ -0,0 +1,57 @@ +--- +title: "{{site.konnect_product_name}}: KIC Control Plane shows not \"Fully Operational\" after creating Ingress objects" +content_type: support +description: KIC's "Failed pushing configuration to Konnect" validation error is a blanket message that, in this case, is caused by route/service names generated from Ingress objects exceeding Konnect's character limit. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does the KIC Control Plane show not "Fully Operational" after creating Ingress objects? + a: | + The "validation error" is a blanket message covering any failure applying KIC-generated configuration to the Gateway — commonly a route or service name exceeding Konnect's character limit (originally 128, since raised to 512). + This limitation doesn't apply to on-premises {{site.base_gateway}}. +related_resources: [] +--- + +## Problem + +We recently added a few ingress objects to our K8S cluster and the KIC Control plane now shows it is not "Fully Operational". + +In the KIC logging we see the following: + +``` + +time="2023-09-21T11:48:07-07:00" level=warning msg="Failed pushing configuration to Konnect" error="performing update for https://us.kic.api.konghq.com/kic/api/control-planes/220edfee-f32c-4b79-a1b6-a9f2860ad1e6 failed: 5 errors occurred:\n\twhile processing event: {Create} route jeffys-namespace.getallpizzaconfiguration0-internal-0.getallplatformconfig.getallpizzaconfigurationjeffys-namespace.gateway.cloudtwo.jeffykong.com.80 failed: HTTP status 400 (message: \"validation error\")\n\twhile processing event: {Create} route jeffys-namespace.getpizzaconfiguration0-internal-0.getplatformconfig.getpizzaconfigurationjeffys-namespace.gateway.cloudtwo.jeffykong.com.80 failed: HTTP status 400 (message: \"validation error\")\n\twhile processing event: {Create} route jeffys-namespace.getrestaurantlocation-0-internal-0.getrestaurantlocation.getrestaurantlocation-jeffys-namespace.gateway.cloudtwo.jeffykong.com.80 failed: HTTP status 400 (message: \"validation error\")\n\twhile processing event: {Create} route jeffys-namespace.getallpizzacrustfw-0-internal-0.getallpizzacrustfw.getallpizzacrustfw-jeffys-namespace.gateway.cloudtwo.jeffykong.com.80 failed: HTTP status 400 (message: \"validation error\")\n\twhile processing event: {Create} route jeffys-namespace.getallpizzasides-0-internal-0.getallpizzasides.getallpizzasides-jeffys-namespace.gateway.cloudtwo.jeffykong.com.80 failed: HTTP status 400 (message: \"validation error\")\n" +``` + +## Cause + +This validation error is a blanket message for any type of issue with the application of a KIC generated configuration to the Gateway. In this case, it is due to a maximum character limit that can be applied to route and service names. + +At the time of the above error, the character limit for route and service names in Konnect was 128. This limit has since been raised to 512. This issue may still occur in the future as the concatenation of Kubernetes maximum size Ingress and Service name is approx. 1031 characters. + +## Solution + +A more informational version of the error can be seen by manually attempting to create the route name via the GUI and having Dev Tools open: + +```json + +{ + "code": 3, + "message": "validation error", + "details": [ + { + "@type": "type.googleapis.com/kong.admin.model.v1.ErrorDetail", + "type": "ERROR_TYPE_FIELD", + "field": "name", + "messages": [ + "length must be <= 128, but got 138" + ] + } + ] +} +``` + +NOTE: This limitation does not exist in Kong on-premises Gateway. diff --git a/app/_support/kong-konnect-provider-registry-terraform-io-kong-konnect-required-by-this-configuration-but-no-version-is-selected.md b/app/_support/kong-konnect-provider-registry-terraform-io-kong-konnect-required-by-this-configuration-but-no-version-is-selected.md new file mode 100644 index 0000000000..80776926de --- /dev/null +++ b/app/_support/kong-konnect-provider-registry-terraform-io-kong-konnect-required-by-this-configuration-but-no-version-is-selected.md @@ -0,0 +1,48 @@ +--- +title: '{{site.konnect_product_name}} Terraform provider "no version is selected" error when the provider is not yet initialized' +content_type: support +description: This will occur when you have not yet initialized the Konnect provider. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does Terraform show "no version is selected" for the {{site.konnect_product_name}} provider? + a: | + This happens when the Konnect Terraform provider hasn't been initialized yet. Add the provider block to your configuration file and run `terraform init` to download the required plugin version. +related_resources: [] +--- + +## Problem + +When attempting to use the {{site.konnect_product_name}} Terraform provider I receive the below error when running plan/apply commands. How can this be resolved? + +``` + +│ The following dependency selections recorded in the lock file are inconsistent with the current configuration: +│ - provider registry.terraform.io/hashicorp/konnect: required by this configuration but no version is selected +``` + +## Solution + +This will occur when you have not yet initialized the Konnect provider. Initializing the provider is necessary to download the required plugins and configure the environment. + +Be sure you have the provider added to your configuration file, for example + +```hcl +terraform { + required_providers { + konnect = { + source = "kong/konnect" + version = "0.2.3" + } + } +} +``` + +and initialize using the following command: + +```bash +terraform init +``` diff --git a/app/_support/kong-kubernetes-ingress-controller-enable-disable-migrations-bootstrap-to-run-when-using-helm.md b/app/_support/kong-kubernetes-ingress-controller-enable-disable-migrations-bootstrap-to-run-when-using-helm.md new file mode 100644 index 0000000000..2f468a1ac9 --- /dev/null +++ b/app/_support/kong-kubernetes-ingress-controller-enable-disable-migrations-bootstrap-to-run-when-using-helm.md @@ -0,0 +1,31 @@ +--- +title: "Kong Kubernetes Ingress Controller: Enable/Disable migrations bootstrap to run when using Helm" +content_type: support +published: false +description: The chart supports a parameter, `migrations.init`, which can be used to control spawning the migrations bootstrap job. +products: + - kic +works_on: + - on-prem + - konnect +tldr: + q: How can I enable or disable the migrations bootstrap job when installing Kong with Helm? + a: | + The Helm chart exposes a `migrations.init` parameter that controls whether the migrations bootstrap job runs. Set `migrations.init=false` to skip it; the job runs by default on install. +related_resources: [] +--- + +## Kong Kubernetes Ingress Controller: Enable/Disable migrations bootstrap to run when using Helm + +When deploying Kong with Helm there may be specific use cases where you want control of whether or not the migrations bootstrap job runs. How can this be achieved? + +The chart supports a parameter, `migrations.init`, which can be used to control spawning the migrations bootstrap job. + +For example: + +```bash + +helm upgrade -i kong kong/kong --set env.database=postgres --set migrations.init=false +``` + +Please note that the default behavior is to always run this job on install and generally should not need to be modified. diff --git a/app/_support/kong-log-rotation.md b/app/_support/kong-log-rotation.md new file mode 100644 index 0000000000..2238833632 --- /dev/null +++ b/app/_support/kong-log-rotation.md @@ -0,0 +1,61 @@ +--- +title: Kong Log Rotation +content_type: support +description: While there is no built in mechanism to rotate logs inside of Kong, we suggest using the Linux tool `logrotate`. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: How do I rotate Kong's logs since there's no built-in log rotation? + a: | + Kong doesn't rotate logs internally, so use the Linux `logrotate` utility. Install it, add a config file such as `/etc/logrotate.d/kong` pointing at `/usr/local/kong/logs/*.log`, and set directives like `rotate`, `daily`, and `copytruncate` to control retention and rotation behavior. +related_resources: [] +--- + +## Problem + +The Kong logs (located `/usr/local/kong/logs` by default) can potentially grow rather large over time and may need to be rotated. How can this be done? + +## Solution + +While there is no built in mechanism to rotate logs inside of Kong, we suggest using the Linux tool `logrotate`. + +Here is a guide to get started using this tool. Before suggesting this to customers, please take some time and ensure you know what you are suggesting. + +- Install logrotate + +- Create a Kong specific logrotate config `vim /etc/logrotate.d/kong` + +```bash + +/usr/local/kong/logs/*.log +{ + rotate 500000 + daily + copytruncate + missingok + postrotate + kong reload | true > /dev/null + endscript +} +``` + +Config Explained: + +location of the logs + +{ + +`rotate` - the number of log files to keep + +`daily` - how often to rotate + +`copytruncate` - Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. + +`missingok` - If the log file is missing, go on to the next one without issuing an error message. + +`postrotate`/`endscript` - The lines between postrotate and endscript (both of which must appear on lines by themselves) are executed after the log file is rotated. + +If you want to manually rotate the logs, the following command can be used `logrotate -f /etc/logrotate.d/kong` diff --git a/app/_support/kong-manager-login-fails-with-ssl-handshake-error-ssl3-get-record-wrong-version-number.md b/app/_support/kong-manager-login-fails-with-ssl-handshake-error-ssl3-get-record-wrong-version-number.md new file mode 100644 index 0000000000..0f6f1d00bb --- /dev/null +++ b/app/_support/kong-manager-login-fails-with-ssl-handshake-error-ssl3-get-record-wrong-version-number.md @@ -0,0 +1,48 @@ +--- +title: "Kong Manager login fails with SSL handshake error: \"ssl3_get_record:wrong version number\"" +content_type: support +description: The most common cause of this SSL error is sending HTTPS requests to an endpoint that only listens on HTTP. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does Kong Manager login fail with an SSL handshake error like "ssl3_get_record:wrong version number"? + a: | + This SSL error usually means Kong (or the identity provider) is sending HTTPS requests to an endpoint that only serves plain HTTP — often due to a backend misconfiguration or an incorrect port. + In more complex cases, SRV DNS records with the wrong port can redirect Kong's connection; fix the SRV record, or set `dns_order` to prefer A records over SRV as a workaround. +related_resources: [] +--- + +## Problem + +I get an SSL error when connecting to an Identity Provider for Kong Manager access. Specifically, the error message states "wrong version number". + +Example log entries seen in Kong when this happens: + +``` + +2025/01/12 20:16:52 [crit] 40#0: *1477 SSL_do_handshake() failed (SSL: error:0A0000C6:SSL routines::packet length too long error:0A000139:SSL routines::record layer failure), client: 10.239.48.220, server: kong_admin, request: "GET /auth HTTP/1.1", host: "https://openidconnect.example.com/", referrer: "https://kongmanager.example.com/" + +2025/01/12 20:16:52 [notice] 40#0: *1477 [lua] cache.lua:258: discover(): [openid-connect] loading configuration for https://openidconnect.example.com/auth/realms/kong_management/.well-known/openid-configuration using discovery failed: handshake failed (falling back to previous configuration), client: , server: kong_admin, request: "GET /auth HTTP/1.1", host: "https://openidconnect.example.com/", referrer: "https://kongmanager.example.com/" +``` + +## Cause + +Note: on current {{site.base_gateway}} 3.14.0.0 (modern bundled OpenSSL), this exact scenario — an `https://` issuer/endpoint that actually only serves plain HTTP — is live-reproducible, but the literal OpenSSL error text has changed from the classic `ssl3_get_record:wrong version number` wording to `SSL_do_handshake() failed (SSL: error:0A0000C6:SSL routines::packet length too long error:0A000139:SSL routines::record layer failure)`, confirmed via `kong-dp`'s error log when `openid-connect`'s `issuer` points at a plaintext-HTTP listener. Older Kong/OpenSSL versions may still show the older string. Either way, the underlying cause and fix below are unchanged. + +The most common cause of this SSL error is sending HTTPS requests to an endpoint that only listens on HTTP. This usually results from a misconfiguration on the backend or Kong using the wrong port number. Correcting this typically resolves the SSL error. + +In more complex cases, the issue may stem from the environment serving SRV DNS records with incorrect port numbers, causing Kong to connect to a different port than originally configured. + +By default, Kong prefers SRV records over A records for DNS resolution. In most environments, SRV records do not exist, so Kong uses A records and the configured port remains effective. However, if SRV records are present and incorrect, Kong may be directed to use the wrong port. + +For example, if Kong is configured to use port 443 for HTTPS but the SRV record specifies port 80, Kong will send HTTPS requests to port 80, causing the SSL error. + +## Solution + +The solution is to correct the SRV record to specify the proper port. Since DNS records must be accurate, if SRV records are managed automatically and not maintained by your team, a workaround is to configure Kong to prefer A records over SRV records by setting the `dns_order` property. For more details, see the Kong documentation on the `dns_order` property. + +1. Add `dns_order` to the Kong configuration with a value of LAST,A,CNAME,SRV for example. +2. Restart or redeploy Kong to apply the change. diff --git a/app/_support/kong-manager-or-other-service-is-in-a-redirect-loop-after-openid-connect-integration.md b/app/_support/kong-manager-or-other-service-is-in-a-redirect-loop-after-openid-connect-integration.md new file mode 100644 index 0000000000..5c6557940c --- /dev/null +++ b/app/_support/kong-manager-or-other-service-is-in-a-redirect-loop-after-openid-connect-integration.md @@ -0,0 +1,33 @@ +--- +title: Kong Manager or other Service is in a redirect loop after OpenID Connect integration +content_type: support +description: There are a number of possible causes for this behavior where a redirect loop is seen when using the OpenID-Connect (OIDC) plugin for authentication. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does Kong Manager (or another service) get stuck in a redirect loop after integrating with the OpenID Connect plugin? + a: | + A post-OIDC redirect loop usually has one of three causes: the `SameSite` cookie attribute set to `Strict` instead of `Lax`, a missing or incorrect `redirect_uri`, or an unreachable Redis server used for session storage. Set `cookie_samesite=Lax`, confirm `redirect_uri` matches the environment, and verify Redis connectivity to resolve it. +--- + +## Problem + +I configured Kong Manager for access via the OpenID Connect plugin, however now Kong Manager logins result in a redirect loop. Why is this loop happening and how do we resolve it? My web browser terminates after around a dozen redirects, declaring that there were too many redirects or stating that `This webpage has a redirect loop`, and Chrome errors out with a `ERR_TOO_MANY_REDIRECTS` message. + +## Solution + +There are a number of possible causes for this behavior where a redirect loop is seen when using the OpenID-Connect (OIDC) plugin for authentication. The three most popular are below for reference: + +1. `SameSite` cookie attribute is set to `Strict` when it may need to be `Lax` for the environment. +2. The `redirect_uri` variable is either missing or incorrectly set to the wrong endpoint. +3. Redis server used for session storage is inaccessible + +For situation #1 above with the `SameSite` cookie attribute, this may need to set this to `Lax` to see it work again in the environment. We recommend customers familiarize themselves with the `SameSite` attribute and then when ready to make the change it can be done by setting `cookie_samesite=Lax` as referenced in the Kong OIDC plugin documentation. + +For situation #2 above with the `redirect_uri` variable, it must be set according to the needs of the environment. If it's missing or hitting the wrong endpoint, it may not behave as expected and can cause the redirect loop. Admins should ensure the `redirect_uri` is set correctly based upon the environmental requirements. + +For situation #3 above with redis servers used for session storage, ensure that your redis configuration in the OIDC plugin is set correctly. This includes the correct hostname, port number, and more. If a port is misconfigured for example, an error will start `Failed to connect to redis-cluster port 9000`. If a hostname is misconfigured, an error should be logged of `Could not resolve host: redis-cluster`. A good test is to run `curl :` in a command line from the {{site.base_gateway}} container as well to verify connectivity. If the server responds correctly, you should see `Empty reply from server` in the curl response output. diff --git a/app/_support/kong-mesh-error-admission-webhook-validator-kuma-admission-kuma-io-denied-the-request-access-denied-user-gruber-system-masters-system-authenticated-cannot-access-the-resource.md b/app/_support/kong-mesh-error-admission-webhook-validator-kuma-admission-kuma-io-denied-the-request-access-denied-user-gruber-system-masters-system-authenticated-cannot-access-the-resource.md new file mode 100644 index 0000000000..2753d9c251 --- /dev/null +++ b/app/_support/kong-mesh-error-admission-webhook-validator-kuma-admission-kuma-io-denied-the-request-access-denied-user-gruber-system-masters-system-authenticated-cannot-access-the-resource.md @@ -0,0 +1,122 @@ +--- +title: "{{site.mesh_product_name}}: 'admission webhook \"validator.kuma-admission.kuma.io\" denied the request: access denied' error when the default AccessRole or AccessRoleBinding is missing or too restrictive" +content_type: support +description: "{{site.mesh_product_name}} denies valid users from creating resources when the default `AccessRole` and `AccessRoleBinding` are missing or too restrictive; recreating them and the validating webhook resolves the access-denied error." +products: + - mesh +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: "Why does creating a {{site.mesh_product_name}} resource fail with an 'admission webhook ... denied the request: access denied' error even though my user has access?" + a: | + This happens when the default `AccessRole` and `AccessRoleBinding` have been modified too restrictively or don't exist, so the `kuma-admission` webhook rejects the request even for users with sufficient Kubernetes RBAC access. Back up and delete the `kong-mesh-validating-webhook-configuration` webhook, re-create the default `AccessRole`/`AccessRoleBinding`, then re-apply the webhook to restore access. +--- + +## Problem + +When attempting to create a new resource we are receiving the below error despite my user being part of the `system:masters` & `system:authenticated` groups. + +``` +Error from server: error when creating "mesh.yaml": admission webhook "validator.kuma-admission.kuma.io" denied the request: access denied: user "arn:aws:iam::12345:role/NonProd_ServiceMesh_DevUsers/system:masters,system:authenticated" cannot access the resource +``` + +I have also confirmed my user has access to creating the resource, in this case, a new mesh. What is causing this? + +```bash +kubectl auth can-i create mesh +yes +``` + +## Solution + +This issue can occur when the default `AccessRole` and `AccessRoleBinding` have either been modified in an overly restrictive manner or do not exist. + +If you do not have another user with a different `AccessRole`/`AccessRoleBinding` associated you will need to modify the webhook to unblock yourself. + +Note: As always, it is important to test this process in lower environments before attempting in production. + +1. IMPORTANT: Backup the existing webhook config before proceeding + + ```bash + kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io kong-mesh-validating-webhook-configuration -oyaml > webhook.yaml + ``` + +2. Delete the webhook + + ```bash + k delete -f webhook.yaml + ``` + +3. Re-create the `AccessRole` and/or `AccessRoleBinding` (depending on what is missing) + + AccessRole + + ```bash + echo " + apiVersion: v1 + items: + - apiVersion: kuma.io/v1alpha1 + kind: AccessRole + metadata: + name: admin + spec: + rules: + - access: + - CREATE + - UPDATE + - DELETE + - GENERATE_DATAPLANE_TOKEN + - GENERATE_USER_TOKEN + - GENERATE_ZONE_CP_TOKEN + - GENERATE_ZONE_TOKEN + - VIEW_CONFIG_DUMP + - VIEW_STATS + - VIEW_CLUSTERS + kind: List + metadata: + resourceVersion: '' + " | kubectl apply -f - + ``` + + AccessRoleBinding + + ```bash + echo " + apiVersion: v1 + items: + - apiVersion: kuma.io/v1alpha1 + kind: AccessRoleBinding + metadata: + name: default + spec: + roles: + - admin + subjects: + - name: mesh-system:authenticated + type: Group + - name: mesh-system:unauthenticated + type: Group + - name: system:authenticated + type: Group + - name: system:unauthenticated + type: Group + kind: List + metadata: + resourceVersion: '' + " | kubectl apply -f - + ``` + +4. Re-create the webhook + + ```bash + kubectl apply -f webhook.yaml + ``` + +5. Confirm you can now remove resources + + ```bash + kubectl delete mesh gruber + mesh.kuma.io "gruber" deleted + ``` diff --git a/app/_support/kong-mesh-observability-tracing-troubleshooting-steps.md b/app/_support/kong-mesh-observability-tracing-troubleshooting-steps.md new file mode 100644 index 0000000000..6e20cea76c --- /dev/null +++ b/app/_support/kong-mesh-observability-tracing-troubleshooting-steps.md @@ -0,0 +1,83 @@ +--- +title: "{{site.mesh_product_name}}: Observability/tracing troubleshooting steps" +content_type: support +description: General troubleshooting steps for diagnosing why {{site.mesh_product_name}} Observability/tracing isn't collecting the expected data, using the `kuma-demo` app and Envoy config dumps. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: What are common pitfalls when troubleshooting {{site.mesh_product_name}} Observability/tracing, and how do I diagnose them? + a: | + If tracing isn't collecting the expected data, gather the Envoy config dump, {{site.mesh_product_name}} debug logs, the Mesh resource, and the TrafficTrace (or MeshTrace) policy, then verify the tracing backend and policy are configured correctly and check the logs for errors. + In the config dump, confirm the expected listeners are present — if the protocol isn't explicitly configured per service (HTTP, HTTP2, or gRPC), jaeger will only show up under `config.bootstrap.cluster.name` and not as a listener. +related_resources: + - text: kuma-demo app deployment manifest + url: https://raw.githubusercontent.com/Kong/kuma-demo/master/kubernetes/kuma-demo-aio.yaml + - text: Kuma observability guide + url: https://kuma.io/docs/latest/explore/observability/ + - text: Kuma TrafficTrace policy - Zipkin backend configuration + url: https://kuma.io/docs/latest/policies/traffic-trace/#zipkin + - text: Kuma TrafficTrace policy - adding a TrafficTrace resource + url: https://kuma.io/docs/latest/policies/traffic-trace/#add-traffictrace-resource +--- + +## Problem + +When deploying Observability on {{site.mesh_product_name}} you may run into a situation where tracing is not gathering data as you expected. What are some common pitfalls and ways to troubleshoot this further? + +## Solution + +Note: `TrafficTrace`, referenced below, is deprecated in favor of `MeshTrace`. For new setups, use `MeshTrace` instead. The troubleshooting technique described here remains valid for either policy. + +Prerequisites: + +We will be referencing the `kuma-demo` app. + +We will also assume you've followed this guide. + +Files to gather: + +1. Config Dump from envoy sidecar for app that is communicating with Jaeger (in the demo app we can use the frontend): + + ```bash + kubectl port-forward svc/frontend > -n kuma-demo 9901:9901 2>1 & + ``` + + Access the UI through localhost:9901 and gather the `config_dump`. + +2. {{site.mesh_product_name}} debug logs + +3. Mesh deployment file + + ```bash + k get mesh -o yaml > meshdata.yaml + ``` + +4. Traffic Trace Policy + + ```bash + k get traffictrace -o yaml > TTData.yaml + ``` + +Troubleshooting steps: + +1. Verify Tracing backend is configured correctly +2. Verify Traffic Trace policy is configured correctly +3. Verify any errors in the logs. + +Debugging the config dump can be intimidating as there is a lot of different data to review. One key thing to look for would be to verify if the expected Listeners are deployed. + +Example: + +Tracing is only supported over HTTP, HTTP2, gRPC protocols. So you must explicitly specify the protocol for each service. + +If the protocol is not configured, then the config dump will only reference jaeger under `config.bootstrap.cluster.name`. + +``` + + "name": "tracing:jaeger-collector" +``` + +In a working scenario, jaeger would be referenced as listeners as well. diff --git a/app/_support/kong-mesh-receiving-error-license-status-is-not-valid-for-this-control-plane-copy-configmap-license-quota-status.md b/app/_support/kong-mesh-receiving-error-license-status-is-not-valid-for-this-control-plane-copy-configmap-license-quota-status.md new file mode 100644 index 0000000000..ce15519c5f --- /dev/null +++ b/app/_support/kong-mesh-receiving-error-license-status-is-not-valid-for-this-control-plane-copy-configmap-license-quota-status.md @@ -0,0 +1,55 @@ +--- +title: "{{site.mesh_product_name}}: Receiving error \"license status is not valid for this control plane. Copy ConfigMap license-quota-status\"" +content_type: support +description: A {{site.mesh_product_name}} Zone Control Plane can lose license validity when the `license-quota-status` ConfigMap is out of sync with the Global CP; deleting the ConfigMap and letting it rebuild resolves the error. +products: + - mesh +works_on: + - on-prem + - konnect +related_resources: + - text: Reference + url: /mesh/control-plane-configuration/#memory +tldr: + q: "Why does deploying a Data Plane fail with \"license status is not valid for this control plane\" in a multi-zone {{site.mesh_product_name}} setup?" + a: | + This happens when the `license-quota-status` ConfigMap on the Zone Control Plane is out of sync with the Global CP. Delete the ConfigMap on the Zone CP and redeploy (for example, via `helm upgrade`) so it rebuilds and picks up a valid license. On Universal mode, back the Zone CP with a database instead of Memory storage, since Memory storage doesn't persist the license state. +--- + +## Problem + +We have a multi-zone mesh setup and when deploying a Data Plane we received the following error message on the Zone Control Plane: + +``` +Warning FailedToGenerateKumaDataplane pod/redis-m30k3nks3-93kd3 Failed to generate Kuma Dataplane: admission webhook "validator.kuma-admission.kuma.io" denied the request: license status is not valid for this control plane. Copy ConfigMap license-quota-status from Global to Zone to restore functionality of the system. +``` + +This is preventing us from deploying and utilizing our data plane. + +## Solution + +To correct this you can have the Zone CP rebuild this configmap. You will need to first delete the original configmap using the following command on the Zone CP: + +```bash +kubectl delete configmap -n +``` + +Afterwards, redeploy the Zone CP and the configmap will be rebuilt. For example, if helm was used to deploy: + +```bash +helm upgrade -n kong-mesh-system --values values.yaml kong-mesh kong-mesh/kong-mesh +``` + +We recommend validating that the license itself is correct on both the Global CP and the Zone CP after the deployment. To validate that you can run the following command: + +```bash +kubectl get configmap license-quota-status -o yaml -n +``` + +The output should be identical when run on the Global and Zone CP. Once the configmap is rebuilt the issue should be resolved. + +If this occurs on Universal, please verify if you are running on Memory or backed by a database. + +If this occurs while running on Memory please deploy out a database to resolve this issue. Memory data storage is intended for demo purposes. + +Each Zone CP is intended to be backed by a database in universal. diff --git a/app/_support/kong-mesh-unable-to-generate-certificate-using-vault-policy-error-permission-denied-use-token-that-allows-to-generate-cert.md b/app/_support/kong-mesh-unable-to-generate-certificate-using-vault-policy-error-permission-denied-use-token-that-allows-to-generate-cert.md new file mode 100644 index 0000000000..d6e7a161ab --- /dev/null +++ b/app/_support/kong-mesh-unable-to-generate-certificate-using-vault-policy-error-permission-denied-use-token-that-allows-to-generate-cert.md @@ -0,0 +1,38 @@ +--- +title: "{{site.mesh_product_name}}: Unable to generate certificate using Vault policy, error: \"permission denied - use token that allows to generate cert\"" +content_type: support +description: "{{site.mesh_product_name}} can fail to generate certificates through Vault Policy when a Hashicorp Vault child token's parent token has a shorter TTL, causing `permission denied` errors; extending the parent TTL or switching to an orphan token resolves it." +products: + - mesh +works_on: + - on-prem + - konnect +related_resources: + - text: Hashicorp Vault documentation on token hierarchies and orphan tokens + url: https://developer.hashicorp.com/vault/docs/concepts/tokens#token-hierarchies-and-orphan-tokens +tldr: + q: "Why does {{site.mesh_product_name}} fail to generate a certificate through Vault Policy with a \"permission denied - use token that allows to generate cert\" error?" + a: | + This typically happens when the Hashicorp Vault token {{site.mesh_product_name}} uses is a `child token` whose `parent token` has a shorter TTL — Vault's token hierarchy lets the parent's expiry override the child's, so the child stops working sooner than expected. Fix it by giving the parent token a longer TTL and making the child token `renewable`, or by switching to an `orphan token`, which has no max TTL and lives indefinitely as long as it's renewed. +--- + +## Problem + +We're trying to set up {{site.mesh_product_name}} with the Vault Policy for integration with Hashicorp Vault but we are receiving errors after a period of time. It's as if the Hashicorp token is no longer valid and thus it's unable to generate the certificate. We see this error in the {{site.mesh_product_name}} logs: + +``` +2023-08-08T17:21:29.041Z ERROR xds-server.dataplane-sync-watchdog OnTick() failed {"dataplaneKey": {"Mesh":"{nameOfMeshCluster}","Name":"cluster-mgi-58cf779d4c-tntjr.kong-mesh-system"}, "error": "failed to generate a snapshot: imports[0]{name=\"gateway-proxy\"}: secrets.Generator failed: failed to generate dataplane identity cert and CAs: could not generate certificates: could not get Dataplane cert pair: could not generate dataplane cert for mesh: \"nonprod\" backend: \"vault-1\" services: \"kuma.io/service={ServiceName} kuma.io/zone={ServiceName}\": permission denied - use token that allows to generate cert of pki-internal/issue/np-kong-mesh", "errorVerbose": "imports[0]{name=\"gateway-proxy\"}: secrets.Generator failed: failed to generate dataplane identity cert and CAs: could not generate certificates: could not get Dataplane cert pair: could not generate dataplane cert for mesh: \"{nameOfMeshCluster}\" backend: \"vault\" services: \"kuma.io/service={ServiceName} kuma.io/zone={ServiceName}\": permission denied - use token that allows to generate cert of pki-internal/issue/np-kong-mesh\nfailed to generate a snapshot\ngithub.com/kumahq/kuma/pkg/xds/server/v3.(*reconciler).Reconcile\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/server/v3/reconcile.go:59\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).syncDataplane\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/sync/dataplane_watchdog.go:130\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).Sync\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/sync/dataplane_watchdog.go:66\ngithub.com/kumahq/kuma/pkg/xds/sync.(*dataplaneWatchdogFactory).New.func2\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/sync/dataplane_watchdog_factory.go:46\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).onTick\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/util/watchdog/watchdog.go:54\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/util/watchdog/watchdog.go:27\nruntime.goexit\n\truntime/asm_amd64.s:1598"} +``` + +Why is this happening and what is the solution? Why does this token seem to expire so quickly? + +## Cause + +When using Vault Policy in {{site.mesh_product_name}} with Hashicorp, there are circumstances in which the ability for {{site.mesh_product_name}} to generate a certificate will fail due to the token expiring or being marked as invalid. An invalid token will lead to an HTTP 403 response from Hashicorp back to {{site.mesh_product_name}}, and the error `permission denied - use token that allows to generate cert` is written to the {{site.mesh_product_name}} logs. The most common situation this occurs in is when the token being used is a `child token` with an expiry of 30 days, for example, but the `parent token` has an expiry of a lower value such as 24 hours. The way Hashicorp has designed their token hierarchy, the parent token overrides the child token when it comes to the TTL / expiry of the token. + +## Solution + +In this situation, there are two solutions that can be used: + +1. Modify the parent token to have a longer expiry TTL than the child token being used. Also ensure the child token is set as `renewable`. +2. Switch to using an `orphan token` instead of a child token. The orphan token concept in Hashicorp means there is no max TTL, and as long as the token is renewed before the TTL, it will live indefinitely. diff --git a/app/_support/kong-proxy-latency-exceeds-timeout-limit-specified-on-service.md b/app/_support/kong-proxy-latency-exceeds-timeout-limit-specified-on-service.md new file mode 100644 index 0000000000..b38f0e6aae --- /dev/null +++ b/app/_support/kong-proxy-latency-exceeds-timeout-limit-specified-on-service.md @@ -0,0 +1,43 @@ +--- +title: "`x-kong-proxy-latency` header exceeds the timeout limit specified on the service" +content_type: support +description: Kong's `x-kong-proxy-latency` header keeps climbing beyond the configured timeout because Kong retries a slow upstream according to the service's `retries` setting, adding each retry's latency to the total. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does the `x-kong-proxy-latency` header exceed the timeout configured on my service? + a: | + Kong retries a slow upstream according to the service's `retries` setting when the initial attempt exceeds the configured timeout, and each retry adds to the total latency reported in `x-kong-proxy-latency`. This is expected behavior — review your `retries` and timeout settings to match your use case. +--- + +## Problem + +Our services upstream is set to 60000ms (60 seconds). If our service delays for 70 seconds, we notice that Kong returns a 504 after about 5 minutes from 1 request. What is causing this behavior and how do we get Kong to time out after 60 seconds like it is specified? + +Example Configuration: + +`x-kong-proxy-latency` records about 5 minutes worth of delays. + +``` +x-kong-proxy-latency: 300007 +``` + +## Cause + +If Kong can't access the upstream in the allotted time frame (in this case 60000ms), it will retry the request based on the `retries` configuration on the service. + +If `retries` is set to 5, then it will attempt to access the upstream 5 more times, causing the header value for `x-kong-proxy-latency` to increase on each retry. + +For example, if `retries` is set to 1, you'll get the following: + +``` +x-kong-proxy-latency: 60006 +``` + +## Solution + +This is expected behavior. To tune your settings, discuss internally to determine if your timeout and `retries` are set correctly for your use case. diff --git a/app/_support/kong-worker-processes-exited-on-signal-9.md b/app/_support/kong-worker-processes-exited-on-signal-9.md new file mode 100644 index 0000000000..cca0d23911 --- /dev/null +++ b/app/_support/kong-worker-processes-exited-on-signal-9.md @@ -0,0 +1,28 @@ +--- +title: Kong worker processes exited on signal 9 +content_type: support +description: This issue can be caused when Kong config parameter `nginx_worker_processes` is set to `auto`, which means it depends on the amount of cores of your nodes. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why are Kong worker processes exiting on signal 9 under high load? + a: | + `nginx_worker_processes` set to `auto` spawns one worker per host vCPU, which is often more workers than needed and can lead to instability under load. + Reduce the worker count to a smaller fixed number (for example, 2 or 4) and scale horizontally to handle additional traffic, and consider increasing the memory assigned to the Kong pods. +--- + +## Problem + +You found that worker processes in Kong are exiting on signal 9. This seems to be happening when there is a high load on Kong. The log messages are similar to:[notice] 1#0: signal 17 (SIGCHLD) received from 2420[alert] 1#0: worker process 2420 exited on signal 9[notice] 1#0: start worker process 2428[notice] 1#0: signal 29 (SIGIO) received[warn] 2428#0: found and cleared 1 stale readers from LMDB[notice] 1#0: signal 17 (SIGCHLD) received from 2421[alert] 1#0: worker process 2421 exited on signal 9[notice] 1#0: start worker process 2429[notice] 1#0: signal 29 (SIGIO) received[warn] 2429#0: found and cleared 1 stale readers from LMDB[notice] 1#0: signal 17 (SIGCHLD) received from 2422[alert] 1#0: worker process 2422 exited on signal 9[notice] 1#0: start worker process 2430[notice] 1#0: signal 29 (SIGIO) received + +## Cause + +This issue can be caused when Kong config parameter `nginx_worker_processes` is set to `auto`, which means it depends on the amount of cores of your nodes. If you have 8 cores, you are using 8 workers. It could happen that not all of those 8 workers are closing, only one of them closes, especially during load. + +## Solution + +Having as many Nginx worker processes as the underlying host has vCPUs is typically too many. You can try reducing this to a number that better resembles the assigned resources. For example you can start setting the number of workers to something small like 2 or 4, and horizontal scaling to deal with additional traffic. This may already help with memory as every worker requires some memory although things like cache are shared but it may also be worth increasing the memory assigned to the Kong pods, and checking if that prevents the issue from happening.Related documentation: Cluster resource allocations diff --git a/app/_support/kong-worker-processes-getting-killed-due-to-memory-issues.md b/app/_support/kong-worker-processes-getting-killed-due-to-memory-issues.md new file mode 100644 index 0000000000..379a52601a --- /dev/null +++ b/app/_support/kong-worker-processes-getting-killed-due-to-memory-issues.md @@ -0,0 +1,45 @@ +--- +title: kong worker processes getting killed due to memory issues (after moving to a new host) +content_type: support +description: A common reason why the OOMKiller may kick in on a kong node is due to the configuration of the number of worker processes in kong. +products: + - gateway + - kic +works_on: + - on-prem + - konnect +related_resources: + - text: the configuration option for the workers + url: /gateway/configuration/#nginx-worker-processes +tldr: + q: Why are Kong worker processes getting killed due to memory issues (OOMKilled)? + a: | + Kong's default `nginx_worker_processes` setting of `auto` starts one worker process per host vCPU, which can use more memory than the host has available — especially after moving to a smaller host — triggering the OOM killer (SIGCHLD, OOMKilled). + Set `nginx_worker_processes` to a fixed, smaller number (2 or 4 is often enough) instead of `auto` to match your host's resources. +--- + +## Problem + +Kong nodes are being terminated, resulting in a signal 17 (SIGCHLD) log message in the kong error log and possibly log entries like `failed while spawning "worker process" (12: Out of memory)`. When using kubernetes, the relevant pod shows a state of terminated with reason: OOMKilled. + +## Cause + +A common reason why the OOMKiller may kick in on a kong node is due to the configuration of the number of worker processes in kong. + +## Solution + +There is a configuration option for the workers that is documented here. + +The default setting for this configuration option is `auto`, which means that Kong will start as many nginx worker processes as the underlying host has vCPUs. This is typically too many, and will cause an increase in memory resource utilization, as each worker has its own memory overhead. If you start seeing this issue after moving the Kong infrastructure to a new set of hosts, it is likely that this is causing the memory issue. + +To address the issue, reduce this to a number of workers configured in kong to something that better resembles the assigned resources. + +The current {{site.kic_product_name}} Helm chart sets `nginx_worker_processes` to 2 by default, and it is expected to use horizontal scaling to deal with additional traffic. + +However, if you are using the `auto` setting, you should set the number of workers to something small like 2 or 4. + +If you are using the kong Helm charts you can change the number of workers in the env section + +E.g.: + +`nginx_worker_processes: "2"` diff --git a/app/_support/kong-zipkin-plugin.md b/app/_support/kong-zipkin-plugin.md new file mode 100644 index 0000000000..e1b5104477 --- /dev/null +++ b/app/_support/kong-zipkin-plugin.md @@ -0,0 +1,46 @@ +--- +title: Jaeger protocol support in the Kong Zipkin plugin +content_type: support +published: false +description: The Zipkin plugin supports various open tracing protocols including Jaeger support. +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Does the Kong Zipkin plugin support the Jaeger protocol, and if so, how is it configured? + a: | + Yes. Set `default_header_type` to `b3` and point `http_endpoint` at your Jaeger collector's `/api/v1/spans` endpoint. + `header_type` and `default_header_type` are deprecated in favor of `config.propagation`, but remain functional and auto-populate the newer field. +related_resources: [] +--- + +## Jaeger support for Kong Zipkin Plugin + +Does the Kong Zipkin Plugin support the Jaeger protocol and if so how is this configured? + +The Zipkin plugin supports various open tracing protocols including Jaeger support. + +Below is an example config. Ensure the `default_header_type` is set to `b3`. Also, ensure the `http_endpoint` URL is in a similar format to the example pointing to the `/api/v1/spans`. + +Note: `header_type` and `default_header_type` are deprecated in favor of `config.propagation`, though they remain fully functional — creating a plugin with these fields still auto-populates the newer, canonical `config.propagation` field. + +Example config: + +```json +{ + "config": { + "tags_header": "Zipkin-Tags", + "include_credential": true, + "sample_ratio": 1, + "traceid_byte_count": 16, + "default_header_type": "b3", + "default_service_name": "kong", + "http_endpoint": "http://jaeger_jaeger-collector_1:9411/api/v1/spans", + "header_type": "preserve", + "static_tags": null, + "local_service_name": "kong" + }, +} +``` diff --git a/app/_support/ldap-authentication-for-kong-manager-failed-with-user-not-found-error-message.md b/app/_support/ldap-authentication-for-kong-manager-failed-with-user-not-found-error-message.md new file mode 100644 index 0000000000..f064f64c82 --- /dev/null +++ b/app/_support/ldap-authentication-for-kong-manager-failed-with-user-not-found-error-message.md @@ -0,0 +1,46 @@ +--- +title: "LDAP authentication for Kong Manager failed with \"User not found\" error message" +content_type: support +description: 'LDAP login to Kong Manager fails with a "user not found" error when the LDAP user does not exist or is not under the configured `base_dn` — a different, later-stage error than "Admin not found."' +products: + - gateway +works_on: + - on-prem + - konnect +tldr: + q: Why does LDAP authentication for Kong Manager fail with a "user not found" error? + a: | + Kong Manager's LDAP auth fails with `user not found` when the `kong_admin` (or other) LDAP user doesn't exist, or its DN falls outside the configured `base_dn`. A different, earlier error, `Admin not found`, means the corresponding Admin object was never created in Kong; create that first, then troubleshoot the LDAP-side lookup. +related_resources: [] +--- + +## Problem + +When logging in to Kong Manager as an LDAP user, authentication fails with the following error: + +``` +2025/04/14 20:23:42 [error] 26#0: *68309 [kong] access.lua:183 user not found, client: 172.19.0.1, server: kong_admin, request: "GET /auth HTTP/1.1", host: "localhost:8001", referrer: "http://localhost:8002/" +``` + +## Cause + +An example of the full error messages is as follows; + +``` +2025/04/14 20:23:42 [debug] 26#0: *68309 [kong] access.lua:130 binding with cn=admin,dc=kong,dc=local and conf.ldap_password +2025/04/14 20:23:42 [debug] 26#0: *68309 [kong] access.lua:140 ldap bind successful, performing search request with base_dn:ou=users,dc=kong,dc=local, scope='sub', and filter=uid=kong_admin +2025/04/14 20:23:42 [debug] 26#0: *68309 [kong] access.lua:165 finding groups with member attribute: memberOf +2025/04/14 20:23:42 [error] 26#0: *68309 [kong] access.lua:183 user not found, client: 172.19.0.1, server: kong_admin, request: "GET /auth HTTP/1.1", host: "localhost:8001", referrer: "http://localhost:8002/" +``` + +This indicated that `kong_admin` was not found in the LDAP server. + +## Solution + +Check the following things; + +1. Make sure the `kong_admin` user exists in your LDAP directory + +2. Make sure the `kong_admin` user's DN is under the sub-directory tree of `base_dn` + +Note: there is an earlier prerequisite gate that produces a different symptom. If the username was never provisioned as a Kong Admin object, Kong logs "Admin not found" instead of "User not found" — a distinct, earlier error in the login flow than the one described above. If you see "Admin not found," create the corresponding Admin object in Kong first, before troubleshooting the LDAP-side lookup. diff --git a/app/_support/ldap-authentication-query-fails-when-referrals-are-used-errors-failed-ldap-search-received-incorrect-op-in-packet.md b/app/_support/ldap-authentication-query-fails-when-referrals-are-used-errors-failed-ldap-search-received-incorrect-op-in-packet.md new file mode 100644 index 0000000000..b10dccf2de --- /dev/null +++ b/app/_support/ldap-authentication-query-fails-when-referrals-are-used-errors-failed-ldap-search-received-incorrect-op-in-packet.md @@ -0,0 +1,36 @@ +--- +title: "LDAP Authentication query fails when referrals are used. Errors: \"failed ldap search\" & \"received incorrect Op in packet\"" +content_type: support +description: As of now, the Kong LDAP Authentication plugin does not support LDAP referrals. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does LDAP Authentication fail with "failed ldap search" and "received incorrect Op in packet" when referrals are used? + a: | + The Kong LDAP Authentication plugin doesn't support LDAP referrals, so a query that triggers a referral to another LDAP server fails even though the underlying search succeeded. Work around this by pointing the plugin at a Global Catalog server instead of a single LDAP server. +--- + +## Problem + +We're trying to use the LDAP Authentication plugin with our LDAP server but it appears to be failing. Our environment uses LDAP referrals. Our Kong logs show errors similar to the following when an LDAP query fails: + +``` +2026/02/06 21:32:35 [error] 22#0: *10808 [lua] access.lua:89: ldap_authenticate(): [ldap-auth-advanced] failed ldap search for cn=sec23206 base_dn=dc=wmservice,dc=corpnet1,dc=com, client: 172.23.0.1, server: kong_admin, request: "GET /userinfo HTTP/1.1", host: "localhost:8001", referrer: "http://localhost:8002/login" +2026/02/06 21:32:35 [error] 22#0: *10808 [lua] responses.lua:121: ldap_authenticate(): Received incorrect Op in packet: 19, expected 5, client: 172.23.0.1, server: kong_admin, request: "GET /userinfo HTTP/1.1", host: "localhost:8001", referrer: "http://localhost:8002/login" +``` + +## Cause + +As of now, the Kong LDAP Authentication plugin does not support LDAP referrals. This limitation is documented in the LDAP Authentication Advanced plugin documentation. + +This means that if the queried LDAP server returns a referral asking Kong to query a different LDAP server, Kong cannot follow the referral. In such cases, the LDAP Authentication plugin will return an error, even if the search itself was successful. + +## Solution + +A feature request to support LDAP referrals has been filed as FTI-582. When discussing this with an Account Executive, please reference this FTI number. + +A recommended workaround is to configure the LDAP Authentication plugin to query the Global Catalog server instead of a single LDAP server. For more information, see the following references: Global Catalog overview and Windows 2000 Server documentation. diff --git a/app/_support/logging-to-json-format-for-kubernetes.md b/app/_support/logging-to-json-format-for-kubernetes.md new file mode 100644 index 0000000000..131e59e78b --- /dev/null +++ b/app/_support/logging-to-json-format-for-kubernetes.md @@ -0,0 +1,51 @@ +--- +title: Logging to JSON format for Kubernetes +content_type: support +description: 'The {{site.kic_product_name}} supports a `--log-format` flag to switch its logs from plain text to JSON, which can be set via `ingressController.args` in the Helm chart.' +products: + - kic +works_on: + - on-prem + - konnect +tldr: + q: How do I configure the {{site.kic_product_name}} to log in JSON format? + a: | + Set the Ingress Controller's `--log-format` flag to `json` (the default is `text`). With the Helm chart, pass it via `--set ingressController.args[0]='--log-format=json'`, or add it under `ingressController.args` in your `values.yaml`. +related_resources: + - text: the `--log-format` option + url: https://github.com/Kong/kubernetes-ingress-controller/blob/main/internal/cmd/rootcmd/config/cli.go#L55 + - text: the `ingressController.args` field in the Kong Helm chart values.yaml + url: https://github.com/Kong/charts/blob/main/charts/kong/values.yaml#L588 +--- + +## Problem + +By default Kong on containerized environments logs in plain text to `STDOUT`/`STDERR`. I would like this to log in a JSON format, how is this possible? + +## Solution + +The current Kong Kubernetes Ingress Controller (KIC) has the `--log-format` option. + +This defaults to text and can be configured to JSON. With the helm chart specifically, this can be added to the arguments for the ingress controller. + +Essentially something like `--set ingressController.args[0]='--log-format=json'` for the helm chart deployment, or adding it to the `ingressController.args` of their `values.yaml` file for their installation if they do it that way. + +Example: + +```bash +helm install ingress-controller kong/kong --create-namespace --namespace kong-system --set ingressController.args[0]='--log-format=json' +``` + +Default logging format: + +``` +time="2025-02-22T17:21:15Z" level=info msg="syncing configuration" component=controller +time="2025-02-22T17:21:15Z" level=info msg="no configuration change, skipping sync to kong" component=controller +``` + +New logging format: + +```json +{"component":"controller","level":"info","msg":"syncing configuration","time":"2025-02-22T17:21:15Z"} +{"component":"controller","level":"info","msg":"no configuration change, skipping sync to kong","time":"2025-02-22T17:21:15Z"} +``` diff --git a/app/_support/meet-cors-error-when-use-openid-connect-to-login-kong-manager.md b/app/_support/meet-cors-error-when-use-openid-connect-to-login-kong-manager.md new file mode 100644 index 0000000000..12a20f6818 --- /dev/null +++ b/app/_support/meet-cors-error-when-use-openid-connect-to-login-kong-manager.md @@ -0,0 +1,35 @@ +--- +title: Meet CORS error when using openid-connect to login Kong Manager +content_type: support +description: "A CORS error appears when logging in to Kong Manager through OpenID Connect because `login_action: redirect` in `KONG_ADMIN_GUI_AUTH_CONF` (or `admin_gui_auth_conf`) makes Kong Manager redirect to itself via the Admin API." +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: + - text: "more detail on the `login_action` config" + url: "/plugins/openid-connect/reference/#schema--config-login_action" +tldr: + q: Why do I get a CORS error when logging in to Kong Manager with OpenID Connect? + a: | + Kong Manager redirects to itself through the Admin API after the IdP login when `login_action: redirect` is set in `KONG_ADMIN_GUI_AUTH_CONF` (or `admin_gui_auth_conf`), which triggers the CORS error. Remove `login_action: redirect` from that config to fix it. +--- + +## Problem + +My Kong Admin API is running at `http://kong:8001` and Kong Manager is running at `http://kong:8002` + +I am using openid-connect to login Kong Manager, but I am not able to login and see below error from browser + +``` +Access to XMLHttpRequest at 'http://kong:8002/#id_token=xxx' (redirected from 'http://kong:8001/auth?state=yyy&session_state=zzz&code=qqq') from origin 'http://kong:8002' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. +``` + +## Cause + +This issue will happen when you have set `login_action: redirect` in `KONG_ADMIN_GUI_AUTH_CONF` (or `admin_gui_auth_conf`). After a successful login at the IDP, Kong Manager tries to redirect to itself via the Admin API. + +## Solution + +To solve this error, please delete `login_action: redirect` from `KONG_ADMIN_GUI_AUTH_CONF` (or `admin_gui_auth_conf`). diff --git a/app/_support/no-route-found-with-those-values-while-prereading-client-data.md b/app/_support/no-route-found-with-those-values-while-prereading-client-data.md new file mode 100644 index 0000000000..be408b2ddd --- /dev/null +++ b/app/_support/no-route-found-with-those-values-while-prereading-client-data.md @@ -0,0 +1,39 @@ +--- +title: "\"no Route found with those values while prereading client data\" error when a TCP connection doesn't match any route" +content_type: support +description: "This error is the TCP equivalent of \"no Route matched with those values,\" indicating that no suitable route was found for the connection." +products: + - gateway + - kic +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does {{site.base_gateway}} log "no Route found with those values while prereading client data" for a TCP connection? + a: | + This is the TCP equivalent of "no Route matched with those values" — the connection was accepted, but its source, destination, or SNI didn't match any configured route. For TLS SNI-based `TCPIngress` routing, confirm the route's expected host header (for example `example.com`) and that the `TCPIngress` resource was created successfully and is associated with the correct `IngressClass`. +--- + +## Problem + +When consuming a TCP route on Gateway you notice the connection is terminated and the below message is logged: + +``` +2025/10/31 13:41:03 [error] 2135#0: *702 stream [lua] handler.lua:1249: before(): no Route found with those values while prereading client data, client: 192.168.59.1, server: 0.0.0.0:9000 +``` + +How can this be resolved? + +## Solution + +This error is the TCP equivalent of "no Route matched with those values," indicating that no suitable route was found for the connection. In the same log, you should see an entry just before the error showing a connection established to confirm port connectivity to the Gateway. + +``` +2025/10/31 13:41:03 [info] 2135#0: *702 client 192.168.59.51:48836 connected to 0.0.0.0:9000 +``` + +Provided the connection was established, this error occurs when the source, destination, or SNI does not match any configured route. For example, a route may require a host header like `example.com`, as with TLS SNI-based routing in `TCPIngress`. + +When using `TCPIngress` on Kubernetes, ensure the `TCPIngress` resource has been successfully created and is associated with the correct `IngressClass`. Reviewing the ingress controller logs can provide additional insight. + diff --git a/app/_support/offset-and-next-parameters-of-audit-log-api.md b/app/_support/offset-and-next-parameters-of-audit-log-api.md new file mode 100644 index 0000000000..0bfcc076e1 --- /dev/null +++ b/app/_support/offset-and-next-parameters-of-audit-log-api.md @@ -0,0 +1,54 @@ +--- +title: About offset and next parameters of Audit Log API +content_type: support +published: false +description: How to page through the {{site.base_gateway}} Audit Log API using the `offset` and `next` response parameters, and what to expect when there are no more records. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +faqs: + - q: How do I use the "offset" and "next" parameters with the Audit Log API? + a: | + By default, the audit API returns at most 100 items: + + ```bash + curl ":8001/audit/requests" | jq .data | jq length + 100 + ``` + + To get the next 100 items, use the `offset` parameter: + + ```bash + # Check current offset and next parameters + curl ":8001/audit/requests" | jq . | grep offset + "next": "/audit/requests?offset=&sort_by=request_timestamp&sort_desc=true", + "offset": "" + + # Get next 100 items with + curl ":8001/audit/requests?offset=" | jq . + ``` + + The following request returns the 100 items after that: + + ```bash + curl ":8001/audit/requests?offset=" | jq . + "next": "/audit/requests?offset=&sort_by=request_timestamp&sort_desc=true", + "offset": "" + + curl ":8001/audit/requests?offset=" | jq . + ``` + + By using `offset` recursively, you can get all the items. + - q: What will the "next" and "offset" attributes be in the response if there are no more records? + a: | + `next` will be `null` and there is no `offset` in the response. +tldr: + q: How do I paginate through the {{site.base_gateway}} Audit Log API using offset and next? + a: | + The Audit Log API returns up to 100 items per page. Use the `offset` value from the response's `next` field to fetch subsequent pages; when there are no more records, `next` is `null` and `offset` is absent. +--- + +## About offset and next parameters of Audit Log API diff --git a/app/_support/oidc-plugin-sending-the-refresh-token-does-not-refresh-the-authentication-token.md b/app/_support/oidc-plugin-sending-the-refresh-token-does-not-refresh-the-authentication-token.md new file mode 100644 index 0000000000..83a7b07b25 --- /dev/null +++ b/app/_support/oidc-plugin-sending-the-refresh-token-does-not-refresh-the-authentication-token.md @@ -0,0 +1,27 @@ +--- +title: "OIDC Plugin: Sending the `refresh_token` does not refresh the authentication token" +content_type: support +published: false +description: With the {{site.base_gateway}} OIDC plugin, the refresh token must be sent on its own; if it is sent alongside the authentication token the refresh is ignored. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: Why does sending the refresh token alongside the authentication token fail to refresh it? + a: | + The {{site.base_gateway}} OIDC plugin ignores the refresh token when it is sent together with the authentication token, and refuses authentication based on the expired token. + Send the refresh token on its own, only when the client detects the authentication token has expired. +--- + +## Problem + +An OIDC client sends the refresh token along with the authentication token, but the authentication token never refreshes. + +## Solution + +When using refresh tokens with the {{site.base_gateway}} OIDC plugin it is important the refresh token is sent on its own and not with the authentication token. + +The OIDC client should check the expiration of the token and decide if a refresh is required. If this is the case then the refresh token should be sent from the client. If both tokens are sent the refresh is ignored and authentication is refused based on the authentication token's expired status. diff --git a/app/_support/oidc-wild-card-support-for-azure-idp.md b/app/_support/oidc-wild-card-support-for-azure-idp.md new file mode 100644 index 0000000000..480078aceb --- /dev/null +++ b/app/_support/oidc-wild-card-support-for-azure-idp.md @@ -0,0 +1,30 @@ +--- +title: OIDC - Wild card support for login redirects with Azure as an IDP +content_type: support +description: A static `redirect_uri` combined with `login_action=redirect` lets the OIDC plugin redirect back to the originally requested URL after authentication, working around Azure AD's lack of wildcard login redirect support. +products: + - gateway +works_on: + - on-prem + - konnect +related_resources: [] +tldr: + q: How can I support wildcard login redirects with Azure AD using the OIDC plugin? + a: | + Azure AD doesn't support wildcard redirect URIs. Configure the OIDC plugin with a static `redirect_uri`, and set `login_action=redirect` and `preserve_query_args=true` so Kong performs an additional redirect to the originally requested URL after authentication. +--- + +## Problem + +We use Azure as our OpenID Connect (OIDC) Identity Provider (IDP). Azure documentation states they have no support for wild card login redirects, is there any workaround for this? + +## Solution + +The following config for the OIDC plugin will perform an additional redirect (to the originally requested URL) at the Kong side once the OIDC has performed its authentication: + +``` +config.redirect_uri= -- as adfs does not support wildcards +config.login_action=redirect -- to make extra redirecting after the login +config.preserve_query_args=true -- to preserve possible query args from original url +config.login_tokens= -- set it to null so that plugin does not add any tokens to redirection +``` diff --git a/app/_support/old-data-plane-entries-are-not-getting-removed-from-clustering-data-plane-endpoint.md b/app/_support/old-data-plane-entries-are-not-getting-removed-from-clustering-data-plane-endpoint.md new file mode 100644 index 0000000000..32cef707a7 --- /dev/null +++ b/app/_support/old-data-plane-entries-are-not-getting-removed-from-clustering-data-plane-endpoint.md @@ -0,0 +1,33 @@ +--- +title: Old data plane entries are not getting removed from clustering data-plane endpoint +content_type: support +description: This is because Kong keeps data plane entries for 14 days by default. +products: + - gateway +works_on: + - on-prem + - konnect +published: false +related_resources: + - text: the `cluster_data_plane_purge_delay` configuration reference + url: /gateway/configuration/#cluster-data-plane-purge-delay +tldr: + q: Why does the `/clustering/data-planes` API endpoint return more data planes than there are active pods? + a: | + Kong keeps data plane entries for 14 days by default, so a control plane that hasn't heard from a data plane within that window still lists it. Lower `cluster_data_plane_purge_delay` to shorten the retention window. +--- + +## Old data plane entries are not getting removed from clustering data-plane endpoint + +Why does the `/clustering/data-planes` API endpoint return more data planes than there are actual working pods? + +This is because Kong keeps data plane entries for 14 days by default. If the CP hasn't heard from a DP for 14 days, its entry will be removed. + +If users want to shorten this time, they can set the time via the `cluster_data_plane_purge_delay` Kong parameter. + +If users deploy Kong with a Helm chart, they need to add this under `env` + +```yaml +env: + cluster_data_plane_purge_delay: "