Skip to content

Support per-row path in named_credential_request_col - #158

Merged
rkang-sf merged 3 commits into
mainfrom
rkang/named-credential-per-row-path
Oct 9, 2026
Merged

rkang-sf merged 3 commits into
mainfrom
rkang/named-credential-per-row-path

Conversation

@rkang-sf

@rkang-sf rkang-sf commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

named_credential_request_col could only vary the request body per row; the url came from the HTTPRequest template and was the same for every row. This adds an optional per-row path Column that is appended to request.url, so each row can target a different endpoint of the same Named Credential (e.g. pick a model, per-entity GET lookups, geocoding):

REQUEST = HTTPRequestBuilder().set_url("callout:gemini").set_method(HTTPMethod.POST).build()
path = when(col("premium"), lit("/gemini-2.5-pro")).otherwise(lit("/gemini-2.5-flash"))
df = df.withColumn("resp", named_credential_request_col(REQUEST, body=col("payload"), path=path))

Join rule (rows can never switch Named Credential):

  • path starts with / or ? → appended as is: callout:gemini + /v1 → callout:gemini/v1; + ?k=a → callout:gemini?k=a
  • any other path → joined with /: callout:gemini + 2/v1 → callout:gemini/2/v1 (not callout:gemini2/v1)
  • null or empty path → request.url
  • path omitted → exactly the previous behavior. named_credential_request_col only forwards path when given, so SparkNamedCredential implementations that don't accept it yet keep working for existing callers.

Changes

  • SparkNamedCredential.request_col: new optional path keyword; docstrings updated.
  • DefaultSparkNamedCredential.request_col (local dev): with path, the UDF is applied over (body, path); each row uses a copy of the template whose url is join_callout_path(request.url, path).
  • named_credential_request_col: new optional path argument.
  • Separate Lint fixes commit: make lint currently fails on main (pretty-format-toml on pyproject.toml, black on tests/spark/test_column_hints_deferred.py). Formatting only; the reformatted pyproject.toml parses to identical data.

The Data Cloud runtime's built-in named_credential_callout UDF already takes the url per row, so no server-side change is needed; the runtime-side join lives in the internal SDK with the same rule.

Testing

  • Unit tests: path forwarding, per-row path join, separator rule (incl. 2/v1, @evil.example.com), null/empty fallback, path without body. Full suite: 625 passed.
  • pre-commit run --all-files --hook-stage manual (same as make lint): all hooks pass.
  • Local Spark smoke test with a real DataFrame over /gemini-2.5-pro, ?key=a, 2/v1, @evil.example.com, "", NULL: urls joined as above, empty/null fall back to callout:gemini, omitting path is unchanged.

Add an optional url Column to named_credential_request_col /
SparkNamedCredential.request_col so each row can call a different endpoint
(e.g. per-entity REST GET lookups). A null row value falls back to
request.url; omitting url keeps the existing expression unchanged.
Rows may only select an endpoint under the template's Named Credential: the
path is appended to request.url, with '/' inserted unless it starts with '/'
or '?', so a row can never change the Named Credential name. A null or empty
path uses request.url.
@rkang-sf
rkang-sf merged commit 403ecdc into main Oct 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants