Skip to content

[SPARK-59404][CORE] Match REST environment filtering in standalone Client submission #58696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
holdenk wants to merge 2 commits into
base: master
Choose a base branch
from
+56 −3
Open

[SPARK-59404][CORE] Match REST environment filtering in standalone Client submission #58696

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f1280a
Match REST environment filtering in standalone Client submission
sfc-gh-hkarau Sep 1, 2026
0b35f5d
Make standalone Client environment filtering configurable
holdenk Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import org.apache.spark.{SecurityManager, SparkConf}
import org.apache.spark.deploy.DeployMessages._
import org.apache.spark.deploy.master.{DriverState, Master}
import org.apache.spark.deploy.master.DriverState.DriverState
import org.apache.spark.deploy.rest.RestSubmissionClient
import org.apache.spark.internal.{config, Logging}
import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.config.Network.RPC_ASK_TIMEOUT
Expand Down Expand Up @@ -101,9 +102,10 @@ private class ClientEndpoint(

val sparkJavaOpts = Utils.sparkJavaOpts(conf)
val javaOpts = sparkJavaOpts ++ extraJavaOpts
val driverEnv = Client.driverEnvironment(conf, sys.env)
val command = new Command(mainClass,
Seq("{{WORKER_URL}}", "{{USER_JAR}}", driverArgs.mainClass) ++ driverArgs.driverOptions,
sys.env, classPathEntries, libraryPathEntries, javaOpts)
driverEnv, classPathEntries, libraryPathEntries, javaOpts)
val driverResourceReqs = ResourceUtils.parseResourceRequirements(conf,
config.SPARK_DRIVER_PREFIX)
val driverDescription = new DriverDescription(
Expand Down Expand Up @@ -275,6 +277,21 @@ object Client {
// scalastyle:on println
new ClientApp().start(args, new SparkConf())
}

/**
* Environment variables to forward to the driver. Only Spark-related variables are forwarded,
* matching the REST submission client, unless `spark.standalone.submit.filterEnvironment` is
* disabled, in which case the full environment of the submitting process is forwarded.
*/
private[deploy] def driverEnvironment(
conf: SparkConf,
env: Map[String, String]): Map[String, String] = {
if (conf.get(config.STANDALONE_SUBMIT_FILTER_ENVIRONMENT)) {
RestSubmissionClient.filterSystemEnvironment(env)
} else {
env
}
}
}

private[spark] class ClientApp extends SparkApplication {
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,7 +540,7 @@ private[spark] object RestSubmissionClient {
/**
* Filter non-spark environment variables from any environment.
*/
private[rest] def filterSystemEnvironment(env: Map[String, String]): Map[String, String] = {
private[spark] def filterSystemEnvironment(env: Map[String, String]): Map[String, String] = {
env.filter { case (k, _) =>
k.startsWith("SPARK_") && !EXCLUDED_SPARK_ENV_VARS.contains(k)
}
Expand Down
10 changes: 10 additions & 0 deletions core/src/main/scala/org/apache/spark/internal/config/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2787,6 +2787,16 @@ package object config {
.booleanConf
.createWithDefault(false)

private[spark] val STANDALONE_SUBMIT_FILTER_ENVIRONMENT =
ConfigBuilder("spark.standalone.submit.filterEnvironment")

@HyukjinKwon HyukjinKwon Sep 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this flips a default in a minor release (standalone cluster mode now forwards only SPARK_* vars to the driver, previously the submitter's full env), it's worth a docs/core-migration-guide.md entry so upgrading users discover the behavior change and the opt-out. The existing 4.3 entries are the pattern -- add a ## Upgrading from Core 4.3 to 4.4 section with something like: "Since Spark 4.4.0, in standalone cluster mode the client forwards only Spark-related environment variables to the driver. To restore the legacy behavior, set spark.standalone.submit.filterEnvironment to false." The config doc here and in spark-standalone.md cover the what; the migration guide is where the change on upgrade is flagged.

.doc("In standalone cluster mode, controls whether the client forwards only " +
"Spark-related environment variables to the driver, matching the REST submission " +
"client. If set to false, the full environment of the submitting process is " +
"forwarded to the driver.")
.version("4.4.0")

@HyukjinKwon HyukjinKwon Sep 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be 4.3.0?

.booleanConf
.createWithDefault(true)

private[spark] val EXECUTOR_ALLOW_SPARK_CONTEXT =
ConfigBuilder("spark.executor.allowSparkContext")
.doc("If set to true, SparkContext can be created in executors.")
Expand Down
18 changes: 17 additions & 1 deletion core/src/test/scala/org/apache/spark/deploy/ClientSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ package org.apache.spark.deploy
import org.scalatest.matchers.must.Matchers
import org.scalatest.matchers.should.Matchers._

import org.apache.spark.SparkFunSuite
import org.apache.spark.{SparkConf, SparkFunSuite}
import org.apache.spark.internal.config.STANDALONE_SUBMIT_FILTER_ENVIRONMENT

class ClientSuite extends SparkFunSuite with Matchers {
test("correctly validates driver jar URL's") {
Expand Down Expand Up @@ -48,4 +49,19 @@ class ClientSuite extends SparkFunSuite with Matchers {
// Invalid syntax.
ClientArguments.isValidJarUrl("hdfs:") should be (false)
}

test("SPARK-59404: forward only Spark-related environment variables to the driver") {
val env = Map(
"SPARK_LOCAL_IP" -> "127.0.0.1",
"SPARK_HOME" -> "/opt/spark",
"PATH" -> "/usr/bin",
"SECRET_TOKEN" -> "hunter2")
Client.driverEnvironment(new SparkConf(), env) should be (Map("SPARK_LOCAL_IP" -> "127.0.0.1"))
}

test("SPARK-59404: forward the full environment when filtering is disabled") {
val env = Map("SPARK_LOCAL_IP" -> "127.0.0.1", "PATH" -> "/usr/bin")
val conf = new SparkConf().set(STANDALONE_SUBMIT_FILTER_ENVIRONMENT, false)
Client.driverEnvironment(conf, env) should be (env)
}
}
10 changes: 10 additions & 0 deletions docs/spark-standalone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -575,6 +575,16 @@ Spark applications supports the following configuration properties specific to s
</td>
<td>3.1.0</td>
</tr>
<tr>
<td><code>spark.standalone.submit.filterEnvironment</code></td>
<td><code>true</code></td>
<td>
In standalone cluster mode, controls whether the client forwards only Spark-related environment
variables to the driver, matching the REST submission client. If set to <code>false</code>,
the full environment of the submitting process is forwarded to the driver.
</td>
<td>4.4.0</td>
</tr>
</table>


Expand Down