Repository navigation
WithEnv + Exec(ForEach) don't work togheter #239
Description
Activity
Or it shouldn't be evaluated at all:
https://go.dev/play/p/XWxxo4eEla_tThis is a valid issue. We don't respect the pipe's
envwhile parsing shell command passed toExecfmt.Printf("Started\n") env := []string{`MY_ENV=foo`} out1, _ := script.NewPipe().WithEnv(env).Exec(`echo $MY_ENV`).String() cmd := exec.Command("sh", "-c", `echo $MY_ENV`) cmd.Env = env out2, _ := cmd.Output() out3, _ := exec.Command("sh", "-c", `MY_ENV=foo; echo $MY_ENV`).Output() fmt.Printf("out1 = '%s'\n", out1) fmt.Printf("out2 = '%s'\n", out2) fmt.Printf("out3 = '%s'\n", out3) os.Setenv("MY_ENV", "foo") out4, _ := script.NewPipe().WithEnv(env).Exec(`echo $MY_ENV`).String() fmt.Printf("(After setting env) out4 = '%s'\n", out4)
This prints
➜ examples git:(temp) ✗ go run env.go Started out1 = ' ' out2 = 'foo ' out3 = 'foo ' (After setting env) out4 = 'foo ' ➜ examples git:(temp) ✗```
The way to fix it is passing a function to get env in
shell.Fieldsas you pointed out.Proof of concept
diff --git a/script.go b/script.go index d7d1bc3..2efebac 100644 --- a/script.go +++ b/script.go @@ -428,7 +428,7 @@ func (p *Pipe) Error() error { // instead be redirected to a supplied writer, using [Pipe.WithStderr]. func (p *Pipe) Exec(cmdLine string) *Pipe { return p.Filter(func(r io.Reader, w io.Writer) error { - args, err := shell.Fields(cmdLine, nil) + args, err := shell.Fields(cmdLine, p.getShellEnvFunc()) if err != nil { return err } @@ -453,6 +453,24 @@ func (p *Pipe) Exec(cmdLine string) *Pipe { }) } +func (p *Pipe) getShellEnvFunc() func(string) string { + pipeEnv := p.environment() + if pipeEnv == nil { + return nil + } + pipEnvMap := map[string]string{} + for _, s := range pipeEnv { + vals := strings.Split(s, "=") + // Needs more error handling + if len(vals) == 2 { + pipEnvMap[vals[0]] = vals[1] + } + } + return func(key string) string { + return pipEnvMap[key] + } +} + // ExecForEach renders cmdLine as a Go template for each line of input, running // the resulting command, and produces the combined output of all these // commands in sequence. See [Pipe.Exec] for details on error handling and
New output of the same code that is in above message
➜ examples git:(temp) ✗ go run env.go Started out1 = 'foo ' out2 = 'foo ' out3 = 'foo ' (After setting env) out4 = 'foo ' ➜ examples git:(temp) ✗
@bitfield Let me know if you think this is a valid bug, and the library should have a fix for it. The implementation can be refined more.
For your information @prochac the below in your snippet (https://go.dev/play/p/XWxxo4eEla_t) doesn't works because what you are doing is passing literal string
${MY_ENV}toechocmd = exec.Command("echo", "${MY_ENV}") cmd.Env = []string{`MY_ENV="foo"`} out, err = cmd.Output() fmt.Println(string(out), err)
typing
echo ${MY_ENV}works in shell, as shell first expandsMY_ENVand then sends it toecho.I think the problem here is that
echodoesn't expand variables; it's the shell that does that. Try https://go.dev/play/p/i-KO3tFxYcS@bitfield and this library is supposed to imitate shell, so the library should expand variables before passing the arguments to
echo.Do you agree or not that the below should return
foo? That what the whole issue is about.script.NewPipe().WithEnv([]string{"MY_ENV=foo"}).Exec("echo $MY_ENV").String()
Now if you say that
bitfield/scriptshould not expand variable, then$HOMEshouldn't be expanded here. But it is expanded.script.NewPipe().Exec(`echo $HOME`).String()
That's a good point, I hadn't spotted that.
But let's go carefully and one step at a time. The purpose of
WithEnvis to set environment variables that should be passed on to commands run withExec, and it does that (not surprising, as there's a test for this, and it's passing). Try this program, for example:https://go.dev/play/p/-w-SQASoGZ5
It prints, as you'd expect:
MY_ENV=fooSo there's nothing wrong with
WithEnv, or the way we pass those environment settings toExec.There's a separate question about whether
Execshould expand variables in the user's command line before it is passed to the spawned command. Currently it does not. There is an argument for opening an issue that it should (I personally am not convinced, but if you are, create the issue and we can discuss it there where it'll be on topic.)Yes, I agree that this issue is incorrectly titled. And yes, Exec works as expected. The question was only about the expansion of variables in the user's command line before it is passed to the spawned command.
After some thinking I also agree that it should not expand. The current
WithEnvcorrectly imitatesenvcommand.➜ ~/Workspace/kuknitin/cxxopts/example git:(lazysegtree) ✗ [11:57:43] env A=1 \ > echo $A ➜ ~/Workspace/kuknitin/cxxopts/example git:(lazysegtree) ✗ [11:57:49] env A=1 \ echo $HOME /Users/kuknitin ➜ ~/Workspace/kuknitin/cxxopts/example git:(lazysegtree) ✗ [11:57:57] ➜ ~/Workspace/kuknitin/cxxopts/example git:(lazysegtree) ✗ [12:00:28] env A=1 \ > python3 -c 'import os; print(os.getenv("A"))' 1 ➜ ~/Workspace/kuknitin/cxxopts/example git:(lazysegtree) ✗ [12:00:40]
My problem is, that the
shell.Fields(cmdLine, nil)does expand the envs. It just expands them as an empty string.I know why
exec.Command("echo", "${MY_ENV}")doesn't expand. But it's preceded by theshell.Fieldsthat does the expansion, but to empty strings.It just expands them as an empty string.
I think I finally get it! The point I was missing is that
shell.Fieldsexpands variables. (I actually wasn't aware of that, and now that I know it does, I wish it didn't.)Unfortunately, what it takes is a
func (string) string(because that's whatos.Getenvis) and what we have is a[]stringof the formKEY=VALUE. That's a really awkward mismatch, as @lazysegtree demonstrated with the code that's necessary to resolve it.While I'm sure that solution is correct, and I doubt if it can be made more elegant, I can't accept that code in this project. It offends me. And I don't think we should be in the business of writing it. We're going to more and more trouble over time just to (poorly) replicate the behaviour of the shell.
Instead, I'm inclined to say "Let's just use the shell". In other words, pass the user's command line directly and without modification to a
sh -ccommand. What if they don't havesh, or they don't want to spawn an extra subprocess? I guess they can useexec.Commanddirectly in that case.I am good with
sh -c. It makes implementation so much simpler with any practical performance overhead that I can forsee. No need parsing or field splitting of user's command.Although what about Windows? I don't know how much support we have for windows. Having to use
cmd.exeorpowershell.exemight have too much overhead.Sorry :D yeah, I filled that ticket at quite late hour, not being native, and had focus on a different issue 😄
Yeah, what bothers me is the inconsistency. I solved my problem by doing the expansion using
fmt.Sprintf.
But the inconsistency remains. TBH, I have no conservative solution.
It all starts with the decision, if thescriptshould mimic Shell with the expansion, or not.An alternative is to not use the
script.Fieldsfunc from theexpand.gofile 😛 cut the expansion out, and use just the parser:
https://github.com/mvdan/sh/blob/master/shell/expand.go#L52func cmdToFields(s string) ([]string, error) { p := syntax.NewParser() var words []*syntax.Word for w, err := range p.WordsSeq(strings.NewReader(s)) { if err != nil { return nil, err } words = append(words, w) } return words, nil }My question would be something like this: If we're to try to replicate the behaviour of the shell, can we guarantee to do so exactly, and in all circumstances? Because if the answer is no (and it obviously is), it's probably better not to try, and instead simply either use the shell or not use the shell.
So we could change Exec to run the command in an implicit sh -c and provide another method, say ExecArgs, which does no expansion and just takes each arg separately, like exec.Command does. I think that's even been requested in another issue.
To answer the question about Windows, we support everything on Windows that we reasonably can. I'm not a Windows user though so I'm not sure what behaviour would make sense here. Perhaps we can give the command line to CMD.exe so that it's processed in the same way that it would be if the user had typed it into a terminal.
Hi @bitfield, @lazysegtree — I'd like to pick this up if that's okay.
To confirm the direction before I start implementing:
Exec: change it to run the command line via sh -c "..." (or platform shell) instead of parsing it with shell.Fields, so variable expansion is handled correctly and consistently by the real shell rather than reimplemented in the library.
New method for the no-expansion case, equivalent to exec.Command, taking args separately with no shell involved. A few possible names: ExecArgs, ExecCmd, ExecRaw — happy to go with whatever you prefer, or if there's already a naming convention I should follow.A couple of things I think need pinning down before a PR, since they affect the design:
Windows behavior — should Exec shell out to cmd.exe, or is sh -c (e.g. via WSL/Git Bash) acceptable, or should Windows just not get shell expansion at all for now? Wanted to flag this explicitly since it was raised but not resolved above.
Backward compatibility — switching Exec from Go's arg-splitting to real shell semantics could change behavior for existing users who rely on the current (non-shell) parsing, e.g. quoting/escaping edge cases. Should this go behind a new method entirely (leaving Exec unchanged) rather than modifying Exec itself, to avoid a breaking change? I lean toward introducing the shell-based behavior as the new default in Exec since that matches "the whole point of this library is to feel like the shell," but wanted to check.
Once naming is settled, I'll put together a small real-world use case (e.g. a script that needs $HOME-style expansion combined with WithEnv values) to include with the PR, per the contributing guide.Let me know your thoughts and I'll get started once we're aligned on the above.
Yes, I think a new method (
Shell?) that passes the unmodified command line to the system shell would be the right thing to do here, leavingExecunchanged.Sounds good, I'll go with Shell and leave Exec untouched. I'll get started and put together a PR with tests, docs, and a use case.
Reacted by John Arundel- added a commit that references this issue
on Aug 15, 2026
The
script.Execcontains this codethat already evaluates the env variables, but
nilis passed in. So the variables are evaluated as empty string.https://go.dev/play/p/P73IM0NLNBv
Or am I using it wrong?