Fix Shell syntax highlighting after nested command substitution

[Keerthana Sathyavada (SK-Sathyavada-008)]

Fix Shell syntax highlighting after nested command substitution

Fixes #5231

Problem

Shell syntax highlighting could become incorrect after a command substitution containing a quoted sed expression with parentheses.

For example:

GIT_HOST=$(echo "$GIT_REPO" | sed -E 's/^.*@([^:/]+).*$/\1/')
DIR_NAME=$(basename "$GIT_REPO" .git)

The ) inside the sed expression could be interpreted as the closing parenthesis of the $() command substitution. This caused the tokenizer to enter an incorrect state, resulting in incorrect highlighting on subsequent lines.

Changes

• Changed @popall to @pop when closing single- and double-quoted strings so nested strings return to their enclosing tokenizer state.
• Updated parameterBodyParen to handle strings and nested parameters correctly.
• Added a regression test for issue [Bug] Shell code color issue after specific line #5231.

Testing

• Added a regression test that reproduces the original issue.
• Verified that the regression test fails with the original implementation.
• Verified that the regression test passes with the fix.
• Verified that the existing Shell tokenizer tests pass.
• git diff --check passes with no whitespace errors.

[Keerthana Sathyavada (SK-Sathyavada-008)]

@microsoft-github-policy-service agree