ddb expression lexer: base64 literals silently truncated, escaped backslashes not unescaped

[Adityaj0]

Describe the bug

The aws ddb expression lexer (awscli/customizations/dynamodb/lexer.py) mishandles quoted strings in three ways. The first one silently corrupts data.

1. Base64 literals are only validated at their start, so invalid input is silently truncated

_consume_base64_string validates with VALID_BASE64.match(...). re.match anchors only at the beginning of the string, so any invalid characters after a valid base64 prefix pass validation. base64.b64decode then discards characters outside the alphabet by default (validate=False), so the value is silently decoded to a truncated result and written to DynamoDB with no error.

The comment directly above that check states this is exactly what the manual validation is meant to prevent:

# Python will simply ignore invalid characters, so we have to
# validate manually.
if raw_string['value'] and not VALID_BASE64.match(raw_string['value']):

Note the check does work when the invalid characters come first (b"!!!!AAAA" is correctly rejected), which is likely why this went unnoticed.

2. An escaped backslash inside a quoted string or identifier is not unescaped

_consume_until preserves \X verbatim, and the callers only strip \" and \'. An escaped backslash therefore stays as two characters, so "C:\\path" produces C:\\path instead of C:\path. There is no way to express a single literal backslash in a ddb expression string.

3. The end offset recorded for quoted tokens is a length, not an offset

_consume_quoted_identifier and _consume_string_literal set 'end': token_len. Everywhere else end is an offset, and awscli/customizations/dynamodb/exceptions.py computes token_length = token['end'] - token['start'] to size the ^^^ underline in error messages. For any quoted token that does not start at offset 0 this goes negative and the underline collapses to a single ^ of the wrong width.

Expected Behavior

1. data = b"AAAA!!!!" is rejected as an invalid base64 literal.
2. path = "C:\\Users" produces the string C:\Users.
3. expression[token['start']:token['end']] yields the token's source text.

Current Behavior

1. data = b"AAAA!!!!" is accepted and decodes to b'\x00\x00\x00' — the !!!! is silently discarded and a truncated binary value is sent to DynamoDB.
2. path = "C:\\Users" produces C:\\Users (two backslashes).
3. end is a length, so the error-message underline is mis-sized.

Reproduction Steps

Against aws-cli/2.36.20, this shows the values that get substituted into the request actually sent to DynamoDB:

from awscli.customizations.dynamodb.extractor import AttributeExtractor

e = AttributeExtractor()
print(e.extract('data = b"AAAA!!!!"')['values'])
print(e.extract('path = "C:\\\\Users"')['values'])

Output on 2.36.20:

{':n1': Binary(b'\x00\x00\x00')}     # expected: an "Invalid base64 string" error
{':n1': 'C:\\\\Users'}               # expected: 'C:\\Users', i.e. C:\Users

These reach DynamoDB through any expression argument that accepts a literal, e.g.:

aws ddb select my-table --filter 'data = b"AAAA!!!!"'
aws ddb put my-table '{...}' --condition 'path = "C:\\Users"'

For the third issue:

from awscli.customizations.dynamodb.lexer import Lexer

tokens = list(Lexer().tokenize('bar = "foo"'))
tok = [t for t in tokens if t['type'] == 'literal'][0]
print(tok['start'], tok['end'])   # 6 4  -- end < start

Possible Solution

1. Validate the entire literal with fullmatch instead of match.
2. Resolve escape sequences while consuming the string, so that \\ collapses to a single backslash and \"/\' continue to yield the delimiter. A backslash that escapes neither the delimiter nor another backslash should stay verbatim, to preserve current behaviour for sequences like \n.
3. Record end as an offset.

I have a branch with these three fixes plus regression tests, and will open a PR referencing this issue.

Additional Information/Context

Found by reading the code rather than by hitting it in production, so I do not have a report of real-world data loss — but (1) fails silently and writes wrong data, which seemed worth reporting regardless.

CLI version used

aws-cli/2.36.20 Python/3.12.8 Darwin/25.2.0 source/arm64

Environment details (OS name and version, etc.)

macOS 15 (Darwin 25.2.0), arm64, Python 3.12.8