gh-156047: Rank tokens before probing for a keyword typo by johnslavik · Pull Request #156087 · python/cpython

johnslavik · 2026-08-20T09:14:57Z

@pablogsal Look at this! Amazing!

>>> a=b=c=d=e=f=g=h=i=j=k=l=m=0;tpye x = 1;n=o=p=q=r=s=t=u=v=w=x=y=z=2
  File "<python-input-0>", line 1
    a=b=c=d=e=f=g=h=i=j=k=l=m=0;tpye x = 1;n=o=p=q=r=s=t=u=v=w=x=y=z=2
                                ^^^^
SyntaxError: invalid syntax. Did you mean 'type'?

cc @m-aciek

Issue: Keyword typo suggestions differ between file and string tokenizers #156047

johnslavik · 2026-08-20T09:25:36Z

FYI here's logs of what tokens are examined in the typo cases from our tests:

Details

============='with block ad something:\n  pass'=============
rank=1 token.string='ad'
SyntaxError: invalid syntax. Did you mean 'and'?

==================='fur a in b:\n  pass'====================
rank=1 token.string='a'
rank=3 token.string='fur'
SyntaxError: invalid syntax. Did you mean 'for'?

============'for a in b:\n  pass\nelso:\n  pass'============
rank=3 token.string='elso'
SyntaxError: invalid syntax. Did you mean 'else'?

==================='whille True:\n  pass'===================
rank=6 token.string='whille'
SyntaxError: invalid syntax. Did you mean 'while'?

===================='iff x > 5:\n  pass'====================
rank=1 token.string='x'
rank=3 token.string='iff'
SyntaxError: invalid syntax. Did you mean 'if'?

============='if x:\n  pass\nelseif y:\n  pass'=============
rank=1 token.string='y'
rank=3 token.string='elseif'
SyntaxError: invalid syntax. Did you mean 'elif'?

============='tyo:\n  pass\nexcept y:\n  pass'==============
rank=3 token.string='tyo'
SyntaxError: invalid syntax. Did you mean 'try'?

================='classe MyClass:\n  pass'==================
rank=1 token.string='MyClass'
rank=9 token.string='classe'
SyntaxError: invalid syntax. Did you mean 'class'?

========================'impor math'========================
rank=1 token.string='math'
rank=6 token.string='impor'
SyntaxError: invalid syntax. Did you mean 'import'?

====================='form x import y'======================
rank=1 token.string='x'
rank=3 token.string='form'
SyntaxError: invalid syntax. Did you mean 'from'?

========'defn calculate_sum(a, b):\n  return a + b'=========
rank=1 token.string='a'
rank=1 token.string='calculate_sum'
rank=4 token.string='b'
rank=15 token.string='defn'
SyntaxError: invalid syntax. Did you mean 'def'?

==============='def foo():\n  returm result'================
rank=1 token.string='result'
rank=8 token.string='returm'
SyntaxError: invalid syntax. Did you mean 'return'?

====================='lamda x: x ** 2'======================
rank=1 token.string='x'
rank=2 token.string='x'
rank=3 token.string='lamda'
SyntaxError: invalid syntax. Did you mean 'lambda'?

==================='def foo():\n  yeld i'===================
rank=1 token.string='i'
rank=3 token.string='yeld'
SyntaxError: invalid syntax. Did you mean 'yield'?

==============='def foo():\n  globel counter'===============
rank=1 token.string='counter'
rank=9 token.string='globel'
SyntaxError: invalid syntax. Did you mean 'global'?

=================='frum math import sqrt'===================
rank=1 token.string='math'
rank=6 token.string='frum'
SyntaxError: invalid syntax. Did you mean 'from'?

============='asynch def fetch_data():\n  pass'=============
rank=5 token.string='asynch'
SyntaxError: invalid syntax. Did you mean 'async'?

=========='async def foo():\n  awaid fetch_data()'==========
rank=1 token.string='fetch_data'
rank=12 token.string='awaid'
SyntaxError: invalid syntax. Did you mean 'await'?

================'raisee ValueError("Error")'================
rank=1 token.string='ValueError'
rank=12 token.string='raisee'
SyntaxError: invalid syntax. Did you mean 'raise'?

==============='[x for x\nin range(3)\nof x]'===============
rank=1 token.string='of'
SyntaxError: invalid syntax. Did you mean 'if'?

=============='[123 fur x\nin range(3)\nif x]'==============
rank=1 token.string='fur'
SyntaxError: invalid syntax. Did you mean 'for'?

==================='for x im n:\n  pass'====================
rank=1 token.string='im'
SyntaxError: invalid syntax. Did you mean 'in'?

========================='mach x:'==========================
rank=1 token.string='x'
rank=3 token.string='mach'
SyntaxError: invalid syntax. Did you mean 'match'?

========================='math x:'==========================
rank=1 token.string='x'
rank=3 token.string='math'
SyntaxError: invalid syntax. Did you mean 'match'?

===================='match 1:\n  cse 1:'====================
rank=1 token.string='cse'
SyntaxError: invalid syntax. Did you mean 'case'?

======================='typ x = int'========================
rank=1 token.string='x'
rank=3 token.string='typ'
SyntaxError: invalid syntax. Did you mean 'type'?

======================'typed x = int'=======================
rank=1 token.string='x'
rank=3 token.string='typed'
SyntaxError: invalid syntax. Did you mean 'type'?

======================'lazi import x'=======================
rank=1 token.string='x'
rank=8 token.string='lazi'
SyntaxError: invalid syntax. Did you mean 'lazy'?

======================'lezi import x'=======================
rank=1 token.string='x'
rank=8 token.string='lezi'
SyntaxError: invalid syntax. Did you mean 'lazy'?

===================='switch x:\n case:'=====================
rank=1 token.string='x'
rank=3 token.string='switch'
SyntaxError: invalid syntax. Did you mean 'match'?

========================='delete x'=========================
rank=1 token.string='x'
rank=3 token.string='delete'
SyntaxError: invalid syntax. Did you mean 'del'?

======================'function f():'=======================
rank=1 token.string='f'
rank=3 token.string='function'
SyntaxError: invalid syntax. Did you mean 'def'?

========================'func f():'=========================
rank=1 token.string='f'
rank=3 token.string='func'
SyntaxError: invalid syntax. Did you mean 'def'?

========================'void f():'=========================
rank=1 token.string='f'
rank=3 token.string='void'
SyntaxError: invalid syntax. Did you mean 'def'?

='a=a=a=a=a=a=a=a=a=a=0;tpye x = int;z=z=z=z=z=z=z=z=z=z=1'=
rank=1 token.string='x'
rank=3 token.string='tpye'
SyntaxError: invalid syntax. Did you mean 'type'?

We can tweak the rank calculation a bit to look for a token on the left first but I don't think it's a huge deal in this context.

pablogsal

Hummmm…. I am not very convinced about this method. It’s true that’s a bit better but it is still not without inconsistencies

pablogsal · 2026-08-20T09:56:20Z

+                the_end = end_line if line == 0 else end_line + 1
+                if token.start[0] + line != the_end:
+                    continue
+                rank = abs(offset - token.end[1])


I think the rank anchor is a bit off here: self.end_offset is 1-based and relative to the original line, but token.end[1] is 0-based and relative to the dedented snippet, so for indented code the anchor drifts right by the dedent width and the neighbours get probed before the typo (you can see it in your traces, where x ranks above iff). Not incorrect because every candidate is validated by recompiling, but can we translate end_offset to the snippet coordinates before ranking?

I'll also add a case where dedent() makes a difference. For the TYPO_CASES we had, dedent() was a no-op so this was harder to spot.

pablogsal · 2026-08-20T09:56:20Z

+                the_end = end_line if line == 0 else end_line + 1
+                if token.start[0] + line != the_end:
+                    continue
+                rank = abs(offset - token.end[1])


This can prefer the token at the parser caret over the actual typo. retrun a + b now suggests and, because replacing a makes the expression compile. We need a ranking that keeps retrun first and a test for this case.

Somewhat related fun fact, you can trigger this in main too:

def outer(): if True: pass retrun a+b

The dedented block includes only the body of the function without the def, so retrun->return won't compile since it is "outside function".

I know I'm getting off-topic but in fact, any functions which have a body with first block being correct syntax and return / yield / await anywhere in that body, won't have any keyword typo suggestions at all, and it's a pre-existing bug. Probably not worth fixing now, since it would either require creating a virtual async function block and another layer of offset translation or adding new flags to compile() to allow all function-only keywords be top-level (the second is a much cleaner fix)... Nevermind, we can just strip lines more intelligently when we create error_code!

bedevere-app · 2026-08-20T09:56:27Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

m-aciek

I found one regression, and left suggestion for the exception catch. It's nice it covers more cases, I hope it is possible to make it work with multiline still?

m-aciek · 2026-08-20T10:49:49Z

+        except Exception:
+            pass


Suggested change

except Exception:

pass

except tokenize.TokenError:

# Incomplete input can still contain useful tokens.

pass

(non-blocking) TokenError handling is safer than catch-all because it continues only for the incomplete-input condition where partial tokens are known to be useful.

There's a catch-all in the caller already

cpython/Lib/traceback.py

Line 1561 in 5a0ab76

with suppress(Exception):

In short: because we unroll the tokens early and process them later in potentially different order, it's best to catch-all on the unrolling part and then always move on to processing.

Details

The idea here is to avoid bailing out if an iteration of the loop fails for any reason, and still process any tokens gathered up until that failure.

While TokenError might be the only possible exception reachable here right now, the invariant is broader than that. The intent is to avoid bailing in case of any exception raised while iterating in this loop, since there can still be tokens to test prior to failure.

To illustrate this: if we kept the status quo, whatever exception during iterating wouldn't stop tokens from being processed, and we want to preserve this property here regardless of the current implementation details. While we may argue that this code will never raise anything else than TokenError ever, broader guard reflects the intent clearer, and is immune to drifts in the implementation details (i.e. from some point in the future something other than TokenError might be raised).

There's a catch-all in the caller already

cpython/Lib/traceback.py

Line 1561 in 5a0ab76

with suppress(Exception):

In this PR, the catch-all in the caller becomes irrelevant to avoiding bailing out specifically before the tokens are processed. The purpose is to ignore whatever exception and still process any tokens that we managed to collect prior to failure.

m-aciek · 2026-08-20T11:18:03Z

+                # Only consider NAME tokens on the same line as the error
+                the_end = end_line if line == 0 else end_line + 1
+                if token.start[0] + line != the_end:
+                    continue


Restricting ourselves to only tokens on the same line makes us ignore valid suggestions:

iff \ x: pass

The parser reports line 2, but the typo is on line 1.

main: Did you mean 'if'?

PR: no suggestion

Candidates on the error line should be prioritized, not made the exclusive candidate set.

And we should cover this case as well.

johnslavik · 2026-08-20T17:15:40Z

            if hint:
                matches.append(hint)
            if _suggestions is not None:
                suggestion = _suggestions._generate_suggestions(keyword.kwlist + keyword.softkwlist, wrong_name)
                if suggestion:
                    matches.append(suggestion)
            matches.extend(
                difflib.get_close_matches(
                    wrong_name,
                    keyword.kwlist + keyword.softkwlist,
                    n=max_matches,
                    cutoff=0.5
                )
            )


This had a pre-existing bug of accumulating duplicate replacement candidates.

E.g. for retrun, return will be checked twice (matches will contain 2 'return' strings)!

I'd like to fix it in this PR too.

Rank tokens before probing for a keyword typo

b9c5baa

bedevere-app Bot added the awaiting review label Aug 20, 2026

johnslavik changed the title ~~Rank tokens before probing for a keyword typo~~ gh-156047: Rank tokens before probing for a keyword typo Aug 20, 2026

bedevere-app Bot mentioned this pull request Aug 20, 2026

Keyword typo suggestions differ between file and string tokenizers #156047

Open

johnslavik requested a review from pablogsal August 20, 2026 09:15

johnslavik commented Aug 20, 2026

View reviewed changes

Comment thread Lib/traceback.py

Check token string first, then do the line math

fdcc3a0

pablogsal requested changes Aug 20, 2026

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting review labels Aug 20, 2026

m-aciek reviewed Aug 20, 2026

View reviewed changes

johnslavik commented Aug 20, 2026

View reviewed changes

-        except Exception:
-            pass
+        except tokenize.TokenError:
+            # Incomplete input can still contain useful tokens.
+            pass

Uh oh!

Uh oh!

Conversation

johnslavik commented Aug 20, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

johnslavik commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogsal Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented Aug 20, 2026

Uh oh!

m-aciek left a comment

Choose a reason for hiding this comment

Uh oh!

m-aciek Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

m-aciek Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

johnslavik commented Aug 20, 2026 •

edited by bedevere-app Bot

Loading

johnslavik commented Aug 20, 2026 •

edited

Loading

johnslavik Aug 20, 2026 •

edited

Loading

johnslavik Aug 20, 2026 •

edited

Loading

johnslavik Aug 20, 2026 •

edited

Loading

johnslavik Aug 20, 2026 •

edited

Loading

johnslavik Aug 20, 2026 •

edited

Loading