T_AMPERSAND token

[staabm]

in sebastianbergmann/php-code-coverage#1125 we can see the token-names for tokenizing the file https://github.com/sebastianbergmann/php-code-coverage/blob/555cd5e6f887648198b96ed89c6880d6f61a0733/tests/_files/source_with_class_and_anonymous_function.php

changed

-        <token name="T_AMPERSAND_FOLLOWED_BY_VAR_OR_VARARG">&amp;</token>
+        <token name="T_AMPERSAND">&amp;</token>

with the 2.0 release.

T_AMPERSAND was a token type in PHP4.x and it only existed in theseer/tokenizer, because it was hard-coded in

tokenizer/src/Tokenizer.php

Line 35 in d1dd771

'&' => 'T_AMPERSAND',

since PHP5+ the builtin token names are
T_AMPERSAND_FOLLOWED_BY_VAR_OR_VARARG
and/or T_AMPERSAND_NOT_FOLLOWED_BY_VAR_OR_VARARG

https://3v4l.org/tU46U#veol

it looks like with 2.0.x we are now using T_AMPERSAND more often as we did with the 1.x counterpart.

---

I am not sure yet what todo about it, but wanted to leave it here so we can discuss the possible impact and/or a fix

[staabm]

dropping this line will make the phpunit/code-coverage regression go away:

tokenizer/src/Tokenizer.php

Line 35 in d1dd771

'&' => 'T_AMPERSAND',

I guess the question is, how we treat the token names in regard to backwards compatbility and whether we need/want to align the token names here with the token names in php-src

[theseer]

My assumption is - and I did not verify that in any way yet - this is caused by the switch to PHPToken::tokenize() and it behaving differently than the old token_get_all. I believe that actually was one reason to introduce the ::tokenize()-method into PHP in the first place, so we have a reliable object only result set?

If that is correct, we might not need the internal Map at all anymore.

So, if we stick with ::tokenize() I guess that's just a change we have to accept, maybe allowing us to clean up the mapping logic as the (use) case should no longer exist.

[theseer]

This class provides an alternative to token_get_all(). While the function returns tokens either as a single-character string, or an array with a token ID, token text and line number, PhpToken::tokenize() normalizes all tokens into PhpToken objects, which makes code operating on tokens more memory efficient and readable.

Reading the documentation helps :-)

So at least my memory was right.

[theseer]

Okay, a quick test shows the Object is equally useless when it comes to names, e.g. the ::name() method returns ) for example instead of an actual token name like T_CLOSE_BRACKET.

So I'm not sure as to what to do here :)

[theseer]

it looks like with 2.0.x we are now using T_AMPERSAND more often as we did with the 1.x counterpart.

To me, that would only make sense when the ::tokenize() result uses & as a name more frequently. Otherwise we did not change anything I can see that should be relevant to this.

[staabm]

I think easiest way forward would be to only keep the Tokenizer map for those single-characters which are still returned by getTokenName()

[theseer]

The leaves the question of which ones are that? ;-)

And, I guess we should change the map access to use ::getTokenName() then, rather than using ->text, right?

[staabm]

see e.g. https://3v4l.org/bfRMm#veol

[staabm]

::getTokenName() then, rather than using ->text, right?

yes

[theseer]

see e.g. https://3v4l.org/bfRMm#veol

Funny.

So from PHP 8.1 onwards, it appears that only the & has been removed. At least that's how I interprete the diff view on 3v4l.

Given we, for 2.0.0 require 8.1+, I guess that means we can indeed remove that line from our map.

[staabm]

oohh I think we can just do: if (isset(self::MAP[$tok->getTokenName()])) {

because of

Object is equally useless when it comes to names, e.g. the ::name() method returns ) for example instead of an actual token name like T_CLOSE_BRACKET.

[theseer]

I'll update the code.

[theseer]

I took the output for the 8.1+ from 3v4l and ran it through sort and uniq:

$ cat single_char.txt | sort | uniq

string(1) "!"
string(1) "%"
string(1) "("
string(1) ")"
string(1) "*"
string(1) "+"
string(1) ","
string(1) "-"
string(1) "/"
string(1) ";"
string(1) "<"
string(1) "="
string(1) ">"
string(1) "@"
string(1) "["
string(1) "]"
string(1) "^"
string(1) "`"
string(1) "{"
string(1) "|"
string(1) "}"
string(1) "~"

So that's 22 chars.
Our Map array still contains more entries after I removed the &:

        '(' => 'T_OPEN_BRACKET',
        ')' => 'T_CLOSE_BRACKET',
        '[' => 'T_OPEN_SQUARE',
        ']' => 'T_CLOSE_SQUARE',
        '{' => 'T_OPEN_CURLY',
        '}' => 'T_CLOSE_CURLY',
        ';' => 'T_SEMICOLON',
        '.' => 'T_DOT',
        ',' => 'T_COMMA',
        '=' => 'T_EQUAL',
        '<' => 'T_LT',
        '>' => 'T_GT',
        '+' => 'T_PLUS',
        '-' => 'T_MINUS',
        '*' => 'T_MULT',
        '/' => 'T_DIV',
        '?' => 'T_QUESTION_MARK',
        '!' => 'T_EXCLAMATION_MARK',
        ':' => 'T_COLON',
        '"' => 'T_DOUBLE_QUOTES',
        '@' => 'T_AT',
        '%' => 'T_PERCENT',
        '|' => 'T_PIPE',
        '$' => 'T_DOLLAR',
        '^' => 'T_CARET',
        '~' => 'T_TILDE',
        '`' => 'T_BACKTICK'

That implies we'd loose more of "our" Tokens.

Are we sure we want that?

[theseer]

The map lookup of course only would work as before if we stick to ->text I guess..