token_get_all

I wanted to use the tokenizer functions to count source lines of code, including counting comments.  Attempting to do this with regular expressions does not work well because of situations where /* appears in a string, or other situations.  The token_get_all() function makes this task easy by detecting all the comments properly.  However, it does not tokenize newline characters.  I wrote the below set of functions to also tokenize newline characters as T_NEW_LINE.<?phpdefine('T_NEW_LINE', -1);function token_get_all_nl($source){    $new_tokens = array();    // Get the tokens    $tokens = token_get_all($source);    // Split newlines into their own tokens    foreach ($tokens as $token)    {        $token_name = is_array($token) ? $token[0] : null;        $token_data = is_array($token) ? $token[1] : $token;        // Do not split encapsed strings or multiline comments        if ($token_name == T_CONSTANT_ENCAPSED_STRING || substr($token_data, 0, 2) == '/*')        {            $new_tokens[] = array($token_name, $token_data);            continue;        }        // Split the data up by newlines        $split_data = preg_split('#(\r\n|\n)#', $token_data, -1, PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY);        foreach ($split_data as $data)        {            if ($data == "\r\n" || $data == "\n")            {                // This is a new line token                $new_tokens[] = array(T_NEW_LINE, $data);            }            else            {                // Add the token under the original token name                $new_tokens[] = is_array($token) ? array($token_name, $data) : $data;            }        }    }    return $new_tokens;}function token_name_nl($token){    if ($token === T_NEW_LINE)    {        return 'T_NEW_LINE';    }    return token_name($token);}?>Example usage:<?php$tokens = token_get_all_nl(file_get_contents('somecode.php'));foreach ($tokens as $token){    if (is_array($token))    {        echo (token_name_nl($token[0]) . ': "' . $token[1] . '"<br />');    }    else    {        echo ('"' . $token . '"<br />');    }}?>I'm sure you can figure out how to count the lines of code, and lines of comments with these functions.  This was a huge improvement on my previous attempt at counting lines of code with regular expressions.  I hope this helps someone, as many of the user contributed examples on this website have helped me in the past.

Yes, some problems (On WAMP, PHP 5.3.0 ) with get_token_all() 1 : bug line numbers Since PHP 5.2.2 token_get_all()  should return Line numbers in element 2.... but for instance (5.3.0 on WAMP), it work perfectly only with PHP code (not HMTL miwed), but if you have some T_INLINE_HTML detected by token_get_all() ,  sometimes you find wrongs line numbers  (return next line)... :(2: bug warning message can impact loopsWarning with php code uncompleted (ex : php code line by line) :for example if a comment tag is not closed  token_get_all()  can block loops on this  warning :Warning: Unterminated comment starting lineThis problem seem not occur in CLI mod (php command line), but only in web mod.Waiting more stability, used token_get_all()  only on PHP code (not HMTL miwed) :First extract entirely PHP code (with open et close php tag), Second use token_get_all()  on the pure PHP code.3 : Why there not function to extract PHP code (to extract HTML, we have Tidy..)?Waiting, I used a function :The code at end this post :http://www.developpez.net/forums/d786381/php/langage/fonctions/analyser-fichier-php-token_get_all/This function not support :- Old notation :  "<?  ?>" and "<% %>"- heredoc syntax - nowdoc syntax (since PHP 5.3.0)

As a caution: when using TOKEN_PARSE with an invalid php-file, one can get an error like this:Parse error: syntax error, unexpected '__construct' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST) in  on line 15Notice the missing filename as this function accepts a string, not a filename and thus has no idea of the latter.However an exception would be more appreciated.

The T_OPEN_TAG token will include the first trailing newline (\r, \n, or \r\n), tab (\t), or space. Any additional space after this token will be in a T_WHITESPACE token.The T_CLOSE_TAG token will include the first trailing newline (\r, \n, or \r\n; as described here http://php.net/manual/en/language.basic-syntax.instruction-separation.php). Any additional space after this token will be in a T_INLINE_HTML token.

Not all tokens are returned as an array. The rule appears to be that if a token is not variable, but instead it is one particular constant string, it is returned as a string instead. You don't get a line number. This is the case for braces( "{", "}"), parentheses ("(", ")"), brackets ("[", "]"), comma (","), semi-colon (";"), and a whole slew of operator signs ("!", "=", "+", "*", "/", ".", "+=", ...).

Well, there is a way to parse for errors. Seehttp://www.php.net/manual/function.php-check-syntax.php#77318