>> But removing comments is not blind-- it does take some account of
tokenization.
>> For example,
>> a // inside a string does not begin a comment,
>> a // inside an escaped identifier does not begin a comment
>> etc.
Note that in these cases, the lexer is already in the middle of a
token when it finds the // (a string literal or escaped identifier
token). They are just legal parts of the token it is already working
on. This does not take any special handling. When a lexer is reading
an identifier like net02, it doesn't get confused by the fact that a
subpart of the token looks like an integer literal. It is just a legal
continuation of the current token.
Keeping track of where it is in lexing the current token is the normal
job of the finite state machine of a typical lexer. But looking ahead
and trying out different possible tokenizations to see if one of them
will allow a legal parse is not.
Steven Sharp
sharp@cadence.com
Received on Mon Aug 30 11:26:44 2004
This archive was generated by hypermail 2.1.8 : Mon Aug 30 2004 - 11:27:01 PDT