org.apache.flex.compiler.internal.parsing.as

Class BaseTokenizerWithFakeCharacters

java.lang.Object

org.apache.flex.compiler.internal.parsing.as.BaseTokenizerWithFakeCharacters

public abstract class BaseTokenizerWithFakeCharacters
extends Object

Constructor Summary

Constructors

Constructor and Description
BaseTokenizerWithFakeCharacters()

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract TokenBase	buildToken(int tokenType, int start, int end, int line, int column, String tokenText) Build a token of the appropriate type (used when we need to truncate a token to strip out fake characters in removeFakeTokens)
protected void	removeFakeTokens(int lastRealIndex, List<TokenBase> tokenList) Remove any dummy tokens that we created using the fake characters we appended to the end of the stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BaseTokenizerWithFakeCharacters

public BaseTokenizerWithFakeCharacters()

Method Detail

buildToken

protected abstract TokenBase buildToken(int tokenType,
                   int start,
                   int end,
                   int line,
                   int column,
                   String tokenText)

Build a token of the appropriate type (used when we need to truncate a token to strip out fake characters in removeFakeTokens)

Parameters:

tokenType - type of token

start - start of token

end - end of token

line - line of token

tokenText - text of token

Returns:

token of the appropriate type (ASToken, CSSToken, MXMLToken, etc.)

removeFakeTokens

protected void removeFakeTokens(int lastRealIndex,
                    List<TokenBase> tokenList)

Remove any dummy tokens that we created using the fake characters we appended to the end of the stream. If there is a token that contains both real characters and fake characters, we'll truncate that token to only include real characters

Parameters:

lastRealIndex - index of last real characters

tokenList - complete token list