org.apache.flex.compiler.internal.parsing.as

Class StreamingASTokenizer

java.lang.Object

org.apache.flex.compiler.internal.parsing.as.StreamingASTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable, ASTokenTypes, IASTokenizer

public class StreamingASTokenizer
extends Object
implements ASTokenTypes, IASTokenizer, Closeable

This Tokenizer provides tokens to be used by various clients, most notably the ASParser. Given the nature of ambiguities in the ActionScript 3 language, this tokenizer also serves to disambiguate tokens based on a combination of look behind and lookahead. For all cases of ambiguity, only one token is needed for look behind, and in our worst case, n tokens forwards where n is the number of tokens that can be produced. Some other state is kept in order to know which type of container we may exist in (function, class, interface, etc). We buffer LA token results to avoid unneeded lookahead

Field Summary

Fields inherited from interface org.apache.flex.compiler.internal.parsing.as.ASTokenTypes

EOF, HIDDEN_TOKEN_BUILTIN_NS, HIDDEN_TOKEN_COMMENT, HIDDEN_TOKEN_E4X, HIDDEN_TOKEN_MULTI_LINE_COMMENT, HIDDEN_TOKEN_SINGLE_LINE_COMMENT, HIDDEN_TOKEN_STAR_ASSIGNMENT, NULL_TREE_LOOKAHEAD, TOKEN_ASDOC_COMMENT, TOKEN_ASDOC_TAG, TOKEN_ASDOC_TEXT, TOKEN_ATTRIBUTE, TOKEN_BLOCK_CLOSE, TOKEN_BLOCK_OPEN, TOKEN_COLON, TOKEN_COMMA, TOKEN_DIRECTIVE_DEFAULT_XML, TOKEN_E4X_BINDING_CLOSE, TOKEN_E4X_BINDING_OPEN, TOKEN_E4X_CDATA, TOKEN_E4X_CLOSE_TAG_START, TOKEN_E4X_COMMENT, TOKEN_E4X_DECIMAL_ENTITY, TOKEN_E4X_DOTTED_NAME_PART, TOKEN_E4X_EMPTY_TAG_END, TOKEN_E4X_ENTITY, TOKEN_E4X_EQUALS, TOKEN_E4X_HEX_ENTITY, TOKEN_E4X_NAME, TOKEN_E4X_NAME_DOT, TOKEN_E4X_OPEN_TAG_START, TOKEN_E4X_PROCESSING_INSTRUCTION, TOKEN_E4X_STRING, TOKEN_E4X_TAG_END, TOKEN_E4X_TEXT, TOKEN_E4X_WHITESPACE, TOKEN_E4X_XMLLIST_CLOSE, TOKEN_E4X_XMLNS, TOKEN_ELLIPSIS, TOKEN_IDENTIFIER, TOKEN_KEYWORD_AS, TOKEN_KEYWORD_BREAK, TOKEN_KEYWORD_CASE, TOKEN_KEYWORD_CATCH, TOKEN_KEYWORD_CLASS, TOKEN_KEYWORD_CONST, TOKEN_KEYWORD_CONTINUE, TOKEN_KEYWORD_DEFAULT, TOKEN_KEYWORD_DELETE, TOKEN_KEYWORD_DO, TOKEN_KEYWORD_ELSE, TOKEN_KEYWORD_FALSE, TOKEN_KEYWORD_FINALLY, TOKEN_KEYWORD_FOR, TOKEN_KEYWORD_FUNCTION, TOKEN_KEYWORD_IF, TOKEN_KEYWORD_IMPORT, TOKEN_KEYWORD_IN, TOKEN_KEYWORD_INCLUDE, TOKEN_KEYWORD_INSTANCEOF, TOKEN_KEYWORD_INTERFACE, TOKEN_KEYWORD_IS, TOKEN_KEYWORD_NEW, TOKEN_KEYWORD_NULL, TOKEN_KEYWORD_PACKAGE, TOKEN_KEYWORD_RETURN, TOKEN_KEYWORD_SUPER, TOKEN_KEYWORD_SWITCH, TOKEN_KEYWORD_THIS, TOKEN_KEYWORD_THROW, TOKEN_KEYWORD_TRUE, TOKEN_KEYWORD_TRY, TOKEN_KEYWORD_TYPEOF, TOKEN_KEYWORD_USE, TOKEN_KEYWORD_VAR, TOKEN_KEYWORD_VOID, TOKEN_KEYWORD_WHILE, TOKEN_KEYWORD_WITH, TOKEN_LITERAL_HEX_NUMBER, TOKEN_LITERAL_NUMBER, TOKEN_LITERAL_REGEXP, TOKEN_LITERAL_STRING, TOKEN_LITERAL_XMLLIST, TOKEN_MODIFIER_DYNAMIC, TOKEN_MODIFIER_FINAL, TOKEN_MODIFIER_NATIVE, TOKEN_MODIFIER_OVERRIDE, TOKEN_MODIFIER_STATIC, TOKEN_MODIFIER_VIRTUAL, TOKEN_NAMESPACE_ANNOTATION, TOKEN_NAMESPACE_NAME, TOKEN_OPERATOR_ASSIGNMENT, TOKEN_OPERATOR_ATSIGN, TOKEN_OPERATOR_BITWISE_AND, TOKEN_OPERATOR_BITWISE_AND_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_LEFT_SHIFT, TOKEN_OPERATOR_BITWISE_LEFT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_NOT, TOKEN_OPERATOR_BITWISE_OR, TOKEN_OPERATOR_BITWISE_OR_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_RIGHT_SHIFT, TOKEN_OPERATOR_BITWISE_RIGHT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_UNSIGNED_RIGHT_SHIFT, TOKEN_OPERATOR_BITWISE_UNSIGNED_RIGHT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_XOR, TOKEN_OPERATOR_BITWISE_XOR_ASSIGNMENT, TOKEN_OPERATOR_DECREMENT, TOKEN_OPERATOR_DESCENDANT_ACCESS, TOKEN_OPERATOR_DIVISION, TOKEN_OPERATOR_DIVISION_ASSIGNMENT, TOKEN_OPERATOR_EQUAL, TOKEN_OPERATOR_GREATER_THAN, TOKEN_OPERATOR_GREATER_THAN_EQUALS, TOKEN_OPERATOR_INCREMENT, TOKEN_OPERATOR_LESS_THAN, TOKEN_OPERATOR_LESS_THAN_EQUALS, TOKEN_OPERATOR_LOGICAL_AND, TOKEN_OPERATOR_LOGICAL_AND_ASSIGNMENT, TOKEN_OPERATOR_LOGICAL_NOT, TOKEN_OPERATOR_LOGICAL_OR, TOKEN_OPERATOR_LOGICAL_OR_ASSIGNMENT, TOKEN_OPERATOR_MEMBER_ACCESS, TOKEN_OPERATOR_MINUS, TOKEN_OPERATOR_MINUS_ASSIGNMENT, TOKEN_OPERATOR_MODULO, TOKEN_OPERATOR_MODULO_ASSIGNMENT, TOKEN_OPERATOR_MULTIPLICATION_ASSIGNMENT, TOKEN_OPERATOR_NOT_EQUAL, TOKEN_OPERATOR_NS_QUALIFIER, TOKEN_OPERATOR_PLUS, TOKEN_OPERATOR_PLUS_ASSIGNMENT, TOKEN_OPERATOR_STAR, TOKEN_OPERATOR_STRICT_EQUAL, TOKEN_OPERATOR_STRICT_NOT_EQUAL, TOKEN_OPERATOR_TERNARY, TOKEN_PAREN_CLOSE, TOKEN_PAREN_OPEN, TOKEN_RESERVED_WORD_CONFIG, TOKEN_RESERVED_WORD_EACH, TOKEN_RESERVED_WORD_EXTENDS, TOKEN_RESERVED_WORD_GET, TOKEN_RESERVED_WORD_GOTO, TOKEN_RESERVED_WORD_IMPLEMENTS, TOKEN_RESERVED_WORD_NAMESPACE, TOKEN_RESERVED_WORD_SET, TOKEN_SEMICOLON, TOKEN_SQUARE_CLOSE, TOKEN_SQUARE_OPEN, TOKEN_TYPED_COLLECTION_CLOSE, TOKEN_TYPED_COLLECTION_OPEN, TOKEN_TYPED_LITERAL_CLOSE, TOKEN_TYPED_LITERAL_OPEN, TOKEN_VOID_0

Constructor Summary

Constructors

Constructor and Description
StreamingASTokenizer() You should probably not use this constructor.
StreamingASTokenizer(Reader reader) You should probably not use this constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Closes the underlying reader
protected static StreamingASTokenizer	create(IFileSpecification fileSpec, IncludeHandler includeHandler) Create a tokenizer from a source file.
protected static StreamingASTokenizer	createForASParser(IFileSpecification fileSpec, IncludeHandler includeHandler, boolean followIncludes, List<String> includedFiles) Create a tokenizer for ASParser#parseFile().
protected static StreamingASTokenizer	createForInlineExpressionParsing(Reader reader, String path) Create a tokenizer to parse an Expression.
protected static StreamingASTokenizer	createForInlineScriptScopeBuilding(Reader reader, String path, IncludeHandler includeHandler, int offsetAdjustment, int lineAdjustment, int columnAdjustment) This creator doesn't "enter file" on creation.
static StreamingASTokenizer	createForMXMLIndexing(String fileName) Creates a tokenizer suitable for the mxml indexing code. fragments the new tokenizer will tokenize.
static StreamingASTokenizer	createForRepairingASTokenizer(Reader reader, String path, IncludeHandler includeHandler) This method can create a StreamingASTokenizer with optional "follow includes".
int	getEndOffset() Retrieve the end offset of the file.
String	getSourcePath() Gets the source path to the file being tokenized.
List<ICompilerProblem>	getTokenizationProblems() Returns a collection of problems that have been encountered while scanning.
ASToken[]	getTokens(Reader reader) Returns an array of IASToken objects for a given range of text represented by a Reader
ASToken[]	getTokens(Reader reader, ITokenStreamFilter filter)
IASToken[]	getTokens(String range) Returns an array of IASToken objects for a given range of text represented by a String
boolean	hasEncounteredIncludeStatements() Indicated whether this tokenizer has encountered include statements, regardless of whether it is set to follow them or not
boolean	hasTokenizationProblems() Indicated that we have tokenization problems.
ASToken	next() Returns the next token that can be produced from the underlying reader.
ASToken	next(ITokenStreamFilter filter) Returns the next token that can be produced from the underlying reader
void	setCollectComments(boolean collect) Sets whether we comments are collected: single line and multi-line.
void	setFollowIncludes(boolean followIncludes) Sets whether we follow include statements, including their tokens.
void	setIgnoreKeywords(boolean ignore) Sets whether we ignore keywords while scanning.
void	setIncludeHandler(IncludeHandler handler) Sets the include handler used by this tokenizer to get IFileSpecification for included files.
void	setIsMetadataAware(boolean aware) Sets whether we will find metadata constructs Default is true
void	setPath(String path) Sets the path to the file this tokenizer is scanning
void	setReader(Reader reader) Sets the Reader that supplies the content to this tokenizer.
void	setReader(Reader reader, int offset, int line, int column) Sets the Reader that supplies the content to this tokenizer.
void	setScanningFullContent(boolean full) Sets whether we are scanning a full file, or a fragment.
void	setSourcePositionAdjustment(int offset, int line, int column) Allows for the adjustment of offset, line and column information when parsing subsequences of text.
void	setTokenFilter(ITokenStreamFilter filter) Sets the ITokenStreamFilter used to filter out unwanted tokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingASTokenizer

public StreamingASTokenizer(Reader reader)

You should probably not use this constructor. There is some legacy code that uses this constructor, but that code should be updated to use one of the static create methods below.

TODO: make this private.

StreamingASTokenizer

public StreamingASTokenizer()

You should probably not use this constructor. There is a lot of code that uses this constructor, but that code should be updated to use one of the static create methods below.

TODO: make this private.

Method Detail

createForMXMLIndexing

public static StreamingASTokenizer createForMXMLIndexing(String fileName)

Creates a tokenizer suitable for the mxml indexing code. fragments the new tokenizer will tokenize.

Returns:

A new tokenizer suitable for tokenizing script fragments in an mxml document that is being tokenized for the full text search index.

create

protected static StreamingASTokenizer create(IFileSpecification fileSpec,
                          IncludeHandler includeHandler)
                                      throws FileNotFoundException

Create a tokenizer from a source file. This is the lexer entry-point used by ASCompilationUnit.

Parameters:

fileSpec - File specification provides the reader and the file path.

includeHandler - Include handler.

Throws:

FileNotFoundException - error

createForASParser

protected static StreamingASTokenizer createForASParser(IFileSpecification fileSpec,
                                     IncludeHandler includeHandler,
                                     boolean followIncludes,
                                     List<String> includedFiles)
                                                 throws FileNotFoundException

Create a tokenizer for ASParser#parseFile().

Parameters:

fileSpec - File specification provides the reader and the file path.

includeHandler - Include handler.

followIncludes - True if included files are also parsed.

includedFiles - A list of included file paths.

Returns:

Lexer.

Throws:

FileNotFoundException - error

createForInlineScriptScopeBuilding

protected static StreamingASTokenizer createForInlineScriptScopeBuilding(Reader reader,
                                                      String path,
                                                      IncludeHandler includeHandler,
                                                      int offsetAdjustment,
                                                      int lineAdjustment,
                                                      int columnAdjustment)

This creator doesn't "enter file" on creation.

createForInlineExpressionParsing

protected static StreamingASTokenizer createForInlineExpressionParsing(Reader reader,
                                                    String path)

Create a tokenizer to parse an Expression.

createForRepairingASTokenizer

public static StreamingASTokenizer createForRepairingASTokenizer(Reader reader,
                                                 String path,
                                                 IncludeHandler includeHandler)

This method can create a StreamingASTokenizer with optional "follow includes". If IncludeHandler is not null, it will follow include directives.

Parameters:

reader - Input to the tokenizer.

path - File path of the input.

includeHandler - If not null, the created tokenizer will follow include directives.

Returns:

A StreamingASTokenizer.

setReader

public void setReader(Reader reader)

Sets the Reader that supplies the content to this tokenizer. It is up to the client to close any previous readers that have been in use. It is also up to the client to close the reader once it has been used

Parameters:

reader - a Reader

setReader

public void setReader(Reader reader,
             int offset,
             int line,
             int column)

Sets the Reader that supplies the content to this tokenizer. It is up to the client to close any previous readers that have been in use. It is also up to the client to close the reader once it has been used

Parameters:

reader - a Reader

offset - Offset adjustment. If the specified reader is reading from a string extracted from a source file, this should be the offset of the first character read from the reader in the source file.

line - Line adjustment.

column - Column adjustment

setPath

public void setPath(String path)

Sets the path to the file this tokenizer is scanning

Specified by:

setPath in interface IASTokenizer

Parameters:

path - a file path

setSourcePositionAdjustment

public void setSourcePositionAdjustment(int offset,
                               int line,
                               int column)

Allows for the adjustment of offset, line and column information when parsing subsequences of text. This should be called before tokenization has started

Parameters:

offset - The offset where the fragment starts.

line - The line where the fragment starts. This should be a ZERO-based line number

column - The column where the fragment starts. This should be a ZERO-based column number

setCollectComments

public void setCollectComments(boolean collect)

Sets whether we comments are collected: single line and multi-line. Default is false

Specified by:

setCollectComments in interface IASTokenizer

Parameters:

collect - true if we should collect comments

setFollowIncludes

public void setFollowIncludes(boolean followIncludes)

Sets whether we follow include statements, including their tokens. Default is true

Specified by:

setFollowIncludes in interface IASTokenizer

Parameters:

followIncludes - true if we should follow includes

close

public void close()
           throws IOException

Closes the underlying reader

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

setIgnoreKeywords

public void setIgnoreKeywords(boolean ignore)

Sets whether we ignore keywords while scanning. Default is false

Parameters:

ignore - true if we should ignore keywords

setScanningFullContent

public void setScanningFullContent(boolean full)

Sets whether we are scanning a full file, or a fragment. Default is true

Parameters:

full - true if we are scanning a full file.

setIsMetadataAware

public void setIsMetadataAware(boolean aware)

Sets whether we will find metadata constructs Default is true

Parameters:

aware - true if we will find metadata

setTokenFilter

public void setTokenFilter(ITokenStreamFilter filter)

Sets the ITokenStreamFilter used to filter out unwanted tokens

Parameters:

filter - the token filter to alter the stream returned from the tokenizer

setIncludeHandler

public void setIncludeHandler(IncludeHandler handler)

Sets the include handler used by this tokenizer to get IFileSpecification for included files.

Parameters:

handler - IncludeHandler this tokenizer should use.

hasTokenizationProblems

public boolean hasTokenizationProblems()

Indicated that we have tokenization problems. Can be called once scanning has begun

Returns:

true if problems have been encountered

hasEncounteredIncludeStatements

public boolean hasEncounteredIncludeStatements()

Indicated whether this tokenizer has encountered include statements, regardless of whether it is set to follow them or not

Returns:

true if we have encountered includes

getTokenizationProblems

public List<ICompilerProblem> getTokenizationProblems()

Returns a collection of problems that have been encountered while scanning.

Returns:

a list of problems, never null

getTokens

public ASToken[] getTokens(Reader reader,
                  ITokenStreamFilter filter)

getTokens

public ASToken[] getTokens(Reader reader)

Description copied from interface: IASTokenizer

Returns an array of IASToken objects for a given range of text represented by a Reader

Specified by:

getTokens in interface IASTokenizer

Parameters:

reader - a Java Reader

Returns:

an array of IASToken objects

getTokens

public IASToken[] getTokens(String range)

Description copied from interface: IASTokenizer

Returns an array of IASToken objects for a given range of text represented by a String

Specified by:

getTokens in interface IASTokenizer

Returns:

an array of IASToken objects

next

public final ASToken next(ITokenStreamFilter filter)

Returns the next token that can be produced from the underlying reader

Parameters:

filter - an ITokenStreamFilter to restrict the tokens that are returned

Returns:

an ASToken, or null if no more tokens can be produced

next

public final ASToken next()

Returns the next token that can be produced from the underlying reader.

If the forked "include file tokenizer" is open (not null), return the next token from it. If the forked tokenizer reaches the end of the included file, close (set to null) the forked tokenizer and return token from the main source file.

Returns:

an ASToken, or null if no more tokens can be produced

getEndOffset

public final int getEndOffset()

Retrieve the end offset of the file.

The result is the end offset of the file, not the offset of the last token, this allows any trailing space to be included so that the parser can span the result FileNode to the entire file.

Returns:

the end offset of the input file

getSourcePath

public String getSourcePath()

Gets the source path to the file being tokenized.