org.apache.flex.compiler.internal.parsing.as

Class RawASTokenizer

java.lang.Object

org.apache.flex.compiler.internal.parsing.as.BaseRawTokenizer<ASToken>

org.apache.flex.compiler.internal.parsing.as.BaseRawASTokenizer

org.apache.flex.compiler.internal.parsing.as.RawASTokenizer

All Implemented Interfaces:

ASTokenTypes

public final class RawASTokenizer
extends BaseRawASTokenizer
implements ASTokenTypes

Field Summary

Fields

Modifier and Type	Field and Description
static int	ASDOC_COMMENT
static int	CDATA
static int	CHARLITERAL_IGNORE
static int	COMMENT
static int	DIRECTIVE
static int	E4X
static int	E4XCHARLITERAL
static int	E4XCOMMENT
static int	E4XSTRINGLITERAL lexical states
static int	E4XTEXTVALUE
static int	ENTITY_RETURN
static int	ESCAPE_SEQUENCE
static int	MARKUP
static int	MARKUP_IGNORE
static int	STATE
static int	STRINGLITERAL
static int	STRINGLITERAL_IGNORE
static int	TYPED_COLLECTION
static int	TYPED_COLLECTION_LITERAL
static int	YYEOF This character denotes the end of file
static int	YYINITIAL

Fields inherited from class org.apache.flex.compiler.internal.parsing.as.BaseRawASTokenizer

bufferToken, collectComments, docTypeLevel, e4xBraceBalance, e4xReturnState, e4xTagDepth, isInCloseTag, typedDepth

Fields inherited from class org.apache.flex.compiler.internal.parsing.as.BaseRawTokenizer

aggregateContents, aggregateStart, aggregateStartColumn, aggregateStartLine

Fields inherited from interface org.apache.flex.compiler.internal.parsing.as.ASTokenTypes

EOF, HIDDEN_TOKEN_BUILTIN_NS, HIDDEN_TOKEN_COMMENT, HIDDEN_TOKEN_E4X, HIDDEN_TOKEN_MULTI_LINE_COMMENT, HIDDEN_TOKEN_SINGLE_LINE_COMMENT, HIDDEN_TOKEN_STAR_ASSIGNMENT, NULL_TREE_LOOKAHEAD, TOKEN_ASDOC_COMMENT, TOKEN_ASDOC_TAG, TOKEN_ASDOC_TEXT, TOKEN_ATTRIBUTE, TOKEN_BLOCK_CLOSE, TOKEN_BLOCK_OPEN, TOKEN_COLON, TOKEN_COMMA, TOKEN_DIRECTIVE_DEFAULT_XML, TOKEN_E4X_BINDING_CLOSE, TOKEN_E4X_BINDING_OPEN, TOKEN_E4X_CDATA, TOKEN_E4X_CLOSE_TAG_START, TOKEN_E4X_COMMENT, TOKEN_E4X_DECIMAL_ENTITY, TOKEN_E4X_DOTTED_NAME_PART, TOKEN_E4X_EMPTY_TAG_END, TOKEN_E4X_ENTITY, TOKEN_E4X_EQUALS, TOKEN_E4X_HEX_ENTITY, TOKEN_E4X_NAME, TOKEN_E4X_NAME_DOT, TOKEN_E4X_OPEN_TAG_START, TOKEN_E4X_PROCESSING_INSTRUCTION, TOKEN_E4X_STRING, TOKEN_E4X_TAG_END, TOKEN_E4X_TEXT, TOKEN_E4X_WHITESPACE, TOKEN_E4X_XMLLIST_CLOSE, TOKEN_E4X_XMLNS, TOKEN_ELLIPSIS, TOKEN_IDENTIFIER, TOKEN_KEYWORD_AS, TOKEN_KEYWORD_BREAK, TOKEN_KEYWORD_CASE, TOKEN_KEYWORD_CATCH, TOKEN_KEYWORD_CLASS, TOKEN_KEYWORD_CONST, TOKEN_KEYWORD_CONTINUE, TOKEN_KEYWORD_DEFAULT, TOKEN_KEYWORD_DELETE, TOKEN_KEYWORD_DO, TOKEN_KEYWORD_ELSE, TOKEN_KEYWORD_FALSE, TOKEN_KEYWORD_FINALLY, TOKEN_KEYWORD_FOR, TOKEN_KEYWORD_FUNCTION, TOKEN_KEYWORD_IF, TOKEN_KEYWORD_IMPORT, TOKEN_KEYWORD_IN, TOKEN_KEYWORD_INCLUDE, TOKEN_KEYWORD_INSTANCEOF, TOKEN_KEYWORD_INTERFACE, TOKEN_KEYWORD_IS, TOKEN_KEYWORD_NEW, TOKEN_KEYWORD_NULL, TOKEN_KEYWORD_PACKAGE, TOKEN_KEYWORD_RETURN, TOKEN_KEYWORD_SUPER, TOKEN_KEYWORD_SWITCH, TOKEN_KEYWORD_THIS, TOKEN_KEYWORD_THROW, TOKEN_KEYWORD_TRUE, TOKEN_KEYWORD_TRY, TOKEN_KEYWORD_TYPEOF, TOKEN_KEYWORD_USE, TOKEN_KEYWORD_VAR, TOKEN_KEYWORD_VOID, TOKEN_KEYWORD_WHILE, TOKEN_KEYWORD_WITH, TOKEN_LITERAL_HEX_NUMBER, TOKEN_LITERAL_NUMBER, TOKEN_LITERAL_REGEXP, TOKEN_LITERAL_STRING, TOKEN_LITERAL_XMLLIST, TOKEN_MODIFIER_DYNAMIC, TOKEN_MODIFIER_FINAL, TOKEN_MODIFIER_NATIVE, TOKEN_MODIFIER_OVERRIDE, TOKEN_MODIFIER_STATIC, TOKEN_MODIFIER_VIRTUAL, TOKEN_NAMESPACE_ANNOTATION, TOKEN_NAMESPACE_NAME, TOKEN_OPERATOR_ASSIGNMENT, TOKEN_OPERATOR_ATSIGN, TOKEN_OPERATOR_BITWISE_AND, TOKEN_OPERATOR_BITWISE_AND_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_LEFT_SHIFT, TOKEN_OPERATOR_BITWISE_LEFT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_NOT, TOKEN_OPERATOR_BITWISE_OR, TOKEN_OPERATOR_BITWISE_OR_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_RIGHT_SHIFT, TOKEN_OPERATOR_BITWISE_RIGHT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_UNSIGNED_RIGHT_SHIFT, TOKEN_OPERATOR_BITWISE_UNSIGNED_RIGHT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_XOR, TOKEN_OPERATOR_BITWISE_XOR_ASSIGNMENT, TOKEN_OPERATOR_DECREMENT, TOKEN_OPERATOR_DESCENDANT_ACCESS, TOKEN_OPERATOR_DIVISION, TOKEN_OPERATOR_DIVISION_ASSIGNMENT, TOKEN_OPERATOR_EQUAL, TOKEN_OPERATOR_GREATER_THAN, TOKEN_OPERATOR_GREATER_THAN_EQUALS, TOKEN_OPERATOR_INCREMENT, TOKEN_OPERATOR_LESS_THAN, TOKEN_OPERATOR_LESS_THAN_EQUALS, TOKEN_OPERATOR_LOGICAL_AND, TOKEN_OPERATOR_LOGICAL_AND_ASSIGNMENT, TOKEN_OPERATOR_LOGICAL_NOT, TOKEN_OPERATOR_LOGICAL_OR, TOKEN_OPERATOR_LOGICAL_OR_ASSIGNMENT, TOKEN_OPERATOR_MEMBER_ACCESS, TOKEN_OPERATOR_MINUS, TOKEN_OPERATOR_MINUS_ASSIGNMENT, TOKEN_OPERATOR_MODULO, TOKEN_OPERATOR_MODULO_ASSIGNMENT, TOKEN_OPERATOR_MULTIPLICATION_ASSIGNMENT, TOKEN_OPERATOR_NOT_EQUAL, TOKEN_OPERATOR_NS_QUALIFIER, TOKEN_OPERATOR_PLUS, TOKEN_OPERATOR_PLUS_ASSIGNMENT, TOKEN_OPERATOR_STAR, TOKEN_OPERATOR_STRICT_EQUAL, TOKEN_OPERATOR_STRICT_NOT_EQUAL, TOKEN_OPERATOR_TERNARY, TOKEN_PAREN_CLOSE, TOKEN_PAREN_OPEN, TOKEN_RESERVED_WORD_CONFIG, TOKEN_RESERVED_WORD_EACH, TOKEN_RESERVED_WORD_EXTENDS, TOKEN_RESERVED_WORD_GET, TOKEN_RESERVED_WORD_GOTO, TOKEN_RESERVED_WORD_IMPLEMENTS, TOKEN_RESERVED_WORD_NAMESPACE, TOKEN_RESERVED_WORD_SET, TOKEN_SEMICOLON, TOKEN_SQUARE_CLOSE, TOKEN_SQUARE_OPEN, TOKEN_TYPED_COLLECTION_CLOSE, TOKEN_TYPED_COLLECTION_OPEN, TOKEN_TYPED_LITERAL_CLOSE, TOKEN_TYPED_LITERAL_OPEN, TOKEN_VOID_0

Constructor Summary

Constructors

Constructor and Description
RawASTokenizer()
RawASTokenizer(InputStream in) Creates a new scanner.
RawASTokenizer(Reader in) Creates a new scanner There is also a java.io.InputStream version of this constructor.

Method Summary

Methods

Modifier and Type	Method and Description
protected char[]	buffer() returns the current active buffer.
protected int	currentPos() returns the current position into the buffer
protected int	getColumn() Gets the current column number of the tokenizer.
protected int	getLine() Gets the current line number of the tokenizer.
protected int	getOffset() Gets the current offset of the tokenizer into the file buffer.
protected int	markedPosition() returns the marked position in the current read
ASToken	nextToken() Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
protected int	readEnd() returns the end of the current read
protected int	readStart() returns the start of the current read
protected void	setAllowLTInE4XStringLiterals(boolean allow) in E4X, < is allowed in char and string literals.
protected void	yybegin(int newState) Enters a new lexical state
protected char	yycharat(int pos) Returns the character at position pos from the matched text.
void	yyclose() Closes the input stream.
protected int	yylength() Returns the length of the matched text region.
void	yypushback(int number) Pushes the specified amount of characters back into the input stream.
void	yyreset(Reader reader) Resets the scanner to read from a new input stream.
protected int	yystate() Returns the current lexical state.
protected String	yytext() Returns the text matched by the current regular expression.

Methods inherited from class org.apache.flex.compiler.internal.parsing.as.BaseRawASTokenizer

aggregateEscapedUnicodeChar, buildAggregateToken, buildE4XTextToken, continueAggregate, decodeEscapedUnicode, escapedUnicodeToHtmlEntity, fillBuffer, getBufferToken, hasBufferToken, initTokenPool, isCollectingComments, isInE4XDatabinding, isInXML, newToken, reportInvalidLineTerminatorInStringLiteral, reportUnclosedASDoc, reportUnclosedStringLiteral, reset, setCollectComments

Methods inherited from class org.apache.flex.compiler.internal.parsing.as.BaseRawTokenizer

addBadCharacterProblem, buildToken, buildToken, buildToken, continueAggregate, continueAggregate, continueAggregate, continueAggregate, fetchToken, getContext, getCurrentSourceLocation, getLastTokenText, getLastTokenType, getProblems, getSourcePath, hasAggregateContents, hasProblems, setLastToken, setReuseLastToken, setSourcePath, startAggregate, startAggregate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

YYEOF

public static final int YYEOF

This character denotes the end of file

See Also:

Constant Field Values

E4XSTRINGLITERAL

public static final int E4XSTRINGLITERAL

lexical states

See Also:

Constant Field Values

CHARLITERAL_IGNORE

public static final int CHARLITERAL_IGNORE

See Also:

Constant Field Values

E4XTEXTVALUE

public static final int E4XTEXTVALUE

See Also:

Constant Field Values

DIRECTIVE

public static final int DIRECTIVE

See Also:

Constant Field Values

MARKUP

public static final int MARKUP

See Also:

Constant Field Values

E4XCHARLITERAL

public static final int E4XCHARLITERAL

See Also:

Constant Field Values

STRINGLITERAL

public static final int STRINGLITERAL

See Also:

Constant Field Values

ESCAPE_SEQUENCE

public static final int ESCAPE_SEQUENCE

See Also:

Constant Field Values

ENTITY_RETURN

public static final int ENTITY_RETURN

See Also:

Constant Field Values

TYPED_COLLECTION

public static final int TYPED_COLLECTION

See Also:

Constant Field Values

CDATA

public static final int CDATA

See Also:

Constant Field Values

COMMENT

public static final int COMMENT

See Also:

Constant Field Values

STRINGLITERAL_IGNORE

public static final int STRINGLITERAL_IGNORE

See Also:

Constant Field Values

MARKUP_IGNORE

public static final int MARKUP_IGNORE

See Also:

Constant Field Values

STATE

public static final int STATE

See Also:

Constant Field Values

E4XCOMMENT

public static final int E4XCOMMENT

See Also:

Constant Field Values

TYPED_COLLECTION_LITERAL

public static final int TYPED_COLLECTION_LITERAL

See Also:

Constant Field Values

YYINITIAL

public static final int YYINITIAL

See Also:

Constant Field Values

ASDOC_COMMENT

public static final int ASDOC_COMMENT

See Also:

Constant Field Values

E4X

public static final int E4X

See Also:

Constant Field Values

Constructor Detail

RawASTokenizer

public RawASTokenizer()

RawASTokenizer

public RawASTokenizer(Reader in)

Creates a new scanner There is also a java.io.InputStream version of this constructor.

Parameters:

in - the java.io.Reader to read input from.

RawASTokenizer

public RawASTokenizer(InputStream in)

Creates a new scanner. There is also java.io.Reader version of this constructor.

Parameters:

in - the java.io.Inputstream to read input from.

Method Detail

getOffset

protected final int getOffset()

Description copied from class: BaseRawTokenizer

Gets the current offset of the tokenizer into the file buffer. This should be implemented in the grammar file by returning JFlex's zzchar.

Specified by:

getOffset in class BaseRawTokenizer<ASToken>

Returns:

The current offset.

getLine

protected final int getLine()

Gets the current line number of the tokenizer. Line numbers start at 0, not 1.

Specified by:

getLine in class BaseRawTokenizer<ASToken>

Returns:

The current line.

getColumn

protected final int getColumn()

Gets the current column number of the tokenizer. Column numbers start at 0, not 1.

Specified by:

getColumn in class BaseRawTokenizer<ASToken>

Returns:

The current column.

setAllowLTInE4XStringLiterals

protected void setAllowLTInE4XStringLiterals(boolean allow)

Description copied from class: BaseRawASTokenizer

in E4X, < is allowed in char and string literals.

Specified by:

setAllowLTInE4XStringLiterals in class BaseRawASTokenizer

Parameters:

allow - true if < is allowed

yyclose

public final void yyclose()
                   throws IOException

Closes the input stream.

Throws:

IOException

yyreset

public final void yyreset(Reader reader)

Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL. Internal scan buffer is resized down to its initial length, if it has grown.

Parameters:

reader - the new input stream

yystate

protected final int yystate()

Returns the current lexical state.

yybegin

protected final void yybegin(int newState)

Enters a new lexical state

Specified by:

yybegin in class BaseRawASTokenizer

Parameters:

newState - the new lexical state

yytext

protected final String yytext()

Returns the text matched by the current regular expression.

Specified by:

yytext in class BaseRawTokenizer<ASToken>

Returns:

The text of current token

yycharat

protected final char yycharat(int pos)

Returns the character at position pos from the matched text. It is equivalent to yytext().charAt(pos), but faster

Specified by:

yycharat in class BaseRawASTokenizer

Parameters:

pos - the position of the character to fetch. A value from 0 to yylength()-1.

Returns:

the character at position pos

yylength

protected final int yylength()

Returns the length of the matched text region.

Specified by:

yylength in class BaseRawASTokenizer

Returns:

a non-negative int

readStart

protected final int readStart()

returns the start of the current read

Specified by:

readStart in class BaseRawASTokenizer

Returns:

a non-negative int

readEnd

protected final int readEnd()

returns the end of the current read

Specified by:

readEnd in class BaseRawASTokenizer

Returns:

a non-negative int

markedPosition

protected final int markedPosition()

returns the marked position in the current read

Specified by:

markedPosition in class BaseRawASTokenizer

Returns:

a non-negative int

currentPos

protected final int currentPos()

returns the current position into the buffer

Returns:

a non-negative int

buffer

protected final char[] buffer()

returns the current active buffer. this will change and is not constant

Specified by:

buffer in class BaseRawASTokenizer

Returns:

the current buffer

yypushback

public void yypushback(int number)

Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method

Parameters:

number - the number of characters to be read again. This number must not be greater than yylength()!

nextToken

public ASToken nextToken()
                  throws IOException

Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.

Returns:

the next token

Throws:

IOException - if any I/O-Error occurs