org.apache.flex.compiler.internal.parsing.as

Class BaseRawASTokenizer

java.lang.Object

org.apache.flex.compiler.internal.parsing.as.BaseRawTokenizer<ASToken>

org.apache.flex.compiler.internal.parsing.as.BaseRawASTokenizer

Direct Known Subclasses:

RawASTokenizer

public abstract class BaseRawASTokenizer
extends BaseRawTokenizer<ASToken>

Base class for RawActionScriptTokenizer. Pulling out code into Java backing class so it's easier to deal with. Uses a token pool to avoid object creation. The token pool will keep a buffer of 10 tokens in memory, and it is generally unsafe to hold on to tokens passed the point they are needed. If tokens are to be preserved, their copy constructor should be used

Field Summary

Fields

Modifier and Type	Field and Description
protected ASToken	bufferToken token that we may need to return for rules that may return more than one token
protected boolean	collectComments Flag to indicate we should collect comments
protected int	docTypeLevel Nested '<' bracket level (for
protected int	e4xBraceBalance The depth of the braces in E4X content.
protected int	e4xReturnState State to return to from E4X.
protected int	e4xTagDepth The depth of open tags, used to determine if we're still in E4X content.
protected boolean	isInCloseTag Tracks if we are in a close tag.
protected int	typedDepth Depth of typed collection constructs.

Fields inherited from class org.apache.flex.compiler.internal.parsing.as.BaseRawTokenizer

aggregateContents, aggregateStart, aggregateStartColumn, aggregateStartLine

Constructor Summary

Constructors

Constructor and Description
BaseRawASTokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	aggregateEscapedUnicodeChar(String escapeSequence) Convert the unicode escape sequence and append the unicode character to text buffer.
protected abstract char[]	buffer() returns the current active buffer. this will change and is not constant
ASToken	buildAggregateToken(int type) Build a token from the aggregation buffer.
protected ASToken	buildE4XTextToken(int type) Build an e4x text token from a given entity type.
protected void	continueAggregate() Add the current yytext() to the current aggregate
protected static int	decodeEscapedUnicode(String escapedUnicode) Convert escaped unicode sequence such as ÿ and \xFF to unicode code point.
protected String	escapedUnicodeToHtmlEntity(String escapedUnicode) Convert escaped unicode sequence such as ÿ and \xFF to HTML entities like &#xFF.
protected void	fillBuffer(StringBuilder builder)
ASToken	getBufferToken()
boolean	hasBufferToken()
protected ASToken[]	initTokenPool()
boolean	isCollectingComments()
boolean	isInE4XDatabinding()
boolean	isInXML()
protected abstract int	markedPosition() returns the marked position in the current read
protected ASToken	newToken(int type, int start, int end, int line, int column, CharSequence text)
protected abstract int	readEnd() returns the end of the current read
protected abstract int	readStart() returns the start of the current read
protected void	reportInvalidLineTerminatorInStringLiteral() Report unexpected line terminators in a string literal.
protected void	reportUnclosedASDoc() Report syntax error: input ended before ASDoc is closed.
protected void	reportUnclosedStringLiteral() Report syntax error: input ended before reaching the closing quotation mark for a string literal.
void	reset()
protected abstract void	setAllowLTInE4XStringLiterals(boolean allow) in E4X, < is allowed in char and string literals.
void	setCollectComments(boolean collect) True if we want to collect comments, besides ASDoc.
protected abstract void	yybegin(int state) begins the current state in the lexer
protected abstract char	yycharat(int pos) returns a char at the given offset into the internal char buffer
protected abstract int	yylength() returns the length of the current read

Methods inherited from class org.apache.flex.compiler.internal.parsing.as.BaseRawTokenizer

addBadCharacterProblem, buildToken, buildToken, buildToken, continueAggregate, continueAggregate, continueAggregate, continueAggregate, fetchToken, getColumn, getContext, getCurrentSourceLocation, getLastTokenText, getLastTokenType, getLine, getOffset, getProblems, getSourcePath, hasAggregateContents, hasProblems, setLastToken, setReuseLastToken, setSourcePath, startAggregate, startAggregate, yytext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

isInCloseTag

protected boolean isInCloseTag

Tracks if we are in a close tag. Used in the generated code

e4xTagDepth

protected int e4xTagDepth

The depth of open tags, used to determine if we're still in E4X content. Each time we see a tag open, this count is incremented. When we see any kind of tag close () we decrement the counter.

e4xBraceBalance

protected int e4xBraceBalance

The depth of the braces in E4X content. Used to determine when to pop out of E4X databindings and back to E4X content

e4xReturnState

protected int e4xReturnState

State to return to from E4X. Used in the generated code

typedDepth

protected int typedDepth

Depth of typed collection constructs. Used in the generated code

docTypeLevel

protected int docTypeLevel

Nested '<' bracket level (for

collectComments

protected boolean collectComments

Flag to indicate we should collect comments

bufferToken

protected ASToken bufferToken

token that we may need to return for rules that may return more than one token

Constructor Detail

BaseRawASTokenizer

public BaseRawASTokenizer()

Method Detail

continueAggregate

protected void continueAggregate()

Description copied from class: BaseRawTokenizer

Add the current yytext() to the current aggregate

Overrides:

continueAggregate in class BaseRawTokenizer<ASToken>

aggregateEscapedUnicodeChar

protected final void aggregateEscapedUnicodeChar(String escapeSequence)

Convert the unicode escape sequence and append the unicode character to text buffer. The input must have the leading escape character "\".

Parameters:

escapeSequence - Unicode escape sequence like ÿ or \xFF.

buildAggregateToken

public final ASToken buildAggregateToken(int type)

Build a token from the aggregation buffer.

Overrides:

buildAggregateToken in class BaseRawTokenizer<ASToken>

Parameters:

type - Type of the new token.

Returns:

New token.

buildE4XTextToken

protected final ASToken buildE4XTextToken(int type)

Build an e4x text token from a given entity type.

fillBuffer

protected final void fillBuffer(StringBuilder builder)

Specified by:

fillBuffer in class BaseRawTokenizer<ASToken>

readStart

protected abstract int readStart()

returns the start of the current read

Returns:

a non-negative int

markedPosition

protected abstract int markedPosition()

returns the marked position in the current read

Returns:

a non-negative int

readEnd

protected abstract int readEnd()

returns the end of the current read

Returns:

a non-negative int

buffer

protected abstract char[] buffer()

returns the current active buffer. this will change and is not constant

Returns:

the current buffer

setCollectComments

public void setCollectComments(boolean collect)

True if we want to collect comments, besides ASDoc. ASDoc comments will always be returned

Parameters:

collect - truefalse to ignore them.

setAllowLTInE4XStringLiterals

protected abstract void setAllowLTInE4XStringLiterals(boolean allow)

in E4X, < is allowed in char and string literals.

Parameters:

allow - true if < is allowed

initTokenPool

protected ASToken[] initTokenPool()

Specified by:

initTokenPool in class BaseRawTokenizer<ASToken>

isInXML

public boolean isInXML()

Returns:

true if we are still parsing XML content

isInE4XDatabinding

public boolean isInE4XDatabinding()

Returns:

true if we are parsing inside of an E4x databinding expression

yybegin

protected abstract void yybegin(int state)

begins the current state in the lexer

Parameters:

state - the state to begin

yycharat

protected abstract char yycharat(int pos)

returns a char at the given offset into the internal char buffer

Parameters:

pos - the offset into the buffer

Returns:

a char

yylength

protected abstract int yylength()

returns the length of the current read

Returns:

a non-negative int

newToken

protected final ASToken newToken(int type,
               int start,
               int end,
               int line,
               int column,
               CharSequence text)

Specified by:

newToken in class BaseRawTokenizer<ASToken>

isCollectingComments

public final boolean isCollectingComments()

Returns:

true if we are collecting comments

getBufferToken

public final ASToken getBufferToken()

hasBufferToken

public final boolean hasBufferToken()

reset

public void reset()

Overrides:

reset in class BaseRawTokenizer<ASToken>

escapedUnicodeToHtmlEntity

protected String escapedUnicodeToHtmlEntity(String escapedUnicode)

Convert escaped unicode sequence such as ÿ and \xFF to HTML entities like &#xFF.

Parameters:

escapedUnicode - Escaped unicode sequence in the form of either or \xFF.

Returns:

Encoded HTML entity string.

reportInvalidLineTerminatorInStringLiteral

protected final void reportInvalidLineTerminatorInStringLiteral()

Report unexpected line terminators in a string literal.

reportUnclosedStringLiteral

protected final void reportUnclosedStringLiteral()

Report syntax error: input ended before reaching the closing quotation mark for a string literal.

reportUnclosedASDoc

protected final void reportUnclosedASDoc()

Report syntax error: input ended before ASDoc is closed.

decodeEscapedUnicode

protected static int decodeEscapedUnicode(String escapedUnicode)

Convert escaped unicode sequence such as ÿ and \xFF to unicode code point.

Parameters:

escapedUnicode - Escaped unicode sequence in the form of either or \xFF.

Returns:

Unicode number.