org.apache.flex.compiler.internal.parsing.as

Class BaseRawTokenizer<T extends TokenBase>

java.lang.Object

org.apache.flex.compiler.internal.parsing.as.BaseRawTokenizer<T>

Direct Known Subclasses:

BaseRawASTokenizer

public abstract class BaseRawTokenizer<T extends TokenBase>
extends Object

Base class for JFlex-based tokenizers (RawScriptTokenizer, RawTagTokenizer, RawCSSTokenizer, RawMetadataTokenizer). Tokenizers should use %extends BaseRawTokenizer and provide a definitino for getColumn(). yytext()) will be generated automatically.

Field Summary

Fields

Modifier and Type	Field and Description
protected StringBuilder	aggregateContents Contents of aggregate
protected int	aggregateStart Start offset of aggregate
protected int	aggregateStartColumn Column number of aggregate
protected int	aggregateStartLine Line number of aggregate

Constructor Summary

Constructors

Constructor and Description
BaseRawTokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addBadCharacterProblem(String badChar)
T	buildAggregateToken(int type) Build a token from the current aggregated text and the given type
T	buildToken(int type) Builds a token with the specified type and text, using the current getOffset(), getLine(), and getColumn().
protected T	buildToken(int type, int start, int end, int line, int column, CharSequence text)
T	buildToken(int type, String text) Build a token of the specified type, using the current yytext(), getOffset(), getLine(), and getColumn().
protected void	continueAggregate() Add the current yytext() to the current aggregate
protected void	continueAggregate(char c)
protected void	continueAggregate(char[] chars)
protected void	continueAggregate(String text)
protected void	continueAggregate(antlr.Token token)
protected T	fetchToken(int type, int start, int end, int line, int column, CharSequence text) Builds a token, or reuses a token, based on the underlying token pool
protected abstract void	fillBuffer(StringBuilder builder)
protected abstract int	getColumn() Gets the current column number of the tokenizer.
protected String	getContext(int line) Get the current context as a string (to help with debugging)
protected ISourceLocation	getCurrentSourceLocation(int tokenLength) Create a ISourceLocation object based on the current lexer state.
String	getLastTokenText()
int	getLastTokenType()
protected abstract int	getLine() Gets the current line number of the tokenizer.
protected abstract int	getOffset() Gets the current offset of the tokenizer into the file buffer.
List<ICompilerProblem>	getProblems()
String	getSourcePath() Gets the source path for the file being tokenized.
protected boolean	hasAggregateContents()
boolean	hasProblems()
protected abstract T[]	initTokenPool()
protected abstract T	newToken(int type, int start, int end, int line, int column, CharSequence text)
void	reset()
protected void	setLastToken(T token)
void	setReuseLastToken() signals that we should reuse the last token in the token pool without filling our buffer
void	setSourcePath(String sourcePath) Sets the source path for the file being tokenized.
protected void	startAggregate() Initialize a new aggregate with the current yytext() and position
protected void	startAggregate(antlr.Token token)
protected abstract String	yytext() Gets the text of the current token.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

aggregateStart

protected int aggregateStart

Start offset of aggregate

aggregateStartLine

protected int aggregateStartLine

Line number of aggregate

aggregateStartColumn

protected int aggregateStartColumn

Column number of aggregate

aggregateContents

protected StringBuilder aggregateContents

Contents of aggregate

Constructor Detail

BaseRawTokenizer

public BaseRawTokenizer()

Method Detail

initTokenPool

protected abstract T[] initTokenPool()

yytext

protected abstract String yytext()

Gets the text of the current token. This is implemented by JFlex.

Returns:

The text of current token

getSourcePath

public String getSourcePath()

Gets the source path for the file being tokenized. This source path is used when reporting problems.

setSourcePath

public void setSourcePath(String sourcePath)

Sets the source path for the file being tokenized. This source path is used when reporting problems.

getOffset

protected abstract int getOffset()

Gets the current offset of the tokenizer into the file buffer. This should be implemented in the grammar file by returning JFlex's zzchar.

Returns:

The current offset.

getLine

protected abstract int getLine()

Gets the current line number of the tokenizer. Lines numbers start at 0, not 1. This should be implemented in the grammar file by returning JFlex's zzline.

Returns:

The current line.

getColumn

protected abstract int getColumn()

Gets the current column number of the tokenizer. Column numbers start at 0, not 1. This should be implemented in the grammar file by returning JFlex's zzcolumn.

Returns:

The current column.

addBadCharacterProblem

protected void addBadCharacterProblem(String badChar)

getCurrentSourceLocation

protected final ISourceLocation getCurrentSourceLocation(int tokenLength)

Create a ISourceLocation object based on the current lexer state.

Parameters:

tokenLength - Length of the problematic input.

Returns:

Current source location used to report a syntax problem.

hasProblems

public boolean hasProblems()

Returns:

true if we encountered errors while tokenizing

getProblems

public List<ICompilerProblem> getProblems()

Returns:

any problems we encountered while parsing

buildToken

protected T buildToken(int type,
           int start,
           int end,
           int line,
           int column,
           CharSequence text)

reset

public void reset()

getLastTokenType

public final int getLastTokenType()

getLastTokenText

public final String getLastTokenText()

setLastToken

protected void setLastToken(T token)

startAggregate

protected final void startAggregate()

Initialize a new aggregate with the current yytext() and position

startAggregate

protected final void startAggregate(antlr.Token token)

continueAggregate

protected void continueAggregate()

Add the current yytext() to the current aggregate

continueAggregate

protected final void continueAggregate(antlr.Token token)

continueAggregate

protected final void continueAggregate(String text)

continueAggregate

protected final void continueAggregate(char c)

continueAggregate

protected final void continueAggregate(char[] chars)

fillBuffer

protected abstract void fillBuffer(StringBuilder builder)

hasAggregateContents

protected final boolean hasAggregateContents()

setReuseLastToken

public final void setReuseLastToken()

signals that we should reuse the last token in the token pool without filling our buffer

fetchToken

protected final T fetchToken(int type,
           int start,
           int end,
           int line,
           int column,
           CharSequence text)

Builds a token, or reuses a token, based on the underlying token pool

Parameters:

type - the type of token to build

start - the token start

end - the token end

line - the token line

column - the token column

text - the token text

Returns:

a token from the passed in parameters. This token is not safe to hold on to, and should be used and discarded before further tokens are queried

newToken

protected abstract T newToken(int type,
         int start,
         int end,
         int line,
         int column,
         CharSequence text)

buildAggregateToken

public T buildAggregateToken(int type)

Build a token from the current aggregated text and the given type

Parameters:

type - token type (based on the appropriate XxxTokenTypes interface)

Returns:

new token

buildToken

public final T buildToken(int type)

Builds a token with the specified type and text, using the current getOffset(), getLine(), and getColumn(). The grammar file should not override this.

Parameters:

type - token type (based on the appropriate XxxTokenTypes interface)

Returns:

new token

buildToken

public T buildToken(int type,
           String text)

Build a token of the specified type, using the current yytext(), getOffset(), getLine(), and getColumn(). The grammar file cannot override this.

Parameters:

type - token type (based on the appropriate XxxTokenTypes interface)

Returns:

new token

getContext

protected String getContext(int line)

Get the current context as a string (to help with debugging)

Parameters:

line - current line number

Returns:

description of current context