org.apache.flex.compiler.internal.parsing.mxml

Class MXMLTokenizer

java.lang.Object

org.apache.flex.compiler.internal.parsing.mxml.MXMLTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable, IMXMLTokenizer

public class MXMLTokenizer
extends Object
implements IMXMLTokenizer, Closeable

Tokenizes MXML files. Uses RawTagTokenizer to get basic tokens. Ignores comments (), processing instructions (), and whitespace. Replaces CDATA tokens with text tokens (strips out the cdata stuff.

Field Summary

Fields

Modifier and Type	Field and Description
protected MutablePrefixMap	rootPrefixMap
protected int	startOffset Start offset (for when you're parsing a section of the document that doesn't start at the beginning)
protected MXMLToken	xmlNSToken

Constructor Summary

Constructors

Constructor and Description
MXMLTokenizer()
MXMLTokenizer(IFileSpecification specification)
MXMLTokenizer(int startOffset) Reparse constructor.
MXMLTokenizer(String path) Constructor

Method Summary

Methods

Modifier and Type	Method and Description
protected void	buildTokenList(MXMLToken token, List<MXMLToken> list) Handles the addition of tokens to the internal token list.
void	close()
PrefixMap	getPrefixMap() If it exists, return the PrefixMap from the last parse
PrefixMap	getRootTagPrefixMap() Processes the given input and builds a PrefixMap for the root tag found within this document
List<ICompilerProblem>	getTokenizationProblems() Returns a collection of problems encountered while processing the given input
IMXMLToken[]	getTokens(Reader reader) Returns an array of IMXMLToken objects for a given range of text represented by a Reader
IMXMLToken[]	getTokens(String range) Returns an array of IMXMLToken objects for a given range of text represented by a String
boolean	hasTokenizationProblems() Determines if the the tokenizer has encountered any problems as it lexed the given input
static void	main(String[] args)
MXMLToken	nextToken() Returns the next token that can be produced from the given input
List<MXMLToken>	parseTokens(Reader input) Parse the contents of input
void	setIsRepairing(boolean isRepairing) Sets a flag to indicate whether this tokenizer should try to repair its token stream
void	setPath(String path)
void	setReader(Reader reader)
boolean	tokensWereRepaired() Determines if any tokens were added as a side effect of repair.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

startOffset

protected int startOffset

Start offset (for when you're parsing a section of the document that doesn't start at the beginning)

xmlNSToken

protected MXMLToken xmlNSToken

rootPrefixMap

protected MutablePrefixMap rootPrefixMap

Constructor Detail

MXMLTokenizer

public MXMLTokenizer(String path)

Constructor

MXMLTokenizer

public MXMLTokenizer()

MXMLTokenizer

public MXMLTokenizer(IFileSpecification specification)

MXMLTokenizer

public MXMLTokenizer(int startOffset)

Reparse constructor. Allows you to start the tokenizer with a start offset (for when you're parsing a section of the document that doesn't start at the beginning).

Parameters:

startOffset - Start offset

Method Detail

setPath

public void setPath(String path)

setReader

public void setReader(Reader reader)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getPrefixMap

public PrefixMap getPrefixMap()

If it exists, return the PrefixMap from the last parse

Returns:

a PrefixMap or null

setIsRepairing

public void setIsRepairing(boolean isRepairing)

Sets a flag to indicate whether this tokenizer should try to repair its token stream

Specified by:

setIsRepairing in interface IMXMLTokenizer

Parameters:

isRepairing - true to repair, false to not repair

getTokens

public IMXMLToken[] getTokens(Reader reader)

Description copied from interface: IMXMLTokenizer

Returns an array of IMXMLToken objects for a given range of text represented by a Reader

Specified by:

getTokens in interface IMXMLTokenizer

Parameters:

reader - a Java Reader

Returns:

an array of IMXMLToken objects

getTokens

public IMXMLToken[] getTokens(String range)

Description copied from interface: IMXMLTokenizer

Returns an array of IMXMLToken objects for a given range of text represented by a String

Specified by:

getTokens in interface IMXMLTokenizer

Returns:

an array of IMXMLToken objects

hasTokenizationProblems

public boolean hasTokenizationProblems()

Determines if the the tokenizer has encountered any problems as it lexed the given input

Returns:

true if we have encountered any problems

getRootTagPrefixMap

public PrefixMap getRootTagPrefixMap()

Processes the given input and builds a PrefixMap for the root tag found within this document

getTokenizationProblems

public List<ICompilerProblem> getTokenizationProblems()

Returns a collection of problems encountered while processing the given input

Returns:

a Collection of ICompilerProblem objects, or an empty Collection

nextToken

public MXMLToken nextToken()

Returns the next token that can be produced from the given input

Returns:

an MXMLToken or null when no more tokens can be produced

parseTokens

public List<MXMLToken> parseTokens(Reader input)

Parse the contents of input

Parameters:

input - Reader containing file to be parsed

Returns:

List of MXMLTokens

tokensWereRepaired

public boolean tokensWereRepaired()

Determines if any tokens were added as a side effect of repair. This can only be called after a tokenize call

Returns:

true if the token stream was modified

buildTokenList

protected void buildTokenList(MXMLToken token,
                  List<MXMLToken> list)

Handles the addition of tokens to the internal token list. Subclasses should override this method to handle different tokenizing strategies

Parameters:

token - The current token.

list - The list of tokens being built.

main

public static void main(String[] args)