Mercury.Scanning.Tokenizer Class Reference

A default implementation of ITokenizer interface using Microsoft Regular Expressions.

Splits tokens in the following fashion:
More...

Inheritance diagram for Mercury.Scanning.Tokenizer:

Public Member Functions
	Tokenizer ()
	Creates a default tokenizer More...
	Tokenizer (IEnumerable< char > alpha)
	Creates a tokenizer with custom alpha characters More...
	Tokenizer (IEnumerable< char > alpha, IEnumerable< string > keywords, TokenizerOptions options)
	Creates a tokenizer with custom alphanumeric characters, keywords and options. More...
IList< Token >	Tokenize (string input)
	Splits source string into tokens. More...

Properties
string[]	Keywords [get, set]
	Tokenizer keywords. More...
char[]	Alpha [get, set]
	Alpha characters. More...
TokenizerOptions	Options [get, set]
	Tokenizer options. More...

Detailed Description

A default implementation of ITokenizer interface using Microsoft Regular Expressions.

Splits tokens in the following fashion:

• Control characters are not permitted
  
• Whitespaces are considered separators unless in quotations
  
• Quoted sequence is always a String token regardless its content
  
• Escape sequences '"', '\'' and '\' will lose their control function and will be interpreted as characters '"', ''' and '\' respectively. Other escapes will be ignored and left unchanged
  
  
• Number
  a) must begin with '+', '-' or a digit
  b) can contain one '.' between two digits
  c) must end with a digit
  d) must contain only digits, +, - and .
  e) if StrictNumbers option is used, a number must be separated by whitespaces
  
• Keyword
  a) must be specified in the keyword list
  b) must be separated from other tokens by whitespaces
  c) must NOT be quoted, otherwise will be marked as string
  
• Symbol
  a) a single non-alphanumeric character that is not a whitespace
  b) must NOT be specified in alpha parameter, otherwise will be marked as string
  
• String
  a) anything else

Definition at line 71 of file Tokenizer.cs.

Constructor & Destructor Documentation

Mercury.Scanning.Tokenizer.Tokenizer

(

)

Creates a default tokenizer

Definition at line 141 of file Tokenizer.cs.

Mercury.Scanning.Tokenizer.Tokenizer

(

IEnumerable< char >

alpha

)

Creates a tokenizer with custom alpha characters

Parameters

alpha

Custom alphanumeric characters.

Definition at line 147 of file Tokenizer.cs.

Mercury.Scanning.Tokenizer.Tokenizer	(	IEnumerable< char >	alpha,
		IEnumerable< string >	keywords,
		TokenizerOptions	options
	)

Creates a tokenizer with custom alphanumeric characters, keywords and options.

Parameters

alpha	Custom alphanumeric characters.
keywords	Keywords.
options	Tokenizer options.

Definition at line 158 of file Tokenizer.cs.

Member Function Documentation

IList<Token> Mercury.Scanning.Tokenizer.Tokenize

(

string

input

)

Splits source string into tokens.

Parameters

input

Source string.

Returns

List of tokens.

Implements Mercury.Scanning.ITokenizer.

Definition at line 196 of file Tokenizer.cs.

Property Documentation

char [] Mercury.Scanning.Tokenizer.Alpha

getset

Alpha characters.

Definition at line 187 of file Tokenizer.cs.

string [] Mercury.Scanning.Tokenizer.Keywords

getset

Tokenizer keywords.

Definition at line 184 of file Tokenizer.cs.

TokenizerOptions Mercury.Scanning.Tokenizer.Options

getset

Tokenizer options.

Definition at line 190 of file Tokenizer.cs.

---

The documentation for this class was generated from the following file:

• Mercury/Mercury/Scanning/Tokenizer.cs