Mercury library 1.0 Translation of CFG grammars into an object model (Bachelor's thesis).

- Main Page
- Packages
- Classes
- Files

- Class List
- Class Index
- Class Hierarchy
- Class Members

All Classes Namespaces Files Functions Variables Enumerations Enumerator Properties
Public Member Functions | Properties | List of all members
Mercury.Scanning.Tokenizer Class Reference

A default implementation of ITokenizer interface using Microsoft Regular Expressions.
Splits tokens in the following fashion:
More...

Inheritance diagram for Mercury.Scanning.Tokenizer:

## Public Member Functions
| Tokenizer ()
| Creates a default tokenizer More...
| Tokenizer (IEnumerable< char > alpha)
| Creates a tokenizer with custom alpha characters More...
| Tokenizer (IEnumerable< char > alpha, IEnumerable< string > keywords, TokenizerOptions options)
| Creates a tokenizer with custom alphanumeric characters, keywords and options. More...
IList< Token > | Tokenize (string input)
| Splits source string into tokens. More...

## Properties
string[] | Keywords`[get, set]`
| Tokenizer keywords. More...
char[] | Alpha`[get, set]`
| Alpha characters. More...
TokenizerOptions | Options`[get, set]`
| Tokenizer options. More...

## Detailed Description

A default implementation of ITokenizer interface using Microsoft Regular Expressions.
Splits tokens in the following fashion:

- Control characters are not permitted
- Whitespaces are considered separators unless in quotations
- Quoted sequence is always a String token regardless its content
- Escape sequences '"', '\'' and '\' will lose their control function and will be interpreted as characters '"', ''' and '\' respectively. Other escapes will be ignored and left unchanged
- Number
  a) must begin with '+', '-' or a digit
  b) can contain one '.' between two digits
  c) must end with a digit
  d) must contain only digits, +, - and .
  e) if StrictNumbers option is used, a number must be separated by whitespaces
- Keyword
  a) must be specified in the keyword list
  b) must be separated from other tokens by whitespaces
  c) must NOT be quoted, otherwise will be marked as string
- Symbol
  a) a single non-alphanumeric character that is not a whitespace
  b) must NOT be specified in `alpha` parameter, otherwise will be marked as string
- String
  a) anything else

Definition at line 71 of file Tokenizer.cs.

## Constructor & Destructor Documentation

Mercury.Scanning.Tokenizer.Tokenizer | ( |  | ) |

Creates a default tokenizer

Definition at line 141 of file Tokenizer.cs.

Mercury.Scanning.Tokenizer.Tokenizer | ( | IEnumerable< char > | alpha | ) |

Creates a tokenizer with custom alpha characters

Parameters

  alpha | Custom alphanumeric characters.

Definition at line 147 of file Tokenizer.cs.

Mercury.Scanning.Tokenizer.Tokenizer | ( | IEnumerable< char > | alpha,
|  | IEnumerable< string > | keywords,
|  | TokenizerOptions | options
| ) |  |

Creates a tokenizer with custom alphanumeric characters, keywords and options.

Parameters

  alpha | Custom alphanumeric characters.
  keywords | Keywords.
  options | Tokenizer options.

Definition at line 158 of file Tokenizer.cs.

## Member Function Documentation

IList<Token> Mercury.Scanning.Tokenizer.Tokenize | ( | string | input | ) |

Splits source string into tokens.

Parameters

  input | Source string.

Returns
  List of tokens.

Implements Mercury.Scanning.ITokenizer.

Definition at line 196 of file Tokenizer.cs.

## Property Documentation

char [] Mercury.Scanning.Tokenizer.Alpha
| get set

Alpha characters.

Definition at line 187 of file Tokenizer.cs.

string [] Mercury.Scanning.Tokenizer.Keywords
| get set

Tokenizer keywords.

Definition at line 184 of file Tokenizer.cs.

TokenizerOptions Mercury.Scanning.Tokenizer.Options
| get set

Tokenizer options.

Definition at line 190 of file Tokenizer.cs.

The documentation for this class was generated from the following file:

- Mercury/Mercury/Scanning/Tokenizer.cs