Tokenizer (JOSM)

java.lang.Object
- org.openstreetmap.josm.tools.template_engine.Tokenizer

```
public class Tokenizer
extends java.lang.Object
```
This class converts a template string (stream of characters) into a stream of tokens.
The result of the tokenization (also called lexical analysis) serves as input for the parser TemplateParser.

Nested Class Summary

Nested Classes
Modifier and Type Class Description

static class Tokenizer.Token
A token for the parser

static class Tokenizer.TokenType
The token type

Field Summary

Fields
Modifier and Type	Field	Description
`private int`	`c`
`private Tokenizer.Token`	`currentToken`
`private int`	`index`
`private java.util.Set<java.lang.Character>`	`specialCharacters`
`private java.lang.String`	`template`
`private java.lang.StringBuilder`	`text`

Constructor Summary

Constructors
Constructor Description

Tokenizer(java.lang.String template)
Creates a new Tokenizer

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`private void`	`getChar()`
`Tokenizer.Token`	`lookAhead()`	Look at the next token
`Tokenizer.Token`	`nextToken()`	Get the next token
`Tokenizer.Token`	`skip(char lastChar)`	Skip until we hit a character

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - specialCharacters
```
private final java.util.Set<java.lang.Character> specialCharacters
```
  - template
```
private final java.lang.String template
```
  - c
```
private int c
```
  - index
```
private int index
```
  - currentToken
```
private Tokenizer.Token currentToken
```
  - text
```
private final java.lang.StringBuilder text
```
- Constructor Detail
  - Tokenizer
```
public Tokenizer(java.lang.String template)
```
    Creates a new Tokenizer
    
    Parameters:
    
    template - the template as a user input string
- Method Detail
  - getChar
```
private void getChar()
```
  - nextToken
```
public Tokenizer.Token nextToken()
                          throws ParseError
```
    Get the next token
    
    Returns:
    
    The next token
    
    Throws:
    
    ParseError - if there is an error getting the next token
  - lookAhead
```
public Tokenizer.Token lookAhead()
                          throws ParseError
```
    Look at the next token
    
    Returns:
    
    The next token
    
    Throws:
    
    ParseError - if there is an error getting the next token
  - skip
```
public Tokenizer.Token skip(char lastChar)
```
    Skip until we hit a character
    
    Parameters:
    
    lastChar - The last character to skip
    
    Returns:
    
    A token with the skipped characters