io.growing.sdk.java.com.googlecode.protobuf.format

Class JsonFormat.Tokenizer

java.lang.Object

io.growing.sdk.java.com.googlecode.protobuf.format.JsonFormat.Tokenizer

Enclosing class:

JsonFormat

protected static class JsonFormat.Tokenizer
extends Object

Represents a stream of tokens parsed from a String.

The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:

• java.io.StreamTokenizer: This almost does what we want -- or, at least, something that would get us close to what we want -- except for one fatal flaw: It automatically un-escapes strings using Java escape sequences, which do not include all the escape sequences we need to support (e.g. '\x').
• java.util.Scanner: This seems like a great way at least to parse regular expressions out of a stream (so we wouldn't have to load the entire input into a single string before parsing). Sadly, Scanner requires that tokens be delimited with some delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"), Scanner would recognize it only as a single token. Furthermore, Scanner provides no way to inspect the contents of delimiters, making it impossible to keep track of line and column numbers.

Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need Matcher.usePattern(), which is new in Java 1.5.) So, we can use that, at least. Unfortunately, this implies that we need to have the entire input in one contiguous string.

Constructor Summary

Constructors

Constructor and Description
Tokenizer(CharSequence text) Construct a tokenizer that parses tokens from the given text.

Method Summary

Modifier and Type	Method and Description
boolean	atEnd() Are we at the end of the input?
void	consume(String token) If the next token exactly matches token, consume it.
boolean	consumeBoolean() If the next token is a boolean, consume it and return its value.
com.google.protobuf.ByteString	consumeByteString() If the next token is a string, consume it, unescape it as a ByteString, and return it.
double	consumeDouble() If the next token is a double, consume it and return its value.
float	consumeFloat() If the next token is a float, consume it and return its value.
String	consumeIdentifier() If the next token is an identifier, consume it and return its value.
int	consumeInt32() If the next token is a 32-bit signed integer, consume it and return its value.
long	consumeInt64() If the next token is a 64-bit signed integer, consume it and return its value.
String	consumeString() If the next token is a string, consume it and return its (unescaped) value.
int	consumeUInt32() If the next token is a 32-bit unsigned integer, consume it and return its value.
long	consumeUInt64() If the next token is a 64-bit unsigned integer, consume it and return its value.
String	currentToken()
boolean	lookingAtBoolean() Returns true if the next token is a boolean (true/false), but does not consume it.
boolean	lookingAtFloat() Returns true if the next token is an float, but does not consume it.
boolean	lookingAtInteger() Returns true if the next token is an integer, but does not consume it.
void	nextToken() Advance to the next token.
JsonFormat.ParseException	parseException(String description) Returns a JsonFormat.ParseException with the current line and column numbers in the description, suitable for throwing.
JsonFormat.ParseException	parseExceptionPreviousToken(String description) Returns a JsonFormat.ParseException with the line and column numbers of the previous token in the description, suitable for throwing.
boolean	tryConsume(String token) If the next token exactly matches token, consume it and return true.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(CharSequence text)

Construct a tokenizer that parses tokens from the given text.

Method Detail

atEnd

public boolean atEnd()

Are we at the end of the input?

nextToken

public void nextToken()

Advance to the next token.

tryConsume

public boolean tryConsume(String token)

If the next token exactly matches token, consume it and return true. Otherwise, return false without doing anything.

consume

public void consume(String token)
             throws JsonFormat.ParseException

If the next token exactly matches token, consume it. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

lookingAtFloat

public boolean lookingAtFloat()

Returns true if the next token is an float, but does not consume it.

lookingAtInteger

public boolean lookingAtInteger()

Returns true if the next token is an integer, but does not consume it.

lookingAtBoolean

public boolean lookingAtBoolean()

Returns true if the next token is a boolean (true/false), but does not consume it.

currentToken

public String currentToken()

Returns:

currentToken to which the Tokenizer is pointing.

consumeIdentifier

public String consumeIdentifier()
                         throws JsonFormat.ParseException

If the next token is an identifier, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeInt32

public int consumeInt32()
                 throws JsonFormat.ParseException

If the next token is a 32-bit signed integer, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeUInt32

public int consumeUInt32()
                  throws JsonFormat.ParseException

If the next token is a 32-bit unsigned integer, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeInt64

public long consumeInt64()
                  throws JsonFormat.ParseException

If the next token is a 64-bit signed integer, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeUInt64

public long consumeUInt64()
                   throws JsonFormat.ParseException

If the next token is a 64-bit unsigned integer, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeDouble

public double consumeDouble()
                     throws JsonFormat.ParseException

If the next token is a double, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeFloat

public float consumeFloat()
                   throws JsonFormat.ParseException

If the next token is a float, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeBoolean

public boolean consumeBoolean()
                       throws JsonFormat.ParseException

If the next token is a boolean, consume it and return its value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeString

public String consumeString()
                     throws JsonFormat.ParseException

If the next token is a string, consume it and return its (unescaped) value. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

consumeByteString

public com.google.protobuf.ByteString consumeByteString()
                                                 throws JsonFormat.ParseException

If the next token is a string, consume it, unescape it as a ByteString, and return it. Otherwise, throw a JsonFormat.ParseException.

Throws:

JsonFormat.ParseException

parseException

public JsonFormat.ParseException parseException(String description)

Returns a JsonFormat.ParseException with the current line and column numbers in the description, suitable for throwing.

parseExceptionPreviousToken

public JsonFormat.ParseException parseExceptionPreviousToken(String description)

Returns a JsonFormat.ParseException with the line and column numbers of the previous token in the description, suitable for throwing.