Class | BufferedTokenizer
In: | lib/protocols/buftok.rb
Parent: | Object

BufferedTokenizer takes a delimiter upon instantiation, or acts line-based by default. It allows input to be spoon-fed from some outside source which receives arbitrary length datagrams which may-or-may-not contain the token by which entities are delimited.

### Methods

empty? extract flush new

### Public Class methods

new(delimiter = "\n", size_limit = nil)

New BufferedTokenizers will operate on lines delimited by "\n" by default or allow you to specify any delimiter token you so choose, which will then be used by String#split to tokenize the input data

### Public Instance methods

empty?()
extract(data)

Extract takes an arbitrary string of input data and returns an array of tokenized entities, provided there were any available to extract. This makes for easy processing of datagrams using a pattern like:

      tokenizer.extract(data).map { |entity| Decode(entity) }.each do ...

flush()

Flush the contents of the input buffer, i.e. return the input buffer even though a token has not yet been encountered

[Validate]