1 #ifndef TOKENIZER_CLASS
2 #define TOKENIZER_CLASS
Provides a dynamically resizable ASCII character string.
Manages a bank of textual definitions of variables.
variable_tokenizer & operator=(const variable_tokenizer &to_copy)
makes this variable_tokenizer identical to "to_copy".
bool okay_for_variable_name(char to_check) const
true if "to_check" is a valid variable name character.
bool comment_char(char to_check) const
true if "to_check" is a registered comment character.
bool is_eol_a_separator() const
reports whether any of the separators are an EOL character.
const structures::string_table & table() const
provides a constant peek at the string_table holding the values.
bool exists(const basis::astring &name) const
returns true if the "name" exists in the variable_tokenizer.
void reset()
clears all of the entries out.
bool assignment(char to_check) const
true if "to_check" is a valid assignment operator.
const basis::astring & separators() const
provides a peek at the separators list.
const basis::astring & assignments() const
provides a peek at the assignments list.
basis::astring find(const basis::astring &name) const
locates the value for a variable named "name" if it exists.
variable_tokenizer(int max_bits=DEFAULT_MAX_BITS)
creates a variable_tokenizer with the default characters.
virtual ~variable_tokenizer()
bool separator(char to_check) const
true if "to_check" is a valid separator.
int symbols() const
returns the number of entries in the variable_tokenizer.
bool quote_mark(char to_check) const
true if "to_check" is a member of the quotes list.
basis::astring text_form() const
creates a new token list as a string of text.
void add_spaces(bool add_them)
void set_comment_chars(const basis::astring &comments)
establishes a set of characters in "comments" as the comment items.
const basis::astring & quotes() const
provides a peek at the quotes list.
DEFINE_CLASS_NAME("variable_tokenizer")
bool parse(const basis::astring &to_tokenize)
parses the string using our established sentinel characters.
Provides a symbol_table that holds strings as the content.