Input Strings

• Java programs can create tokens from any input string values or variables. If a program is using the split method of the string class to create tokens, the string value can be assigned to a variable reference as in the following sample code:
  
  String myWords = "Here are some words";
  
  If a program is using the StringTokenizer class to create tokens, the constructor method of the class can take the initial string value as a parameter using the following syntax:
  
  StringTokenizer myTokenizer = new StringTokenizer("Here are some words");
  
  Once a program creates either of these variable types, it is ready to create tokens from the input string.
  
  

Delimiters

• Java programs can split input text strings on specific delimiters. A delimiter is a character or set of characters to split the string on. For example, Java can split a sentence into words using the space character as a delimiter. When a program is using the string class, the delimiter can be passed as a parameter to the split method. When a program is using the StringTokenizer class, the delimiter can be passed to the class constructor method along with the input string as follows:
  
  StringTokenizer myTokenizer = new StringTokenizer("Here are some words", " ");
  
  Java programs can use any character or combination of characters to act as delimiters, as well as optionally defining these using regular expressions. Common delimiters include new line and carriage return characters.
  
  

Token Results

• Java programs can process text tokens in various ways. If a program is using the split method of the string class, the tokens can be stored and processed within an array. When code calls the string split method, Java returns a string array. The code can then iterate through this, or access particular elements within it as required. When the StringTokenizer class is managing string tokens, programs use an object instance of the class itself to iterate through the string tokens one at a time.
  
  

Implementation

• Implementation of string tokens in a Java program depends on which classes are involved. The following sample syntax demonstrates splitting a string variable into an array of tokens:
  
  String[] myTokens = myWords.split(" ");
  
  The following code demonstrates iterating through string tokens using the StringTokenizer class:
  
  while (myTokenizer.hasMoreTokens()) {
  
  String thisToken = myTokenizer.nextToken();
  
  System.out.println(thisToken);
  
  }
  
  The official Java resources encourage developers to use the string class rather than the StringTokenizer, although recent language releases have continued to provide the class.