In case you're interested in how the StreamTokenizer works, check out the API documentation for a more detailed description of its implementation.