Programming Forums - View Single Post

black_dream · Mar 29th, 2005, 11:18 AM

For example,
"I am trying to write a code in python."

This sentence contains 9 tokens which is also the length of the sentence. Also notice that that sentence ends with "."

Is there any (tokenizer methods) in python like in java and if there is, how can be implemented.

--black_dream

Mar 29th, 2005, 11:18 AM	#10
black_dream Newbie Join Date: Mar 2005 Posts: 6 Rep Power: 0	For example, "I am trying to write a code in python." This sentence contains 9 tokens which is also the length of the sentence. Also notice that that sentence ends with "." Is there any (tokenizer methods) in python like in java and if there is, how can be implemented. --black_dream