Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.
Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).
对于NLP来说，token就是分词，即一个句子要想让学习其特征，需要将其分成一个一个的word，那么这个word就是token。
