English Explanatory Dictionary

tokenizer

English dictionary entry

Meanings

noun
  1. A system that parses an input stream into its component tokens.

Word forms

tokenizer tokenizers

Etymology

From tokenize + -er.

This entry uses open data from Wiktionary (CC BY-SA/GFDL). Word forms are used for search and are not indexed as separate pages.
Back to search All on T Russian dictionary Dahl dictionary