Word Segmentation: The main function of segmentation module is to identify and separate the tokens present in the text in such a way that every individual word, as well as every punctuation mark, will be a different token. The segmentation module considers words, numbers with decimals, or dates in numerical format in order not to separate the dot, the comma, or the slash (respectively) from the preceding and/or following elements. For this purpose, it uses a dictionary (this is an allocation list) and a small set of rules to detect numbers and dates.
POS Tagging: The output of the segmentation module is taken as input by the POS tagging module. Almost any kind of POS tagging could be applied. In the figure, it was shown a hybrid model for the problem of POS Tagging. This model combines a rule-based method and a statistical learning method. With regard to data, we use a lexicon with information about possible POS tags for each word, a manually labeled corpus, syntax and context of texts. Usually a Hidden Markov model and Viterbi algorithm are used for part-of-speech tagging. The elements of the model and the procedures used to estimate its parameters are devised.
No comments:
Post a Comment