Paradigm Model based Hybrid Morph Analyzer for Gujarati using Partial Stemmer

IJSRD - International Journal for Scientific Research & Development| Vol. 3, Issue 11, 2016 | ISSN (online): 2321-0613

Paradigm Model based Hybrid Morph Analyzer for Gujarati using Partial Stemmer Chirag Patel1 Montu Patel2 Department of Computer Science & Engineering 2Department of Gujarati 1 Gujarat Technological University, Gujarat 2Sardar Patel University, Gujarat, India 1

Abstract— Morphological Analysis plays a major role in understanding and generating words and their forms for an efficient Machine Translation system. Morph Analyzer analyzes a word form, provides us the root or stem word and the linguistic information attached to the word form. It also generates a word form based on the root word and the linguistic information as provided. This linguistic information is very helpful during every stages of Natural Language Processing. In this paper we propose a Morph Analyzer which we have built keeping Gujarati language in mind. A Morph Analyzer can be made more efficient if the target language is known. We have added a partial stemmer in the paradigm model of a Morph Analyzer to achieve better accuracy. Our approach is the supervised approach where we manually build a dictionary of root words and classify them into different paradigms. A built-in partial stemmer performs the analysis and generation of word forms based on language dependent linguistic rules. Using this approach on Gujarati language we were able to achieve a considerable coverage of approximately 5,000 nouns.

Key words: Hybrid Morph Analyzer, Partial Stemmer

Analyzers for Gujarati. But as per our knowledge there has not been any initiative to build a Gujarati Morph Analyzer till date. As Gujarati being a widely spoken language across the globe and also Gujarati speakers too are one of the major communities on earth, the necessity of a Gujarati Morph Analyzer becomes our motivation to write this paper. Every language has the smallest information bearing units which are known as morphemes which contain the linguistic information of the word. These morphemes attach to a root or stem word and make different word forms. The extent of number of morphemes that attach to a root word depends on how agglutinative is the language. The more a language is agglutinative the more it is morphologically rich and hence more information can be extracted from the word form of the language. Gujarati is a very agglutinative and morphologically rich language. A word form in Gujarati can sometimes attach up to four morphemes together. Hence an efficient Morph Analyzer for Gujarati can be a very important part in the final Machine Translation system for Gujarati Language.

II. PREVIOUS WORK

I. INTRODUCTION Machine Translation is a very demanding research domain in the current world communication system. Inter language communication and information exchange can help a vast sector of users who are trying to interact internationally.

Natural Language Processing is a field that is focused to analyze a particular natural language and computationally understand and process the information hidden inside a sentence. To understand and extract information from a document, we can go from macro to micro analyses where understanding the summary of the whole document, understanding a particular paragraph, analyzing a sentence and at the most micro level understanding a particular word in the sentence. Here we can say that maximum extraction of information can be taken place at the word level. Hence in any phase of Natural Language Processing the Morph Analyzer becomes the base for finding the information from a word. Machine Translation focuses on translating a sentence from one natural language to another and vice versa provided the meaning and the essence of the sentence remains intact. So to convert a source language to a target language, a word needs to be broken down and analyzed at the source language and the same needs to be regenerated for the target language based on the linguistic information it carried at the source side. This process is very crucial and is the job of a Morph Analyzer. A morph analyzer extracts information from morphemes that attach to a particular root word. The more a language is agglutinative the more morphemes can be attached to a particular root word.

Gujarati language which is our focus in this paper is highly agglutinative language. Pratik et al [1] and Kartik et al [2] have developed stemmers to assist in building Morph

Much work has been done in building Morph Analyzers till date. There are two major approaches for building a Morph Analyzer viz. Supervised and Unsupervised. A supervised morph will require manually annotated data to learn the linguistic rules of the language. Approaches in this direction have widely used Conditional Random Fields to segment the morphemes from the stem or root words (Teemu et al [3]).

At the current state of the art CRFs have been used for performing morphological segmentation of Arabic language as a part of English to Arabic Machine translation system (Green et al [4]). An extent to this approach is the rules based approach where language specific rules are designed to analyze the word form and are hardcoded with the manually annotated data in the system. Another approach of building efficient morph analyzers is the unsupervised learning where lots of un-annotated raw data is provided to the system. The system uses different learning algorithms to identify the stem words from the corpus. A benchmark and notable work in this direction is performed by John Goldsmith et al [5] where he divides this whole process into a set of heuristics, which first determines the first analysis of stems using an initial bootstrapping heuristics and then modifies this analysis using some incremental heuristics, and a Minimal Description Length evaluation process which decides whether the modifications made by the incremental heuristics needs to be applied or not. An intermediate approach is the semi supervised approach. In this approach the software learns linguistic rules of a particular language like morph boundaries etc. from a manually annotated data and then using these rules and knowledge it build the morph dictionary of roots and forms from a large un-annotated data. Kohonen et al [6] Use a semi supervised approach

All rights reserved by www.ijsrd.com

585

Paradigm Model based Hybrid Morph Analyzer for Gujarati using Partial Stemmer (IJSRD/Vol. 3/Issue 11/2016/142)

where the CRFs are used to extract morph boundaries from a small annotated data and then applied on a large unannotated data to find the root words of the language. Poon el al [7] Proposes a log linear modeling approach which uses unsupervised learning for morphological segmentation.

Arne et al [8] has proposed a semi supervised approach based on Morfessor baseline which applies the simplest of the Morfessor methods. As per Pooja et al [9] Some of the common approaches are Rule based Approach where language specific suffixes are defined in the form of rules (Kenneth et al [10]), Paradigm based approach where the roots and morphemes are clustered into paradigms and finite state automata are used to generate the word-forms Shilpa et al [11] and Mugdha et al [12], Suffix striping approach where the word form is chopped character by character from the end until it is identified in some roots dictionary Rajeev et al [13], Finite state transducer approach where sequence of morphemes is checked using automates and many other Machine learning approaches which try to predict the morph boundaries (Attia et al [14]).

III. OUR APPROACH A Morph Analyzer takes a word as input and displays the root and the linguistic information attached to the word as output. Hence it is clear that the ideal way is to make a simple dictionary of all words and their linguistic information. But as this exercise becomes a very exhaustive and a tedious task, we need to use some intelligent approach to reduce the manual entry in the dictionary. As discussed earlier, many approaches have been applied to automatically find the root words of a language. It is observed that in all languages the root words can be classified into particular categories based on their conceptual meaning. For e.g.

Root Female Neurtra (Male, , l, Male, Female Neutral Singular Singula Singula Plural , Plural , Plural ) r r છોકરો (Boy)

છોકરી

છોકરું

બકરો (Goat)

બકરી

બકરું

Root (Male, Singular)

છોકરા

છોકરી

છોકરા

ઓ

ઓ

ઓ

બકરા

બકરી

બકરા

ઓ

ઓ

ઓ

Table 1: Conceptual Meaning Female Neurtral Male , , , Female, Singula Singula Plura Plural r r l

લેખક (Writer)

લેખખકા

લેખક

લેખકો

સેવક (Follower )

સેવવકા

સેવક

સેવકો

લેખખકા ઓ સેવવકા ઓ

Neutral , Plural લેખકો સેવકો

Table 2: Conceptual Meaning Here the words share the same procedure to build their word forms from the root words respectively. Hence we find out such different categories in which the root words in a particular category exhibit similar characteristics to build their word-forms. We call these categories as paradigms. Henceforth now we only need to build a

dictionary that contains only the root words which reduces our work to a considerable percentage. Also in the dictionary we keep only the root words and the paradigm it belongs to. Example entries of a dictionary look like: છોકરો

N

છોકરો

બકરો

N

છોકરો

લેખક

N

લેખક

સેવક

N

લેખક

Here the words છોકરો, બકરો, સેવક and લેખક are the root words and the words છોકરો and લેખક are paradigms, saying that the root words (છોકરો and બકરો), (સેવક and લેખક ) exhibit the same linguistic characteristics hence fall in the same paradigm viz છોકરો and લેખક respectively.

A Morph Analyzer performs two tasks. Firstly it takes a word form as an input, analyzes it and returns the word forms root word and the linguistic information attached to it. This is required at the source language side in a Machine Translation system. On the other hand a Morph Analyzer also generates a word form based on the root word and the linguistic information given to it as input. This part is required at the target language generation part in a Machine Translation system. Our Morph Analyzer performs both the tasks for Gujarati Language.

A. Analyzer: In this section a word-form is given to the system as input.

In our system we have developed a simple partial stemmer which will perform stemming on the given word-form. After every chopping of the last character the word is compared in the dictionary with root words. Until a word matches with a root word in the dictionary the chopping is continued. Once a match is found the paradigm of the root word is taken and then we try to recreate the word-forms of the root word using the linguistic procedures hard coded with the paradigm word to generate its word forms. If any of the word-forms generated matches the given input word-form then we display the root word and the linguistic information attached to the word-form generated as output. For e.g. if the input given to the Morph Analyzer is: સેવવકાઓ.

Then the stemmer will chop the last characters until it gets a perfect fit in one of the root words. For e.g. above input will result as “સેવ”. This left over stem matches the word “સેવક” & “સેવ”. Both are different root words. Now the stemmer will apply the rules of their paradigms to generate the word-forms on both the roots and check whether any of the word-forms generated matches the input or not. In this case the root “સેવક” will generate the word-form “સેવવકાઓ” and the root “સેવ” will not. Hence the root “સેવક” along with the linguistic information attached to the word-form “સેવવકાઓ” will be displayed as output.

B. Generator: This part of the software is very simple. Once the root and the linguistic information are given as input the system compares the root with the dictionary. If there is a match then based on the linguistic information provided the wordform is generated using the linguistic rules attached with the paradigm word.

All rights reserved by www.ijsrd.com

586

Paradigm Model based Hybrid Morph Analyzer for Gujarati using Partial Stemmer (IJSRD/Vol. 3/Issue 11/2016/142)

IV. EXPERIMENTS AND RESULTS

REFERENCES

Some of the linguistic features of a language may run the results in an ambiguous fashion. Some language ambiguities that came out while experimenting are the following: 1) Words “અફવા” and “અફવાહ” / “અનગમ” and “અનગમન”: Both the words mean the same, but are considered correct when used. But when we generate their wordforms they become same, for e.g. their plural form is “અફવાઓ”. Hence we take a decision that the minimum length root to be taken as the root word hence during stemming both words will be taken care of.

2) Words “હજાર” and “હજારી”: Both words have different meaning and in fact the words are proper nouns.

Tackling with proper nouns is always an issue in the field of Natural Language Processing. Both being root words will be assigned different paradigms in the dictionary. Now if the word forms of both words clash then the Morph Analyzer will display all different entries as many as same word-forms appear.

3) Words “ભેટ”, “ભેટો” and “ભેંટ”: These words all have different meaning. The first word is a verb which means ‘to hug’. The second word means to meet or make others meet but is a noun. The third word is also a noun and means a gift. The ambiguity arises when the first word is called in the honorific manner, it exactly becomes as the second word. Also when the third word is converted to the plural form it again becomes as the second word. Here for the same word we may have three different meanings. To some extent we can reduce ambiguity of such words by adding the part of speech in the dictionary.

4) If the root word does not exist in the dictionary, then the Morph should return that the word does not exists and instead should not give an incorrect root word. For e.g. in the above case of “સેવવકાઓ” if the root “સેવક” does not exists in the dictionary then the stemmer will continue chopping and will land up considering “સેવ” as the root.

But in such conditions we also perform a reverse check of generating the word-form and checking whether it matches with the given input word-form, which here will not match. Hence this issue will not arise.

We have built a dictionary containing approximately 5,000 nouns. Our algorithm can analyze any word-form whose root word is present in the dictionary. For roots not present in the dictionary the algorithm will return as unknown word.

[1] Pratikkumar Patel, Kashyap Popat and Pushpak Bhattacharya, “Hybrid Stemmer for Gujarati”, Proceeding of the 1st workshop on South and Southeast Asian Natural Language Processing(WSSANLP),International Conference on Computational Linguistics(COLING), Beijing, August 2010 [2] Kartik Suba, Dipti Jiandani, Pushpak Bhattacharya, “Hybrid Inflectional Stemmer and Rule-Based Derivational Stemmer for Guajarati”, Proceedings of the 2nd Workshop on South and Southeast Asian Natural Language Processing (WSSANLP), IJCNLP 2011, pages 1–8,Chiang Mai, Thailand, November 8, 2011.

[3] Teemu Ruokolainen, Oskar Kohonen, Sami Virpioja, Mikko Kurimo, “Supervised Morphological Segmentation in a Low-Resource Learning Setting using Conditional Random Fields”, Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 29–37, Sofia, Bulgaria, August 8-9 2013.

[4] Green and J. DeNero, “A class-based agreement model for generating accurately inflected translations”, In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 146–155, Association for Computational Linguistics 2102.

[5] John Goldsmith, “Linguistica: An Automatic Morphological Analyzer”, Papers from the 36th Meeting of the Chicago Linguistics Society, Volume1: The Main Session. 2000.

[6] O. Kohonen, S. Virpioja, and K. Lagus. 2010.

“Semisupervised learning of concatenative morphology”, In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, pages 78–86, Uppsala, Sweden, July.

[7] H. Poon, C. Cherry, and K. Toutanova. 2009.

“Unsupervised morphological segmentation with loglinear Models”, In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 209–217.

[8] Stig-Arne Gronroos, Sami Virpioja, Peter Smit and Mikko Kurimo, “Morfessor FlatCat: An HMM-Based Method for Unsupervised and Semi-Supervised Learning of Morphology” Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1177–1185, Dublin, Ireland, August 23-29 2014.

[9] Pooja Patel, Jhatayu Baxi and Rohan Prajapati, “A Survey on Various Methods to Develop Morph Analyzer”, IJIRST –International Journal for Innovative Research in Science & Technology| Volume 1 | Issue 7 | December 2014 ISSN (online): 2349-6010 [10] Kenneth R. Beesley and Lauri Karttunen, “Finite State Non-Concatenative Morphotatics”, SIGPHON2000.Proceedings of the fifth workshop of the ACL Special Interest Group in Computational Phonology, p.1-12.August 6,2000.Luxembourg.

V. CONCLUSION AND FUTURE WORK Here we have successfully build a Morph Analyzer for Gujarati language which works based on the paradigm model. Based on the algorithm it will correctly analyze and generate a word-form if its root word is present in the dictionary. Or else it will reply that the root word is not present in the dictionary. Currently we have performed for approximately 5,000 nouns but in future we plan to include adjectives, Verbs and Adverbs also to the dictionary which will increase the coverage of the dictionary to around 50,000 words.

All rights reserved by www.ijsrd.com

587

Paradigm Model based Hybrid Morph Analyzer for Gujarati using Partial Stemmer (IJSRD/Vol. 3/Issue 11/2016/142)

[11] Shilpa Desai, Jyoti Pawar and Pushpak Bhattacharya, “Automated Paradigm Selection for FSA Based Konkani Verb Morphological Analyzer”, Proceeding of COLING 2012, Demonstration Papers, Pages 103110 [12] Mugdha Bapat, Harshada gune and Pushpak Bhattacharya, “A Paradigm-Based Finite State Morphological Analyzer for Marathi”, Proceeding of the 1st workshop on South and Southeast Asian Natural Language Processing(WSSANLP), Pages 2634, the 23rd International Conference on Computational Linguistics(COLING), Beijing, August 2010.

[13] Rajeev R, Rahendran N and Elizabeth Sherly, “A Suffix Stripping Based Morph Analyser for Malayalam Language” Proceedings of 20th Kerala Science Congress, 2008.

[14] Mohammed Attia, Pavel Pecina, Antonio Toral, Lamia Tounsi and Josef van Genabith, “An Open-Source Finite State Morphological Transducer for Modern Standard Arabic”, Proceedings of the 9th International Workshop on Finite State Methods and Natural Language Processing, pages 125–133, Blois (France), July 12-15, 2011.

All rights reserved by www.ijsrd.com

588