An exploratory research on grammar checking of Bangla sentences using statistical language models

M. D. Riazur Rahman, M. D. Tarek Habib, M. D. Sadekur Rahman, Gazi Zahirul Islam, M. D. Abbas Ali Khan

Abstract


N-gram based language models are very popular and extensively used statistical methods for solving various natural language processing problems including grammar checking. Smoothing is one of the most effective techniques used in building a language model to deal with data sparsity problem. Kneser-Ney is one of the most prominently used and successful smoothing technique for language modelling. In our previous work, we presented a Witten-Bell smoothing based language modelling technique for checking grammatical correctness of Bangla sentences which showed promising results outperforming previous methods. In this work, we proposed an improved method using Kneser-Ney smoothing based n-gram language model for grammar checking and performed a comparative performance analysis between Kneser-Ney and Witten-Bell smoothing techniques for the same purpose. We also provided an improved technique for calculating the optimum threshold which further enhanced the the results. Our experimental results show that, Kneser-Ney outperforms Witten-Bell as a smoothing technique when used with n-gram LMs for checking grammatical correctness of Bangla sentences.

Keywords


Grammar checking; Language models; Natural language processing; N-grams; Smoothing

Full Text:

PDF


DOI: http://doi.org/10.11591/ijece.v10i3.pp3244-3252
Total views : 139 times


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

ISSN 2088-8708, e-ISSN 2722-2578