Abstract—Stemming is a common preprocessing method aggregating all word variants to a standard stem to aid various Natural Language Processing (NLP) tasks. This work proposes a new unsupervised corpus-based stemmer that identifies the candidate suffixes using pivot word matching. Then candidate suffix statistics are used to remove the potential suffixes. After this, lexical similarity is measured to cluster the morphological related words. Finally, the smallest word in each cluster is designated as a stem. To quantify the performance of proposed method, two corpus-based and two linguistic knowledge-based stemmers for Urdu and English languages are used. The performance of each stemmer is evaluated on two different datasets for each language. The results show that the proposed PWMStem method outperforms the selected stemmers, achieving an accuracy of 0.876 for Urdu and 0.877 for English. To assess the performance of PWMStem through different aspects multiple evaluation metrics are used. The evaluation scores of other metrics are Index Compression Factor (ICF) = 73, Mean Number of Words per Conflation Class (MWC) = 3.7 for Urdu, and ICF = 71 and MWC = 3.5 for English. In the Urdu dataset, PWMStem achieved the lowest Under-stemming Index (UI) of 0.026479, Over-stemming Index (OI) of 0.000021, and an Error Rate Relative to Truncation (ERRT) of 0.610. In the English dataset, the values for UI, OI, and ERRT were measured as 0.102089, 0.000015, and 0.498, respectively.
 
Keywords—corpus-based stemming, morphology, natural language processing, Urdu stemmer, words inflection


Copyright © 2023 by the authors. This is an open access article distributed under the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-commercial and no modifications or adaptations are made.