Arabic Stemming

[antaz]

We should address the flowing points:

• What is Stemming?
• What do we need it for?
• What are the challenges for Stemmers of the Arabic language compared to other languages?
• What are the available resources?
• How do we go about implementing our own Stemmer for Arabic?

[larrasket]

What is stemming?

Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. See the following figure

(definition and figure from Introduction to Information Retrieval,
Christopher D. Manning)

Solving this problem is being a little bit easy in the case of Latino languages, especially English, and there is adequate amount of already-working programs and algorithms that works fast and efficiently. A ubiquitous one is the Snowball Stemming Algorithms which supports a longish list of languages¹, obviously Arabic is not one of them.

connection
connections
connective          --->   connect
connected
connecting

The principles for doing stemming are also well-designed to deal with Latin words, for example see the following rules for removing endings from words:

  -IC (adj) -+->  -ATION (noun)
             +->  -ITY (noun)
             +->  -MENT (adv)
             \->  -AT (verb)     -+->   -IV (adj)    -+->   -ITY (noun)
                                  |                   \->   -MENT (adv)
                                  \->   -OR (noun)

-ABLE (adj) -+->  -ITY (noun)
             \->  -MENT (adv)

 -OUS (adj) --->  -MENT (adv)

When it comes to doing Arabic stemming, however, the job is a little bit harder, and we will have to do more work as just removing endings (and even appendage in the start, in some languages) will never work in Arabic, because word forms are not being structured by extensions/removals. Take التاريخ which being fully stemmed to أرخ, another example is الاتصال -> وصل, you can never find a static rules as it with English algorithms.

What do we need it for?

Stemmers are needed for natural language processing, information retrieval, and text mining applications as: search engines, sentiment analysis, text classification, and machine translation. There is no a 100% accurate Arabic stemmer out there so far; thus all these fields lacks an open source one (I reckon that corporates like Google do have proprietary software to implement stemming).

In our case, using a reliable stemmer is critical to make Mu’jam perform in a convenient way. I guess that’s probably the reason that existing Arabic dictionary are not very punctual. Let’s take for example Almaany and Al Mou’aser with the word المتلاعبان, which its gerund is تلاعب. The stemming levels should go as follows:

A: User’s input.
B: First level of stemming, removing grammatical appendages.
C: Removing grammatical appendages continues.
D: Reaching for the gerund.
F: Reaching for the root.

The user will be interested in C and sometimes F. Let’s try Almaany with
المتلاعبان: (link)

And Al Mou’aser does not accept it at all, both of these dictionaries are incapable of relating a word in a different form to its original definition.

Options?

• Implementing a stemming algorithm.

  I believe by studying patterns in Arabic, we will be able to define a reliable algorithm.

• Implementing a stemming software using a big library of patterns

  Since every word (except for some anomalies like prepositions and postpositions) has a corresponding word in the Mizan Sarf الميزان الصرفي, we can define a set of all the possible موازين (Dr Imīl Yaʻqūb have collected this set already² ), after that the problem will be transformed from a stemming problem into a pattern recognition problem. Take for example المتلاعبون, the program will easily find by exclusion the corresponding وزن, which is متفاعلون, the gerund of it (which is also should be defined in the set) is تفاعل, in our case تلاعب, the root can be reached using the three corresponding letters to ف ع ل, in this case ل ع ب; and the pure word can be reached by getting the lowest level of the وزن, in this case متلاعب.

---

¹ Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French, German,
Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish,
and Turkish

² https://web.archive.org/web/20210925233743/https://www.worldcat.org/title/mujam-al-awzan-al-sarfiyah/oclc/301487025&referer=brief_results

[antaz]

Thanks for breaking this down, this is useful. I have a couple of questions:

• Let's take the case of the word المتلاعبان, what do we expect a the dictionary search engine to retrieve? because i think the closest to this word is تلاعب and not لعب ?
• Is أرخ a root of التاريخ or التأريخ ?

[larrasket]

Let's take the case of the word المتلاعبان, what do we expect a the dictionary search engine to retrieve? because i think the closest to this word is تلاعب and not لعب ?

Yes, that's why I mentioned we are looking for the gerund, not the root. In this case, the user should be redirected to the definition of متلاعب, however لعب and التلاعب should also be shown to the user optionally (if they want to get more details into a word's definition)

Is أرخ a root of التاريخ or التأريخ ?

The first, أرخ is the root. تأريخ, is its gerund, and its وزن is تفعيل, which we call مصدر ثلاثي مُزيد بتضعيف العين.