Arabic retrieval
Lesson 7 of 10
7Arabic retrieval
The big picture«إجازة», «اجازه» and «الإجَازَة» are the same word to a reader and three different words to a computer. Users type without hamzas or diacritics; official documents use both. Normalising Arabic text before indexing and before searching is the cheapest improvement in an Arabic RAG system, and the most forgotten.
What the picture shows: Above: the computer sees every form as a different word. Below: one function removes diacritics and tatweel and unifies hamzas, ta marbuta and alef maqsura, so the forms meet. Documents and questions must both pass through this same function.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
One Arabic word can be written many ways: with or without hamza, taa marbuta or haa, with or without diacritics, with Arabic-Indic or Western digits. A computer sees them as completely different words. Unless you normalise them before searching, search fails to find what is right in front of it.
Normalisation strips diacritics and tatweel, unifies alef forms أ إ آ to ا, taa marbuta and alef maqsura, and converts Arabic-Indic digits. It is applied the same way to documents and questions alike, and is supported by stemming or Arabic-aware tokenisation.
Before You Start
⚗️Hybrid search
You know BM25 compares words letter by letter, so any spelling difference loses the match.
🧹Project: messy Arabic data
You have cleaned Arabic text before, and you know regular expressions with re.sub.
The Core Idea
«إجازة» and «اجازه» are one word to a reader and two to a computer. In this lesson you build a normalisation function, see its effect on search, unify digits and presentation forms, then learn where to stop.
1The normalisation function
Normalisation turns different forms of the same word into one form before comparing. Look at the picture: five rules every word passes through, in order.
import re
DIACRITICS = re.compile(r"[ً-ْٰ]") # fatha, damma, kasra, tanween, shadda, sukun, dagger alef
TATWEEL = "ـ"
def normalize(text):
text = DIACRITICS.sub("", text)
text = text.replace(TATWEEL, "")
text = re.sub("[إأآٱ]", "ا", text)
text = text.replace("ى", "ي")
text = text.replace("ة", "ه")
return text
for word in ["الإجَازَةُ", "إجازة", "اجازه", "مُســـتشفى", "مستشفي", "أحمد", "احمد"]:
print(f"{word:12} -> {normalize(word)}")
Removing diacritics and tatweel, unifying the hamza forms of alef to «ا», alef maqsura to «ي», and ta marbuta to «ه». The result: all seven forms became just three words. Note this happens in the search copy only; the original text stays as it is for display and citation.
Unifying the hamzas loses a rare distinction between a few words, and gains a match for most of what people type.
2The effect on search
People type without hamzas or diacritics, and official documents use both. Look at the picture: five questions as people type them, before and after normalisation.
import re
docs = ["تُقدَّم طلبات الإجازة قبل أسبوعين.",
"يحق للموظف إجازة مرضية بتقرير طبي.",
"تصرف بدلات السفر خلال عشرة أيام.",
"الأسئلة الشائعة عن المستشفى المتعاقد معه."]
queries = ["اجازه", "الاجازة", "اسبوعين", "مستشفي", "تقدم"]
def normalize(t):
t = re.sub(r"[ً-ْٰـ]", "", t)
t = re.sub("[إأآٱ]", "ا", t)
return t.replace("ى", "ي").replace("ة", "ه")
def found(query, docs, norm):
f = normalize if norm else (lambda s: s)
return [i for i, d in enumerate(docs) if f(query) in f(d)]
for q in queries:
print(f"{q:10} raw: {found(q, docs, False)!s:8} normalised: {found(q, docs, True)}")
Without normalisation, five questions typed the way people really type found nothing. After normalisation all of them found their documents. This is a plain text search, but the same effect happens in BM25 and any keyword index, and is smaller in embedding models without disappearing.
3Digits and presentation forms
Arabic-Indic (١٢٣), Persian (۱۲۳) and Latin (123) digits are one thing to a reader. If you do not unify them, "request 1043" will not find a document saying «طلب ١٠٤٣».
Text from old PDFs sometimes carries «لا» as one ligature character instead of two letters, so no comparison finds it.
import unicodedata
DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹", "01234567890123456789")
for text in ["الطلب رقم ١٠٤٣", "الطلب رقم 1043", "الطلب رقم ۱۰۴۳"]:
print(f"{text:18} -> {text.translate(DIGITS)}")
lam_alef = "ﻻ" # the ligature form of lam-alef, from old PDFs
print("ligature:", lam_alef, "length", len(lam_alef),
"-> NFKC:", unicodedata.normalize("NFKC", lam_alef), "length", len(unicodedata.normalize("NFKC", lam_alef)))
str.translate unifies digits with one table. And NFKC from unicodedata breaks presentation forms into their base letters: one character of length 1 became «لا» of length 2. Run NFKC on all text extracted from PDFs before any other normalisation.
4Where to stop
Normalisation has a small price: in rare cases two words with different meanings become one. So do not go beyond the five rules without measuring.
And avoid aggressive stemming that reduces a word to its root. Look at the picture: three words from one root, with three meanings.
How It Works, Step by Step: how to normalise Arabic text before search
Run NFKC on text extracted from PDFs to break presentation forms such as «ﻻ» into their letters.
Remove diacritics and tatweel, and map Arabic-Indic and Persian digits to Latin with one table.
Unify أ, إ, آ and ٱ to ا, ى to ي, and ة to ه.
Store the normalised copy for search beside the original for display, and pass the question through the same function.
Worked example
Normalisation is one function you apply to documents at indexing and to the question at search time, with the same rules. Start with the five most common rules.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Multilingual embedding models need no normalisation. | Its effect is smaller there but real: diacritics and different digits change tokenisation and so the vector, and BM25 in hybrid search certainly needs it. |
| Reducing words to their root improves Arabic search. | «كتاب» (book), «مكتب» (office) and «كاتب» (writer) share a root with different meanings, so a question about the office pulls documents about books. |
| You can normalise the original text and store it that way. | The user should see «الإجازة» as written when it is cited. Store two copies: the original for display and the normalised one for search. |
Try it yourself
The user typed «اسعار التذاكر» and the document says «أسعار التذاكر». Why did BM25 not find it?
BM25 compares words letter by letter. Unifying أ to ا on both sides makes them one word.
Why avoid aggressive root stemming in Arabic search?
«كتاب», «مكتب» and «كاتب» all come from «ك ت ب». A question about the office would pull documents about books. Light normalisation wins most of the benefit without this harm.
Quick review cards
Check Your Understanding
Do you need normalisation if you only use a multilingual embedding model?
Usually yes, though its effect is smaller. Embedding models cope with some variation, but diacritics and different digits change tokenisation, so the vector changes. More important, hybrid search needs BM25, and BM25 without normalisation fails as you saw. Measure on your "badly typed" questions and decide.
A user typed «طلب ١٠٤٣» and the document says «طلب 1043». Which rule fixes it?
Unifying digits with str.translate and a table mapping Arabic-Indic and Persian digits to Latin, on both the document and the question.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.