https://eis1600.aai.uni-hamburg.de/shiny/OpenITI_NgramReader/

قال أبو الحسن: نظر جحا إلى رجل بين يديه يسير على بغلة، فقال للرجل: الطريق يا حمصي! فقال الرجل: ما يدريك أني حمصي؟ قال: رأيت حر بغلتك، فإذا هو يشبه الحاء، ورأيت فقحتها فرأيتها تشبه الميم، ورأيت ذنبها فإذا هو يشبه الصاد، فقلت: إنك حمصي!

• 0255Jahiz.Bighal.Shamela0010428BK18-ara1.mARkdown, PageV02P239
• Abū al-Ḥasan said: «Juḥā looked at a man walking in front of him on a mule and said to the man: ‘[Off] the way, oh person from Homs (ḥimṣī)!’ The man replied, ‘How do you know I am from Homs?’ Juḥā said: ‘I saw the heat of your mule, and it resembled the letter ḤĀʾ. I saw its belly, and it resembled the letter MĪM. And I saw its tail, and it resembled the letter ṢĀD. So I said: ‘You are from Homs!’»

How much is 1.12 billion tokens?

• equivalent to an English corpus of about 1.34 billion tokens (English translations are 15-23% longer than Arabic originals)
• Word Count per Book: assuming each book contains about 90,000 words (300 words per page and 300 pages per book).
• Total Books Equivalent: approximately 14,889 books.
• Library Size: roughly 50 bookshelves (ca. 300 books per shelf).
• Physical Dimensions: about 50 meters of shelf space (1 meter per shelf).
• In Bibles: approximately 1,711 copies of the Bible.
• In Qurʾāns: about 17,403 copies of the Qurʾān.

A Story and Its Versions: Ṯimār al-qulūb of al-Ṯaʿālibī (d. 429/1038)

ومنهم عبد الله بن خازم السلمى والى خراسان لعبد الله بن الزبير ومن عجيب أمره أنه كان نهاية فى الشجاعة والنجدة وكان يخاف الفأر أشد مخافة فبينما هو ذات يوم عند عبيد الله بن زياد إذ أدخل عليه جرذا أبيض فتعجب منه فقال لعبد الله يا أبا صالح هل رأيت أعجب من هذا وإذا عبد الله قد تضاءل كأنه فرخ وأصفر كأنه جرادة فقال عبيد الله أبو صالح يعصى الرحمن ويتهاون بالسلطان ويقبض على الثعبان ويمشى إلى الأسد الورد ويلقى الرماح بوجهه والسيوف بيده وقد اعتراه من جرذ ما ترون أشهد أن الله على كل شيء قدير

• Among them was Abd Allãh b. Ḫāzim al-Sulamī, the governor of Ḫurāsān for Abd Allãh b. al-Zubayr. What was remarkable about him was that he was extremely brave and resourceful, yet he was terrified of mice. One day, while he was with ʿUbayd Allãh b. Ziyād, a white rat was brought before him and he was astonished. ʿUbayd Allãh said to ʿAbd Allãh, ‘O Abū Ṣāliḥ, have you ever seen anything more astonishing than this?’ And there was Abd Allãh, who had shrunk as if he were a chick and turned yellow as if he were a male locust. ʿUbayd Allãh then said, “Abū Ṣaliḥ disobeys the Merciful, takes lightly the authority, seizes the snake, walks toward the blooming lion, faces spears with his face and swords with his hands, and yet, he is overcome by a rat as you see. I testify that Allãh is capable of everything.

Projects driven by computational analysis

• Open Islamicate Texts Initiative (OpenITI)
• Modeling text reuse in the Arabic written tradition (KITAB)
• Social history of the Islamic world (EIS1600)
• Next project: Typology of the Arabic Written Tradition (c. 700-1900) (Arabesquery)

Possibilities

• Large scale text analysis:
• Text reuse: tracing quotations within the entire corpus and building a model of “borrowings”
• Developments of “genres” (types);
• Authorship attribution;
• Experimental studies;
• Teaching;
• Social history of the Islamic World;
• The evolution of the Islamic world;
• Formation of various social, religious, professional groups and their contribution;
• Cultural production;
• Historical developments (political, environmental, societal);
• Development of Islamic / Arabic historiography;

Arabic Biographical Collections / Dictionaries

• One of the major genres/types in Arabic written tradition;
• Structure: chronological, generational, alphabetical;
• Coverage: professional, religious, regional groups; everyone (later);
• Biographies: well-structured short texts texts:
• onomastic section; birth [date &] place; teachers, students, contribution, miscellaneous facts, death date & place, reaction of the community
• Number of biographies: 700,000 ?!

Biographical Collections: Richness of Onomastics

• ʿAbd al-Raḥmãn ibn ʿAlī ibn Muḥammad ibn ʿUbayd Allāh ibn ʿAbd Allāh ibn Ḥamādá ibn Muḥammad ibn Jaʿfar ibn ʿAbd Allāh ibn al-Qāsim ibn al-Naḍar ibn al-Qāsim ibn Muḥammad ibn ʿAbd Allāh ibn ʿAbd al-Raḥmān ibn al-Qāsim ibn Muḥammad ibn Abī Bakr al-Ṣiddīq (may God be pleased with him!) al-Qurašī al-Taymī al-Bakrī al-Baġdādī al-Ḥāfiẓ al-Mufassir al-Ḥanbalī al-Wāʿiẓ al-Ṣaffār, Ǧamāl al-Dīn, Abū-l-Faraǧ, known as Ibn al-Ǧawzī (d. 597/1201 ce)

OpenITI Corpus and EIS1600 Subcorpus

• EIS1600 subcorpus of OpenITI includes historical, biographical, and geographical texts (around 1,500 texts of potential relevance).
• *Date statements are a proxy to the number/volume of biographies.

EIS1600 :: DFG Emmy Noether Project (2021-2027)

• The Emmy-Noether Project (#445975300) undertakes an innovative study of “The Evolution of Islamic Societies (c. 600-1600 CE)” [EIS1600] through the computational analysis of a large number of historical and biographical texts, which are treated holistically as a unified corpus of historical information. EIS1600 employs a series of advanced computational methods of text analysis and data modeling, which are the key to discovering, evaluating, and modeling all relevant textual evidence at an unprecedented scale.
• The EIS1600 team works on identifying and analyzing long-term historical trends through three closely connected research areas. The first area focuses on social factors—major ethnic, religious, and professional groups and how they shaped the development of local communities and fused them into what we call the Islamic world. The second one focuses on various economic factors that had an effect on local communities. The third one traces patterns of environmental factors—plagues, famines, droughts, pest infestations, earthquakes, and climate change—and their effect on the life of local communities. Complementing and informing each other, these case studies will be the foundation for the PI’s robust synthesis of the evolution of the Islamic world over the period under study.

EIS1600 :: DFG Emmy Noether Project (2021-2027)

• Team:
• Hamid Reza Hakimi, PhD Researcher (Arabic Studies)
• Lisa Mischer, PhD Researcher (CS / Arabic Studies, till March 2024)
• Tariq Yousef, PhD Researcher (CS, until Sept 2023)
• Alicia González Martínez, PostDoc Researcher (CS, from Oct 2023)
• Maxim Romanov, Research Group Leader

OpenITI Corpus and EIS1600 Subcorpus

• EIS1600 subcorpus of OpenITI includes historical, biographical, and geographical texts (around 1,500 texts of potential relevance).
• *Date statements are a proxy to the number/volume of biographies.

EIS1600 :: DFG Emmy Noether Project (2021-2027)

• How do we do that?
• “Digital Humanities”: computer-assisted automatic data extraction;
• Machine learning / Deep Learning / Artificial Intelligence:
• Machine-readable corpus;
• Annotated data > training models > automated data extraction;
• Teaching: Courses and Workshops
• DH courses focusing on data analysis:
  • Textual
  • Geographical / Networks
  • Prosopographical

EIS1600: Current Progress

• OpenITI Corpus and Texts Preparation
• OpenITI Corpus > EIS1600 Subcorpus;
• MIUs, and EIS1600 mARkdown Tagging Scheme;
• Automated Processing:
• Rule-based annotation; Deep-Learning / Machine-Learning - driven annotation;
• Data for research experiments;
• MasterChronicle Concept;
• Biographical Data:
• Extraction of biographical information;
• Identification of duplicate biographies;
• Geographical Data:
• NER Model to identify all toponyms;
• Operational Gazetteer;
• DL-Based Identification of Places via Proxies;
• Historical Data:
• Punctuation and segmentation Model;

Approach to Textual Sources: Minimal Information Units

• the main focus is on Minimal Information Units, e.g., descriptions of events from chronicles, and biographical records from biographical collections:
• premise: narratives are consciously constructed, while myriads of details scattered across vast texts simply cannot be subjected to the same level of agenda-driven editing; in large quantities, such agenda-resistant data is likely to provide more reliable historical evidence
• MUI will be aggregated into clusters of related historical information, which will be consequently studied
• Each primary source is broken into such units and they all to be re-aggregated into MasterChronicle, which will serve as a unified research ecosystem for the project and its collaborators

Linked Open Data > Linked Local Data

• URIs: 0748Dhahabi.TarikhIslam.MGR20180917-ara1.785357193186.EIS1600
• Author URI: 0748Dhahabi
• Book URI: 0748Dhahabi.TarikhIslam
• Edition URI: 0748Dhahabi.TarikhIslam.MGR20180917-ara1
• MIU URI: 0748Dhahabi.TarikhIslam.MGR20180917-ara1.785357193186

Linked Open Data > Linked Local Data

• Biographical Entities:
• Onomastics + an Onomastic Gazetteer
• Toponyms + Geographical Gazetteer (https://althurayya.github.io/
• Dates, Thematic Keywords, Misc Entities
• Persons + linked back to Biographies (future)
• Historical Events Entities:
• Subject - Predicate - Object
• Toponyms + Geographical Gazetteer (https://althurayya.github.io/
• Dates, Thematic Keywords, Misc Entities
• Persons + linked back to Biographies (future)

EIS1600 mARkdown (OpenITI mARkdown Flavor) / dynamic /

• Custom Annotation Scheme: short tags, opening only, color coding
• https://github.com/EIS1600/referenceable_illustrations/blob/main/markdown/EIS1600_mARkdown.gif
• https://raw.githubusercontent.com/EIS1600/referenceable_illustrations/main/markdown/EIS1600_mARkdown.gif

Data Annotation

• Persons, toponyms, miscellaneous (NER)
• Part of Speech (POS) and Lemmas (Lemmatization)
• Dates (RBC)
• String expression is parsed to numerical value
• Onomastic section (TC)
• Onomastic information (TC)

• Relationship classification between biographee and toponyms (TC), persons (TC), and dates (RBC) — work in progress

• RBC: Rule-Based Classification

• TC: Token Classification — Fine-Tuned CamelBERT Models
• NER: Named Entity Recognition — Fine-Tuned CamelBERT Models

Onomastic Section Detection

• Detect the boundaries of the onomastic section in the biography
• Data: 2,242 biographies
• 1,193 manually annotated
• 1,059 automatically annotated and manually corrected (ONOM section only)
• Model’s accuracy: 88.32%
• Most inaccuracies, however, are extremely minor (a single token difference or a punctuation sign), which makes the usability of this model significantly higher than the given number of 88.32%; in fact, all results are perfectly usable, even with an extra token or two that should not be included.

Data Extraction

• Based on added tags from the annotation pipeline
• Extracted entities are added as metadata
• Toponyms:
• Identification (Regex matching)
• Classification into settlement or province
• For settlements: adding corresponding province
• Calculating edges between settlements and between provinces
• Dates:
• Birth and death dates as boundaries for active period
• If no birth or death date: approximation for active period

Biographical Data

• Conflating biographies into individuals
• Creating training data (pre-generated, using different text comparison algorithms);
• Revising results: same, not-same;
• Using resulting training data to train a model for classifications
• Splitting model (nearest future):
• Model to split untagged texts into biographies automatically

Biographical Data: Same-Person Identification Model

• Data: 3.305 biography pairs
• 2.035 Not the same Person, 1.270 The same person
• 8 Features: tfidf, tfidf_bigrams, dice3gram, dice4gram, dice5gram, length_rate, min_length, ent_overlap
• * Dice: Sørensen-Dice Coefficient; ent_overlap: Entity Overlap; RBF: Radial Basis Function; SVM: Support Vector Mavhine; NN: Nearest Neighbour

Biographical Data: Onomastics Model

• All onomastic elements, mainly nisbaŧs are to be categorized
• This will allow to identify larger social, religious, tribal, etc. groups;

Geographical Data

• 101 sources:
• 477,880 thousand biographical records;
• 854,394 toponyms were found with our NER model;
• 235,466 MIUs contain toponyms (~49%; geographical nisbaŧs were not checked yet)
• 45,386 Unique Toponyms (included prefixed variants)
• 20,195 Unique Toponyms with MIU occurrences > 1
• 13,084 Unique Toponyms with MIU occurrences > 2
• 10,144 Unique Toponyms with MIU occurrences > 3
• 8,397 Unique Toponyms with MIU occurrences > 4
• 7,308 Unique Toponyms with MIU occurrences > 5

Geographical Data: Current Results

• Althurayya is a starting point, but only about half of it is “operational”;
• Together with newly identified toponyms, only about 2,100 occur in at least 10 biographical records;
• This is still a lot for expanding the gazetteer manually…
• Identifying the location and its coordinates is one of the most time-consuming tasks;

Geographical Data: Operational Gazetteer

• Premise: rare toponyms are problematic to medieval authors as well; they often define them through some proxy:
• For example, “Bashan is a village near Herat”;
• for most cases, the mention of Herat will be most sufficient;
• Plan:
• We can train a model to recognize such descriptions in texts;
• We can collect them from the entire corpus (or, at least, EIS1600 subcorpus);
• Extracted information can be processed automatically; ChatGPT can convert the above example (in Arabic) into a structured data like:
• Name of place: Bashan;
• Type of place: village;
• Location (Graviton): Herat;
• Primary experiments are very promising
• Other information, like distances can also be extracted in this manner;
• Nisbaŧs can be automatically generated from toponyms (with some minor post-correction)

Geographical Data: Operational Gazetteer

• Since collecting coordinates is an extremely time consuming task, we can limit it only for the most frequent places;
• Other locations can be added as proxies to larger locations:
• Village > its city;
• Quarter > it city;
• Market > its city;
• etc.
• The gazetteer will thus allow to preserve the original geographical information, yet allowing to effectively group it into meaningful regions;

Geographical Data: Modeling Regions - Voronoi Magnets

• Althurayya offers divisions into provinces
• however, they are relevant to 10-11th century;
• Alternative:
• Use most prominent urban centers as magnets and automatically associate other locations with them using Voronoi diagram:
  • Select magnets (highest frequencies);
  • Generate distances to those magnets > the closest one is the main magnet;
  • Approach parametrical:
  • Potentially, we can factor in other parameters:
    • Large bodies of water between place x and a magnet
    • Mountain ranges;
    • Deserts;
    • etc.

Historical Data: Punctuation / Segmentation

• Blobs of events in chronicles:
• Punctuation model (BAM):
  • Baššār ʿAwād Maʿrūf’s editions as training data
  • Rather decent results;
• Segmentation model (SEGM0):
  • some training data > first SEGM0 model to simplify things;
  • training data: blobs of text are automatically punctuated and split with BAM model into paragraph-like sections;
  • Human annotator rechecks all generated results;
• Now: the second iteration of active learning cycle;
• More training data has been recently prepared for further training;