OpenITI NgramReader

Like the Google Ngram Viewer, the OpenITI NgramReader charts the frequencies of words and phrases over time, using the OpenITI corpus (8,468 texts; 1,054,622,781 tokens). Unlike it, the NgramReader lets you combine different forms of the same word into one curve — kitāb, al-kitāb, wa-l-kitāb — and build classes of terms, such as Baġdād together with Madīnat al-salām, or all cities of Ḫurāsān. By default, searches run on about 95% of the corpus: 414 texts (61 mln tokens) still contain paraeditorial elements and are left out unless you choose keep all texts (100%). Time is given in periods of 25 years by the author's death date (AH).

Search lines

Each line is one curve: #Label #item1 #item2 …. The first # part is the label shown in the legend; the other parts are the items searched for, and their matches are added together. For example, #Baġdād #bgdAd #bbgdAd #wbgdAd #wbbgdAd, or shorter with a regular expression, #Baġdād #[wb]?bgdAd. Add as many lines as you like with + add line.

Transliteration and Arabic input

Items are matched against the corpus in simplified Buckwalter transliteration: ء = c, ا إ أ آ = A, ب = b, ة = o, ت = t, ث = v, ج = j, ح = H, خ = x, د = d, ذ = V, ر = r, ز = z, س = s, ش = E, ص = S, ض = D, ط = T, ظ = Z, ع = C, غ = g, ف = f, ق = q, ك = k, ل = l, م = m, ن = n, ه = h, ؤ ئ = c, و = w, ى ي = y. You can also type items in Arabic script: letters are converted automatically, short vowels (harakat) and tatweel are removed, and a space between Arabic words becomes _. Labels stay as you type them.

Unigrams, bigrams, trigrams

Words of a phrase are joined with _: kataba ilay-hi is ktb_Alyh. The number of _ sets the length: none = unigram, one = bigram, two = trigram. All items of a search must have the same length.

Choosing forms

After Search, every line lists the forms its items match, with their Arabic script and corpus count. Untick forms you do not want and press Update results; only ticked forms are counted. The page address keeps the whole search, ticks included, so you can bookmark or share it.

Regular expressions

Items are regular expressions in JavaScript syntax, matched against whole words: each line becomes \b(item1|item2)\b. The usual constructs work as in the earlier R version of this app: [wb]?, (Al)?, \w+, ., *, +, {2,3}. Some ICU constructs of the R version are not available: possessive quantifiers (a++), atomic groups ((?>…)), \Q…\E quoting, POSIX classes such as [[:alpha:]], and inline flags such as (?i); the app says so when a search uses them. Matching is case-sensitive (s = س, S = ص). Spaces inside items are ignored.

Data download

The first search for unigrams, bigrams or trigrams downloads that list of forms once (3 MB for unigrams, about 27 MB for bigrams or trigrams); later searches reuse it. Counts are fetched only for the forms you search.