Regular Expressions for Syllabaries

Daniel Yacob

The Ge'ez Frontier Foundation

Overview

• Review of Syllabaries
• Why Regex for Syllabaries?
• Amharic Use Cases
• The Men Who Say "ꑳ"
• Unicode 4.1, 5.0 and Beyond

Review of Syllabaries

• A syllabary is a writing sytem comprised of syllables.
• A syllable is a CVC pattern.
• Most syllabaries use "open syllables" - CV only.
• We are concerned with (CV) syllabaries not (C)(V).
  For example 'ካ' vs 'കാ'.
• Syllabaries of Unicode are: Canadian, Cherokee, Ethiopic, Hiragana, Katakana & Yi.

Review of Syllabaries

English as a Syllabary

Why Regex for Syllabaries?

• Support for thinking in syllabaries.
• More consise, easier to read and maintain syntax for syllabic patterns.
• Express and match inherent properties of syllabaries.
  -need a way to match the "C" or "V" part of a (CV)
  e.g. how do you match the "can" of (ca)(na)(da) - "ᗺᘇᑕ"

Why Regex for Syllabaries?

For example...

• (0|1|2|3|4|5|6|7|8|9)
• [0123456789]
• [0-9]
• [:digit:]
• \p{IsDigit}
• \d

Amharic Use Cases

• Rather than devise an abstract syllabary, lets use a real one.
• Examples are easily found.
• Quite possibly the worst case scenario for a syllabary.
• We have Regexp::Ethiopic::Amharic to work with.

Amharic Use Cases

The Matrix

Amharic Use Cases

The Matrix

Amharic Use Cases

Find All The Houses

Amharic Use Cases

Find All The Houses

Amharic Use Cases

Find All The Houses

Amharic Use Cases

Find All The Houses

Amharic Use Cases

Find All Plurals

/[#7#]ች/

Amharic Use Cases

Find All Plurals

/[#7#]ች/

vs [ሆሎሖሞሦሮሶሾቆቦቮቶቾኆኖኞኦኮዖዞዦዮዶጆጎጦጮጶጾፆፎፖ]ች

Amharic Use Cases

Find All Plurals

/[#7#]ች/

vs [ሆሎሖሞሦሮሶሾቆቦቮቶቾኆኖኞኦኮዖዞዦዮዶጆጎጦጮጶጾፆፎፖ]ች

Negation: /[^#7#]/

Amharic Use Cases

Match Phonetic Mispellings

/[#3,6#]ያ/

• አንባቢያን & አንባብያን
• ሚያዚያ & ሚያዝያ
• ኢትዮጵያዊያን & ኢትዮጵያውያን

Amharic Use Cases

Match Phonological Mispellings

/[ምን][#በፈ#]/

• ላምፋ & ላንፋ
• ግምፎ & ግንፎ
• አምፋር & አንፋር
• ወምበር & ወንበር
• ዝምብ & ዝንብ

Amharic Use Cases

Restricted Order Matches

/[መ-ቀ]{#6#}/

• ላምፋ
• ቅኔ
• ሳድስ

Amharic Use Cases

Restricted Order Matches

/[መ-ቀ]{#6#}/

• ላምፋ
• ቅኔ
• ሳድስ

/[መ-ቀ]{#4,6#}/

• መሬት
• ደቂቃ
• ሳድስ

Amharic Use Cases

Symbol Redundancy / Phonemic Decay

The Ge'ez Syllabary

Amharic Use Cases

Symbol Redundancy / Phonemic Decay

The Amharic Syllabary

Amharic Use Cases

Homophonic Equivalence

Amharic Use Cases

Homophonic Equivalence

Amharic Use Cases

Homophonic Equivalence

Amharic Use Cases

Homophonic Equivalence

ዓለምፀሐይ Patterns

• Amharic has 56 Possible Spellings
  -only 18 are probable: ዓለምፀሐይ ዓለምጸሐይ ዓለምጸሃይ ዓለምፀሃይ ዓለምፀሀይ ዓለምጸሀይ አለምፀሐይ አለምጸሐይ አለምጸሃይ አለምፀሃይ አለምፀሀይ አለምጸሀይ ዐለምፀሐይ ዐለምጸሐይ ዐለምጸሃይ ዐለምፀሃይ ዐለምፀሀይ ዐለምጸሀይ
• Tigrinya has 8 Possible Spellings
  -only 4 are probable: ዓለምፀሐይ ዓለምጸሐይ ዓለምጸሓይ ዓለምፀሓአይ
• Ge'ez has 4 Possible Spellings
  -only 2 are probable: ዓለምፀሐይ ዐለምፀሐይ

Amharic Use Cases

Homophonic Equivalence

ዓለምፀሐይ Patterns

Regexp::Ethiopic Usage & Limitations

#!/usr/bin/perl -w

use Regexp::Ethiopic::Amharic 'overload';
⋮
	if ( /[#መየበከ#]{#1,4#}\w+[=ሀ=]\w+?[#7#]ች/ ) {
	⋮
	}
⋮

The Men Who Say "ꑳ"

The Men Who Say "ꑳ"

Yi is a language, syllbary and people. The syllbary is used by between 2 and 5.5 million people in the Chinese provinces of Yunnan and Sichuan. Once had over 8,000 characters, now standardized to 819. Has 44 consonants... (x-axis) 10 vowels... (y-axis) ...and 4 tones (z-axis)! 10 x 44 x 4 = 1760 positions

The Men Who Say "ꑳ"

The Men Who Say "ꑳ"

Match all ꀖ in the orders 2-8 and the high and middle high tones

The Men Who Say "ꑳ"

Match all ꀖ in the orders 2-8 and the high and middle high tones

The Men Who Say "ꑳ"

Match all ꀖ in the orders 2-8 and the high and middle high tones

The Men Who Say "ꑳ"

Match all ꀖ in the orders 2-8 and the high and middle high tones

The Men Who Say "ꑳ"

Match all ꀖ in the orders 2-8 and the high and middle high tones

Unicode 4.1, 5.0 and Beyond

Unicode Technical Standard #18

// Folds katakana and hiragana together
class KanaFolder implements RegExFolder {
  // from RegExFolder, must be overridden in subclasses
  String fold(String source) {...}

  // from RegExFolder, may be overridden for efficiency

  RegExFolder clone(String parameter, Locale locale) {...}
  int fold(int source) {...}
  UnicodeSet fold(UnicodeSet source) {...}
}
  ...

  RegExFolder.registerFolding("k_h", new KanaFolder());

  ...

  p = Pattern.compile("(\F{k_h=argument}マルク (\s)* ダ (ヸ | ビ) ス \E : \s+)*");

Unicode 4.1, 5.0 and Beyond

Unicode Technical Standard #18

• The argument for k_h is the text upto \E.

Unicode 4.1, 5.0 and Beyond

Unicode Technical Standard #18

• The argument for k_h is the text upto \E.
• The k_h function folds everything, we want selective folding.

Unicode 4.1, 5.0 and Beyond

Unicode Technical Standard #18

• The argument for k_h is the text upto \E.
• The k_h function folds everything, we want selective folding.
• We need a way to pass additional args to restrict folding (i.e. [#መ-ቀ#], [#1,3-5#], etc).

Unicode 4.1, 5.0 and Beyond

Unicode Technical Standard #18

• The argument for k_h is the text upto \E.
• The k_h function folds everything, we want selective folding.
• We need a way to pass additional args to restrict folding (i.e. [#መ-ቀ#], [#1,3-5#], etc).
• More like k_h("[#መ-ቀ#]", "{3,4}").

Unicode 4.1, 5.0 and Beyond

Unicode Technical Standard #18

• The argument for k_h is the text upto \E.
• The k_h function folds everything, we want selective folding.
• We need a way to pass additional args to restrict folding (i.e. [#መ-ቀ#], [#1,3-5#], etc).
• More like k_h("[#መ-ቀ#]", "{3,4}").
• Otherwise we end up writing lots of overly specific functions for each scenario.

Unicode 4.1, 5.0 and Beyond

Unicode Technical Standard #18

• The argument for k_h is the text upto \E.
• The k_h function folds everything, we want selective folding.
• We need a way to pass additional args to restrict folding (i.e. [#መ-ቀ#], [#1,3-5#], etc).
• More like k_h("[#መ-ቀ#]", "{3,4}").
• Otherwise we end up writing lots of overly specific functions for each scenario.
• Not generalized enough, doesn't scale.

Unicode 4.1, 5.0 and Beyond

• Ethiopic is extended by 115 syllables and spread over 3 blocks.
• A new 8^th order appears, incrementiong the old orders 8-12.
• Five series previously with 8 orders now have 13.

Unicode 4.1, 5.0 and Beyond

Encoding of Vai (284 Characters)

Unicode 4.1, 5.0 and Beyond

• Mende - 195 characters
• Loma - 185 characters
• Kpelle - 88 characters
• Bete - 401 characters
• Bambara - 123 characters
• Bamum - 80 characters
• Magam/Eghap - 100 characters
• Djuka

~ · fini · ~

Where do we go from here?

• Regexp::Ethiopic
• Regexp::Cherokee
• Regexp::UCAS, Regexp::Yi, Regexp::Hiragana, Regexp::Katakana
• Work towards native regex adoption.
• Syllabic world domination, etc...

~ · fini · ~

• http://search.cpan.org/~dyacob/
• http://yacob.org/presentations/
• http://yacob.org/papers/
• http://syllabary.sf.net/