Korean morphology produced by 115 segmentation rules

In Korean, words keep changing shape inside a single eojeol (space-delimited word unit). Predicates conjugate, syllables contract, vowels drop, and characters are omitted. Bareun's segmentation (Segmenter) stage undoes these changes and restores the original morphemes, and underlying it is a system of 115 segmentation labels (rules). Of these 115, 14 are labels for error detection (spell checking), which catch morphological errors at the segmentation stage and act as a bridge to the spell checker.

The rules are learned in the model weights

One thing needs to be made clear here. These 115 rules do not exist as a separate rule text file. They are encoded in the weights of the trained segmentation model (seg-model.onnx). In other words, people did not code the rules in one by one. The model learned these rules on its own while being trained on a cleaned corpus.

Among all 115 labels, 14 are labels for error detection. While the remaining morphological rules take on restoring the surface form to the original morphemes, these 14 identify morphological errors at the segmentation stage (irregular conjugation errors, contraction and vowel progressive assimilation errors, and so on). In other words, while restoring morphemes, the segmentation model also gives a signal that "this eojeol has the wrong form," and this signal becomes the bridge to the spell checker.

Eight categories of labels

The 115 segmentation labels are classified into the following eight categories by the morphological phenomenon they deal with.

A representative example is the 104th rule, 생략_모음체언_이_ㄴㄹㅂ_1. From the name alone you can guess that it handles the case where "after a noun ending in a vowel, the particle 이 is omitted and gets entangled with the following consonant (ㄴ/ㄹ/ㅂ)."

Four segmentations of "킵": homonyms told apart by context

Even with the same surface form, segmentation differs by context. 킵 can split in as many as four ways, and the segmentation model looks at the context and picks one of them.

개인 킵니다.   →  개인(개인/NNG) + 킵니다.(키/NNG + 이/VCP + ㅂ니다/EF + ./SF)
고리킵니까?    →  고리킵니까?(고리키/NNP + 이/VCP + ㅂ니까/EF + ?/SF)

In both examples, the 105th rule 생략_모음체언_이_ㄴㄹㅂ_0 was at work, where after a noun ending in a vowel the copula 이 is omitted and gets entangled with the following consonant. Even for the same 킵, the noun + copula composition differs depending on the preceding word, so it is hard to resolve by dictionary matching alone, and a segmentation model that looks at context is needed.

A label system that grew gradually

The segmentation label system was not completed all at once. It developed gradually, adding labels each time a new variation was encountered. For example, the 73rd label 축약_방언_었 was added to catch cases where the pre-final ending 시 is contracted or appears in a dialect form (돌아가싯다 = 돌아가 + 시 + ㅅ + 다, 사삣다 = 사 + 어 + 삐 + ㅅ + 다). Labels like this were added one after another as spoken and dialect data were reflected in training, arriving at the current 115-label system (including the 14 error detection labels).

Catalog of Korean variations the segmentation rules handle

The segmentation rules deal broadly with four Korean morphological phenomena.

graph LR
  EOJ[Eojeol] --> SEG[Segmentation model];
  SEG --> A[Restore conjugation];
  SEG --> B[Restore contraction];
  SEG --> C[Restore elision];
  SEG --> D[Restore omission];
  A --> M[Original-form morphemes];
  B --> M;
  C --> M;
  D --> M;

The result of restoration: from surface form to original form

The purpose of the segmentation rules is to turn the form seen on the surface back into the original form found in the dictionary.

구르+어 먹+어서  ←  굴러 먹어서
벼르고 별러      ←  벼르고 벼러

The morphemes restored this way keep the position information of which syllable in the original text they came from. Even when restoration changes the number of characters, the post-processing stage recalculates the offsets so that each morpheme points to its exact starting position in the original text.

Frequently asked questions

Q. Can I see the list of 115 segmentation rules as a file?

No. The rules are not in a separate text file. They are encoded in the weights of the trained segmentation model (seg-model.onnx). It is not a method where people coded the rules in.

Q. What does it mean that 14 of the 115 segmentation labels are for error detection?

Of the 115, 14 are labels that identify morphological errors at the segmentation stage (irregular conjugation errors, contraction and vowel progressive assimilation errors, and so on), rather than morphological rules that restore the surface form. This error signal connects to the spell checker, so the segmentation model performs morpheme restoration and error detection at the same time.

Q. What Korean phenomena do the segmentation rules handle?

They handle Korean morphological variations such as predicate conjugation, syllable contraction, sound elision, and particle and ending omission. The labels are classified into eight categories: Basic, Irregular conjugation, Elision, Contraction, Omission, Conjugation-copula, Other, and Exception, and the purpose is to restore the surface form to the original dictionary form.

Q. What rule is 생략_모음체언_이_ㄴㄹㅂ_1?

It is the 104th segmentation rule, which restores the case where a particle is omitted after a noun ending in a vowel and gets entangled with the following consonant. The 105th rule of the same family, 생략_모음체언_이_ㄴㄹㅂ_0, is used to tell apart 킵 by context, as in 개인 킵니다 (키/NNG + 이/VCP + ㅂ니다/EF) and 고리킵니까 (고리키/NNP + 이/VCP + ㅂ니까/EF).