Segmentation-linked POS tagging: a new structure for segmentation and POS tagging

Bareun's morphological analysis is made of two models. Previously, once the segmentation model cut the text into pieces, the POS tagging model read those pieces in chunks of 40 and attached POS tags. In the new structure, the segmentation model splits the sentence into slots (positions where morphemes go), and the segmentation-linked POS model (a model that picks a POS for each morpheme the segmentation model cut) looks at the whole sentence at once and attaches a POS to each slot. What changed is that the context before and after is used without interruption when deciding the POS.

graph TD
  IN[Input sentence] --> SEG[Segmentation model<br/>seg-model.onnx];
  SEG --> SLOT[Split into slots<br/>fix morpheme positions];
  SLOT --> TAG[Segmentation-linked POS model<br/>slot-tag-model.onnx];
  TAG --> POST[Post-processing];
  POST --> OUT[Sentence, eojeol, and morpheme results];
  TAG -. over 1,024 characters or over 512 slots .-> OLD[Previous POS tagging model<br/>tag-model.onnx];
  OLD --> POST;

What changed?

• Context carries through long sentences. The POS tagging model split a sentence once it went past 40 morphemes, so eojeol (space-delimited word units) near the split boundary had their POS decided from only half of the surrounding context. The segmentation-linked POS model reads up to 1,024 characters at once.
• Fewer 80-character boundaries in segmentation. The length the segmentation model handles at once was widened from 80 to 160 characters. Sentences of 159 characters or fewer are not split, and even longer sentences have fewer pieces, so fewer eojeol sit near the boundaries between pieces.
• The two models were not merged into one. It is still two stages: the segmentation model decides the slots and the segmentation-linked POS model attaches POS tags. The segmentation-linked POS model was trained starting from the syllable encoder of the segmentation model, so it reads syllables the same way.
• The previous model remains. When a sentence exceeds 1,024 characters or has more than 512 slots, and when inference of the segmentation-linked POS model fails, the previous POS tagging model handles it automatically. Results still come out without errors.

What changes for users

• The API call method, response structure, and POS tag system (47 parts of speech) are unchanged.
• Both analysis and spelling correction use the same structure. If the analysis result changes, the correction result changes accordingly.
• Even if you send a very long sentence, it is no longer cut every 40 morphemes as before, so there is no need to split the input into pieces of around 80 characters in advance.

Accuracy

On about 33,000 evaluation sentences, we measured whether the analyzer produced the same analysis as the answer. We compared fairly after aligning notation differences (unifying the determiner subdivision and the vowel-harmony allomorphs of endings into one form).

Frequently asked questions

Q. Were the segmentation model and the POS tagging model merged into one?

No. The new structure is also two stages. The segmentation model splits the sentence into slots, and the segmentation-linked POS model attaches a POS to each slot. What changed is that the model attaching POS tags reads the whole sentence at once.

Q. Is there still an 80-character or 40-morpheme limit as before?

The 40-morpheme limit on the POS-attaching side is gone. The segmentation-linked POS model reads up to 1,024 characters and 512 slots at once. The length the segmentation model handles at once is 160 characters. Sentences longer than 160 characters are split at spaces into pieces of nearly equal length and processed.

Q. What happens to sentences longer than 1,024 characters?

The analysis does not fail. The previous POS tagging model handles that sentence the same way as before.

Q. Did the response format change?

No. The requests and responses of the analysis API and the spell check API are the same as before.