Overview of the whole Bareun morphological analysis pipeline

Bareun's Korean morphological analysis is not a method where one giant model gives the answer in a single pass. Through three stages, segmentation (Segmenter) → POS tagging (PosTagger) → post-processing (Postprocess), it analyzes the input sentence accurately into a hierarchy of sentence, eojeol (a space-delimited word unit), and morpheme.

This article is an overview that shows at a glance "how Bareun analyzes Korean." The detailed principles of each stage continue in the documents on segmentation rules, compound noun decomposition, and predicate conjugation and form restoration.

graph TD
  IN[Input sentence] --> SEG[Segmentation: Segmenter];
  SEG --> TAG[POS tagging: PosTagger];
  TAG --> POST[Post-processing: Postprocess];
  POST --> SPACE[Word spacing and joining correction];
  POST --> COMP[Compound noun and compound predicate handling];
  POST --> OFFSET[Form restoration and position tracking];
  SPACE --> OUT[Sentence, eojeol, morpheme result];
  COMP --> OUT;
  OFFSET --> OUT;

Stage 1: Segmentation (Segmenter)

The segmentation stage splits the inside of an eojeol into meaningful pieces. At this stage Bareun uses a transformer-based segmentation model (seg-model.onnx).

• The input sentence is fed to the model character by character, and each character gets a segmentation label.
• The length processed at once is 160 characters, and sentences longer than this are split appropriately before processing.
• In this process the text is divided into 8 major morphological units: substantive (N), particle (J), predicate (V), ending (E), adverbial (A), determiner (M), interjection (I), and symbol (S).

Stage 2: POS tagging (PosTagger)

This is the stage that attaches a POS to each segmented piece. The POS tagging stage assigns each morpheme one of the 47 parts of speech of the National Institute of Korean Language. This stage is handled by the segmentation-linked POS model (slot-tag-model.onnx). It reads the whole sentence at once for each slot the segmentation decided and attaches POS; details are in Segmentation-linked POS tagging.

• The length it reads at once is the whole sentence (up to 1,024 characters, 512 slots). If the sentence is longer than this or the segmentation-linked POS model cannot be used, the existing POS tagging model (tag-model.onnx, in units of 40 morphemes) takes over.
• Each morpheme gets a probability (softmax confidence) along with its tag.
• It also marks where the resulting morpheme came from (out_of_vocab). It distinguishes whether the word was in the word embeddings, an out-of-vocabulary word, or a word from the custom dictionary, the default dictionary, or Urimalsaem (the NIKL Open Korean Dictionary).

Stage 3: Post-processing (Postprocess)

This stage polishes the morpheme array the model produced into a result that is easy for people to read. Bareun's post-processing is not simple cleanup but a core stage that concentrates knowledge of Korean morphology.

• Word spacing and joining correction: Adjusts eojeol boundaries according to the auto_spacing and auto_jointing options. It fixes cases such as 아름다운강산 → 아름다운 강산 and 떨여 졌는데 → 떨어졌는데.
• Compound noun and compound predicate handling: Decomposes registered compound nouns and recognizes auxiliary predicate constructions.
• Form restoration and position tracking: Recalculates offsets so that morphemes restored from conjugation, contraction, and omission have the exact position of the original syllables. For example, 킨데요 is restored as 키+이+ㄴ데+요, and 키, 이, and ㄴ are all handled so that they have the same start position.

Result structure

The analysis result has the following hierarchical structure. A sentence is an array of eojeols (Token), and an eojeol is an array of morphemes (Morpheme). Every piece is expressed as a TextSpan unit that carries position information.

classDiagram
    Sentence *-- Token
    Token *-- Morpheme
    class Sentence {
        list~Token~ tokens
        TextSpan text
        String refined
    }
    class Token {
        list~Morpheme~ morphemes
        TextSpan text
        String lemma
        String tagged
    }
    class Morpheme {
        Tag tag
        float probability
        OutOfVocab outOfVocab
    }

Frequently asked questions

Q. Why separate segmentation and POS tagging?

In Korean, several morphemes are attached inside one eojeol in conjugated and contracted forms. If you first decide exactly "where to split" and then attach POS, each stage solves an easier problem and overall accuracy goes up.

Q. How is the position (offset) of a morpheme calculated?

Each morpheme's TextSpan holds begin_offset and length. A morpheme restored from conjugation, contraction, or omission points to the position of the original syllable, and the position values vary with the requested encoding type (UTF8/UTF16/UTF32).

Q. Can it analyze long sentences too?

Yes. Segmentation is processed in units of 80 characters, so long sentences are analyzed without trouble. If you need sentence splitting, turn on auto_split_sentence, and to keep the input sentences as they are, use AnalyzeSyntaxList.