Bareun vs Khaiii: the advantage of a transformer-based design

Khaiii is a Korean morphological analyzer released by Kakao. It uses a deep learning model based on a CNN (convolutional neural network). Bareun is also deep learning based, but it uses a transformer and splits analysis into two stages, segmentation (Segmenter) → POS tagging (PosTagger). Both are learning-based analyzers that do not depend on a dictionary alone, but they differ in model architecture and in how the processing stages are designed.

Comparison at a glance

Transformer vs CNN

Khaiii's CNN is good at catching local patterns inside a convolution window. It is a light and efficient structure, but it may be limited in taking distant context into account. Bareun's transformer looks at wide context across the sentence through attention, so you can expect strengths with ambiguous expressions and long dependencies. The model design is covered in more detail in Transformer architecture.

Bareun also trains segmentation separately, and in post-processing it recalculates offsets so that morphemes restored from conjugation or contraction have the exact position of the original syllables. For example, 킨데요 is restored as 키+이+ㄴ데+요, and 키, 이, and ㄴ데 are all made to point to the same start position.

Bareun's segmentation model assigns a segmentation label to each syllable. It uses 115 labels, including the whitespace handling rule (SP), and 14 of them are for detecting spell check errors. In addition, embeddings that include morphological information distinguish 12,175 words that have the same surface form but different morphology. This design restores morphological information from the segmentation stage onward. For the full flow of analysis, see Analysis pipeline overview, and for morphological embeddings, see Embedding.

Differences in operations

Bareun provides the analysis engine as a server, so you can call it through gRPC, REST, or connect-web and attach clients in many languages. It detects changes to the custom dictionary files in real time and updates with no downtime, and you can turn on word spacing and joining correction together as analysis options.

Related documents

• Bareun vs Kiwi · Bareun vs Mecab-ko: comparisons with other analyzers
• How was the 99.6% accuracy measured?: evaluation set and metric definitions

Frequently asked questions

Q. Is the transformer-based Bareun always more accurate than the CNN-based Khaiii?

You cannot declare a winner from model architecture alone. However, transformers are good at taking wide context into account, and Bareun simplified each stage by separating segmentation from tagging. On our own evaluation set it recorded 99.6% for POS tagging and 99.7% for eojeol splitting/restoration.

Q. Can I embed it as a library like Khaiii?

Bareun is provided as a server, and you call it from clients in various languages. On a closed network you can use the installable package, and in the cloud you can use api.bareun.ai.

Q. Doesn't a separate segmentation stage make it slower?

Splitting segmentation and tagging lets each stage solve an easier problem, which raises accuracy. You can tune processing speed with executionProvider (CPU/GPU) and batch settings.

Q. How does Bareun's segmentation model work?

Bareun's segmentation model is a transformer encoder that assigns a segmentation label to each syllable. It uses 115 labels, including the whitespace handling rule. 14 of them are for detecting spell check errors. In addition, embeddings that include morphological information distinguish 12,175 words that have the same surface form but different morphology, so morphological information is restored from the segmentation stage onward.