When morphological analysis results differ from expectations: a debugging checklist

Is the morphological analysis result awkward even though the sentence is clearly correct? When you wonder "why is it split like this?", here is a debugging checklist to go through in order.

Symptoms

Eojeol (space-delimited word units) are segmented oddly, or the POS tags differ from what you expect.
Word spacing and joining are not handled the way you want.
The same word is analyzed differently depending on context (ambiguity).

Cause

Bareun goes through the stages segmentation → POS tagging → post-processing. When a result looks awkward, usually one of the following is at work.

Cause	Clue
Out-of-vocabulary word	`out_of_vocab` is `OUT_OF_VOCAB`
Low model confidence	`probability` is low
Word spacing options	Effect of `auto_spacing` / `auto_jointing`
Ambiguity	The same spelling has a different POS depending on context

Solution: checklist

Look at out_of_vocab first. If the source is a dictionary such as IN_BUILTIN_DICT or IN_URIMALSAEM, the word was recognized as a normal word, and if it is OUT_OF_VOCAB, an out-of-vocabulary word is the cause. In the latter case, consider registering it in the custom dictionary (see the NA and UNK documents).
Check probability (confidence). Each morpheme carries a softmax confidence. A morpheme with a low value is a part where the model was not sure, so start doubting from there.
Check the word spacing options. If auto_spacing (spacing correction) or auto_jointing (joining correction) is on, the original eojeol boundaries change. To analyze the original as it is, turn these options off and compare.
```
{
  "document": { "content": "아름다운강산", "language": "ko_KR" },
  "auto_spacing": true
}
```
If you look at the refined (corrected result) and modified (recognized after fixing the original by joining) markers together, you can tell what correction happened.
Suspect ambiguity. It is normal for the same spelling to be analyzed with a different POS depending on context. Bareun looks at the context and picks the most plausible analysis. If it differs from your intent, you can steer a particular interpretation with the custom dictionary.
If positions look off, check the encoding. Morpheme positions (begin_offset/length) have different values depending on the requested encoding_type (UTF8/UTF16/UTF32). Check that it matches the client's encoding.

If you need to preserve the original, use AnalyzeSyntaxList

To keep input order and positions as they are, without splitting sentences arbitrarily, use AnalyzeSyntaxList. It does no sentence splitting, which helps when tracking order and positions.

Prevention and tips

When debugging analysis results, looking at out_of_vocab, probability, and refined/modified together lets you narrow down the cause quickly.
If domain terms are often off, supplementing the custom dictionary is the most fundamental fix.

Frequently asked questions

Q. What do I look at first when the result differs from what I expected?

Look at the morpheme's out_of_vocab and probability. If it is OUT_OF_VOCAB, an out-of-vocabulary word is the cause, and if probability is low, it is a part where the model was not confident. These two are the starting point for debugging.

Q. The same word is analyzed differently in different contexts.

That is because of ambiguity. Bareun looks at the context and picks the most plausible POS, so this is normal behavior. If you want to fix a particular interpretation, you can steer it with the custom dictionary.

Q. What happens if I turn off auto_spacing?

It does not correct word spacing and leaves the original eojeol boundaries as they are. To compare results before and after correction, turn the option off, analyze again, and check whether the refined and modified markers appear.

Q. A POS tag looks awkward. Is the model wrong?

First look at the morpheme's probability (confidence). A morpheme with a low value is a part where the model was not sure, so start doubting from there. If out_of_vocab is OUT_OF_VOCAB, an out-of-vocabulary word is the cause and registering it in the custom dictionary is the fundamental fix, and if the source is normal but the result still looks awkward, it may be ambiguity.

Q. Morpheme positions (offsets) look off.

Morpheme positions (begin_offset/length) have different values depending on the requested encoding_type (UTF8/UTF16/UTF32). Check that the encoding the client uses matches the encoding_type in the request. To keep the original order and positions as they are, using AnalyzeSyntaxList is advantageous.