5 things to check when UNK tokens appear often

Do you keep seeing UNK in your analysis results? UNK is the unclassified state (tag=0) in which Bareun could not classify a morpheme as any part of speech. This document lists five things to check, in order, when UNK is frequent.

Symptoms

A morpheme's tag comes out as UNK, or a morpheme's out_of_vocab value is shown as OUT_OF_VOCAB (unregistered, automatically guessed). It appears often in domain-specific text (medicine, law, games, new words).

Cause

For each morpheme, Bareun tells you where that word came from in the out_of_vocab value.

Frequent UNK means there are many OUT_OF_VOCAB morphemes, which in turn means the input has many words that are not in the training data or the built-in dictionaries.

Solution: 5 things to check

1. Check the out_of_vocab value first. If it is OUT_OF_VOCAB, it is an out-of-vocabulary word. If it is IN_*, it is a normal word that came from a dictionary. If the word is normal but only the tag looks odd, see Results that differ from expectations.

2. Register proper nouns and technical terms in the custom dictionary. This is the most effective method. If you add an entry with UpdateCustomDictionary, the morpheme changes to IN_CUSTOM_DICT.

3. Use cp_set/cp_caret_set for compound nouns. Put long words made of several nouns into the compound noun dictionary, and use cp_caret_set, which marks the split point with ^, to make decomposition accurate.

4. Check word spacing. If eojeol (space-delimited word unit) boundaries are broken, even perfectly good words become UNK. Turn on auto_spacing to have it corrected.

5. Check the input encoding and cleanup. Broken characters, control characters, and mixed-in foreign text increase UNK. Also check that encoding_type (UTF8/UTF16/UTF32) matches the input.

Prevention and tips

• Building a domain dictionary in advance noticeably lowers the UNK rate. You can keep several dictionaries and specify them in custom_dict_names in priority order (the dictionary specified first takes priority).
• Custom dictionaries detect file changes in real time and are updated with zero downtime. You can add words even during operation.

Frequently asked questions

Q. How are UNK and OUT_OF_VOCAB different?

UNK is the tag (tag=0) for a morpheme that was not classified as any part of speech, and OUT_OF_VOCAB is the mark (the out_of_vocab value) that the word's source is "unregistered, automatically guessed." Both often come from out-of-vocabulary words.

Q. What is the fastest way to reduce UNK?

Register frequently appearing proper nouns and technical terms in the custom dictionary. After registering, if you specify custom_dict_names in the analysis request, those words are recognized as IN_CUSTOM_DICT.

Q. Domain terms keep showing up as UNK. What should I do?

Domain-specific text such as medicine, law, and games has many out-of-vocabulary words that are not in the training data or built-in dictionaries, so UNK is frequent. Register those terms in the custom dictionary (np_set, or cp_set/cp_caret_set for compound nouns) and specify them with custom_dict_names. You can give several dictionaries in priority order.

Q. Do I need to restart the server after adding UNK words to the dictionary?

No. Custom dictionaries are managed with the CustomDictionaryService API, and Bareun detects file changes in the dictionary directory in real time and applies them immediately with zero downtime. If you add words during operation, they apply from the next request.