Priority of the custom dictionary versus the built-in dictionary
Bareun (bareun) has rich built-in dictionaries, including about 1 million entries from Urimalsaem (the Open Korean Dictionary). So if you register the same word in a custom dictionary, which one applies? In short, the custom dictionary specified in the analysis request takes priority.
This document explains the priority between the custom dictionary and the built-in dictionaries, and how to check which dictionary a word came from.
Priority at a glance
The source of a morpheme is indicated by the out_of_vocab value. A word that comes from the custom dictionary is marked IN_CUSTOM_DICT.
| Source | out_of_vocab | Meaning |
|---|---|---|
| Word embedding | IN_WORD_EMBEDDING | A word included in the training data |
| Out-of-vocabulary word (OOV) | OUT_OF_VOCAB | An automatically guessed word that is not found anywhere |
| Custom dictionary | IN_CUSTOM_DICT | A word registered in the custom dictionary |
| Basic dictionary | IN_BUILTIN_DICT | The built-in basic dictionary |
| Urimalsaem | IN_URIMALSAEM | Urimalsaem (Open Korean Dictionary) from the National Institute of Korean Language (NIKL) |
| Wikipedia | IN_WIKI_DICT | The Wikipedia dictionary |
| Onterm | IN_ON_DICT | Specialized terminology dictionary (Onterm, a Korean terminology platform) |
Sources shown only in the correction (spell check) build
The three sources IN_URIMALSAEM, IN_WIKI_DICT, and IN_ON_DICT are shown only in the spell check build. These values do not appear when you use the morphological analyzer alone.
Why does the custom dictionary take priority?
The built-in dictionaries cover general Korean broadly, but they cannot know the intent of a specific company or domain. The intent a user states explicitly, "recognize this word this way," is the most specific, so the custom dictionary specified in the analysis request is applied first.
When the same word is in both
Even if Bareun's built-in knowledge splits 네네치킨 into "네네/NNG + 치킨/NNG", if you register 네네치킨 (a proper noun) in a custom dictionary and specify that domain in the analysis request, it is recognized as a single unit, 네네치킨/NNP.
graph TD
W[Input word] --> C{In the specified<br>custom dictionary?};
C -- Yes --> CD[Custom dictionary first<br>IN_CUSTOM_DICT];
C -- No --> B[Analyze with the built-in dictionary,<br>Urimalsaem, and so on];
The custom dictionary takes priority only when specified
Saying the custom dictionary takes priority applies when you specify it in the analysis request.
If you do not specify it with custom_dict_names (or set_domain), the custom dictionary is not applied, and the analysis uses only built-in knowledge.
Checking the source
You can check whether a registered word really came from the custom dictionary with out_of_vocab and custom_dict_name.
from bareunpy import Tagger
my_tagger = Tagger('YOUR-API-KEY', 'localhost', port=5656)
my_tagger.set_domain('brand') # Specify the custom dictionary
res = my_tagger.tags(['네네치킨을 먹었다.'])
for tok in res.msg().sentences[0].tokens:
for m in tok.morphemes:
print(m.text.content, m.tag, m.out_of_vocab, m.custom_dict_name)
네네치킨 NNP IN_CUSTOM_DICT brand
을 JKO IN_BUILTIN_DICT
먹 VV IN_WORD_EMBEDDING
었 EP IN_WORD_EMBEDDING
다 EF IN_WORD_EMBEDDING
. SF
The out_of_vocab of 네네치킨 is IN_CUSTOM_DICT and its custom_dict_name is brand, so you can tell that the brand custom dictionary was applied first instead of built-in knowledge.
Frequently asked questions
Q. Which takes priority, the custom dictionary or the built-in dictionary?
The custom dictionary specified in the analysis request takes priority. Even if the same word is entered differently in the built-in dictionary, it is recognized as registered in the specified custom dictionary.
Q. How do I know which dictionary a word came from?
Check the morpheme's out_of_vocab value. IN_CUSTOM_DICT means the custom dictionary, IN_BUILTIN_DICT means the basic dictionary, and IN_URIMALSAEM means Urimalsaem.
For the custom dictionary, custom_dict_name also contains the domain name.
Q. I registered the word, but the built-in dictionary result still appears.
Check whether you specified the domain in the analysis request (custom_dict_names/set_domain). If you do not specify it, the custom dictionary is not applied.
Q. Are the Urimalsaem, Wikipedia, and Onterm sources always shown?
The three sources IN_URIMALSAEM, IN_WIKI_DICT, and IN_ON_DICT are shown only in the spell check build. They do not appear when you use the morphological analyzer alone. The custom dictionary source IN_CUSTOM_DICT and the out-of-vocabulary word source OUT_OF_VOCAB can be seen in every configuration.