ext-dict (extended dictionary): supplementing personal names, quantity expressions, and external vocabulary
ext-dict.fb is an extended dictionary that supplements vocabulary Bareun has trouble handling with the trained model alone.
It holds surnames, enumeration expressions, unit-bearing dependent nouns (counter nouns), action nouns, and Korean Wikipedia vocabulary,
so it recognizes personal names and quantity expressions more accurately and splits compound nouns appropriately.
Together with dict.res, it is included in every build.
Vocabulary the extended dictionary holds
| Component | Contents | Analysis it helps |
|---|---|---|
| Surname list | Surnames | Personal name recognition |
| Enumeration expression list | Enumeration expressions | Recognizing list structures |
| Quantity expression list | Unit-bearing dependent nouns | Recognizing quantity expressions |
| Action (activity) expression list | Action nouns | Recognizing action nouns |
| Wiki-based vocabulary | Korean Wikipedia vocabulary | Supplementing external proper nouns |
| Wiki-based compound noun split information | Compound noun split information | Splitting compound nouns |
Personal names and quantity expressions
The surname list is used to catch personal names.
With surname information, it recognizes names such as 김영희 and 이순신 more reliably.
The list of unit-bearing dependent nouns lets it analyze quantity expressions naturally.
| Input | Recognition |
|---|---|
세 명 (three people) | Numeral + unit-bearing dependent noun |
다섯 권 (five volumes) | Numeral + unit-bearing dependent noun |
두 마리 (two animals) | Numeral + unit-bearing dependent noun |
Wiki vocabulary and compound noun splitting
The wiki-based vocabulary supplements external proper nouns that are not in a general dictionary.
Morphemes recognized from this dictionary get IN_WIKI_DICT as their out_of_vocab value.
The wiki-based compound noun split information is used to split long compound nouns into semantic units.
graph LR CN["인공지능연구소"] --> MAP[Wiki-based split information]; MAP --> R["인공지능 + 연구소"];
Why bring in external vocabulary?
Proper nouns and compound words in the world keep growing without end, so a training corpus alone cannot hold them all. Bareun brings broad external vocabulary such as Wikipedia into the extended dictionary and widens its recognition range to newly emerging names and compound words.
Frequently asked questions
Q. What vocabulary does the extended dictionary supplement?
It holds surnames, enumeration expressions, unit-bearing dependent nouns, action nouns, and Korean Wikipedia vocabulary, supplementing personal names, quantity expressions, and external proper nouns, and helping with compound noun splitting.
Q. Which builds include the extended dictionary?
ext-dict.fb is included, together with dict.res, in both the default build and the build that includes the Korean spell checker.
Q. How are words recognized from the wiki vocabulary marked?
Morphemes recognized from the Korean Wikipedia vocabulary get the out_of_vocab value IN_WIKI_DICT,
so you can tell their source. However, this source value is shown only in builds that include the Korean spell checker,
and does not appear when you use morphological analysis alone.
Q. How are compound nouns split?
Long compound nouns are split into semantic units using the wiki-based compound noun split information.
For example, 인공지능연구소 is split into 인공지능 + 연구소.
This split lets it accurately recognize the component words inside a compound noun.