Built-in dictionary of proper nouns and compound nouns: handling names of people, places, and organizations from news
To recognize proper nouns such as personal names, place names, and organization names well, Bareun uses
vocabulary learned from a large-scale news corpus together with Wikipedia vocabulary.
It is designed to guess and recognize even new proper nouns that are not listed in the dictionary as whole entries.
The extended vocabulary that underlies proper noun recognition is held in ext-dict.fb, which is included in every build.
Proper nouns learned from a news corpus
Bareun uses about 100 million eojeols (space-delimited word units) of the BigKinds news corpus for training. News is text in which proper nouns such as names of people, places, and organizations appear all the time, so it is well suited for the model to learn a wide range of proper noun usage.
| Training material | Size | Strength for proper nouns |
|---|---|---|
| BigKinds news | About 100 million eojeols | Rich in up-to-date names of people, places, and organizations |
| Korean Wikipedia (KoWiki) | Included in the extended dictionary | Supplements external proper nouns that are rarely in dictionaries |
Why use news?
News has names of people, institutions, and regions appearing in large numbers along with their context. Training on such text gives the model a sense of "words that come in this position are proper nouns," so it guesses even names it sees for the first time as proper nouns.
Even unseen proper nouns are guessed: OUT_OF_VOCAB
No dictionary can hold every name in the world.
When Bareun meets a word that is not in the dictionary, it guesses the POS with the sense it learned in training,
and attaches the out_of_vocab value OUT_OF_VOCAB to that morpheme to show it is an "out-of-vocabulary guess."
graph TD
W[Input word] --> D{In the dictionary or trained vocabulary?};
D -- Yes --> KNOWN[Show the matching source];
D -- No --> GUESS[Guess the POS from context];
GUESS --> OOV[Mark OUT_OF_VOCAB];
| OutOfVocab | Meaning in proper noun handling |
|---|---|
IN_WIKI_DICT | A proper noun recognized from wiki vocabulary |
OUT_OF_VOCAB | A word not in the dictionary, guessed from context |
Dictionary + guessing combined
Bareun recognizes proper nouns in the dictionary accurately and guesses new names that are not in the dictionary from context. If a proper noun you use often is only being caught by guessing, register it as a proper noun in the custom dictionary so that it is recognized reliably.
Frequently asked questions
Q. What data is used for proper noun recognition?
A model trained on about 100 million eojeols of BigKinds news and an extended dictionary (ext-dict.fb) holding Korean Wikipedia (KoWiki)
vocabulary are used together.
Q. How are new names that are not in the dictionary handled?
The POS is guessed with the contextual sense learned in training, and OUT_OF_VOCAB is attached to that morpheme
to mark it as an out-of-vocabulary guess.
Q. How can I make Bareun recognize a proper noun I use often more accurately?
Register it as a proper noun in the custom dictionary. A registered word is recognized as IN_CUSTOM_DICT and
handled more reliably than a guess. Custom dictionaries are managed through the CustomDictionaryService API.
Q. How do I tell an out-of-vocabulary word from a proper noun registered in the dictionary?
A proper noun recognized from a dictionary or trained vocabulary is marked with the matching source value (such as IN_WIKI_DICT),
and a word that is not in the dictionary and was guessed from context gets OUT_OF_VOCAB, meaning out-of-vocabulary word.
With this value you can tell in the result whether a word is "confirmed vocabulary" or a "guessed out-of-vocabulary word."