Out-of-vocabulary (OOV) word handling: embeddings vs custom dictionary vs built-in dictionary

You cannot put every word in the world into a dictionary. New words, technical terms, and proper nouns appear constantly. When Bareun meets a word that is not in a dictionary, that is, an out-of-vocabulary (OOV) word, it marks on each morpheme where it found out about that word. This source information is the out_of_vocab field.

Seven markers that tell the source

Each morpheme carries an out_of_vocab value that tells how the morpheme was recognized.

Value	Name	Meaning
0	`IN_WORD_EMBEDDING`	A word included in the word embedding (training)
1	`OUT_OF_VOCAB`	An out-of-vocabulary word that the model guessed automatically because it was not in a dictionary
2	`IN_CUSTOM_DICT`	A word in the custom dictionary
3	`IN_BUILTIN_DICT`	A word in the default (built-in) dictionary
4	`IN_URIMALSAEM`	A word in Urimalsaem (Open Korean Dictionary, from the National Institute of Korean Language)
5	`IN_WIKI_DICT`	A word in the Wikipedia dictionary
6	`IN_ON_DICT`	A word in the Onterm (Korean terminology platform) dictionary

The last three sources appear only in the Korean spell checker

IN_URIMALSAEM, IN_WIKI_DICT, and IN_ON_DICT are dictionaries used by the Korean spell checker (the distribution that includes the correction feature). If you use only the morphological analyzer, these three values do not appear.

For a morpheme marked IN_CUSTOM_DICT, you can also check which custom dictionary it came from through the custom_dict_name value.

Priority for finding out a word

Bareun recognizes morphemes by using several sources step by step.

graph TD
  W[Word input] --> EMB{In the embedding?};
  EMB -->|Yes| A[IN_WORD_EMBEDDING];
  EMB -->|No| CUS{Custom dictionary?};
  CUS -->|Yes| B[IN_CUSTOM_DICT];
  CUS -->|No| DIC{Built-in, Urimalsaem, Wiki, or Onterm dictionary?};
  DIC -->|Yes| C[IN_BUILTIN_DICT etc.];
  DIC -->|No| D[OUT_OF_VOCAB<br/>model guesses automatically];

It guesses even when the word is not in a dictionary

When Bareun meets a new word that is in no dictionary, it does not give up on the analysis. The transformer model looks at the surrounding context to guess the segmentation and POS, and marks that morpheme OUT_OF_VOCAB. This is the biggest difference from dictionary-based analyzers.

Extract OOV words with word embeddings

Bareun extracts out-of-vocabulary words through word embeddings. It handles not only nouns and proper nouns but also particles and endings, as form units. At this point, all low-frequency tokens, those that almost never appeared in the training data, are treated as UNKNOWN.

Why low-frequency tokens are gathered into UNKNOWN

If you take in even tokens that showed up only once or twice as vocabulary, the model memorizes unreliable signals and ends up unsteady. If low-frequency tokens are gathered into UNKNOWN, then even when a new word it has never seen appears, the model can recognize it through the same path and accept it by analyzing it from context. That is why Bareun can analyze even new words that did not exist at training time, in various ways.

Use it for automatic new-word extraction

The out_of_vocab marker can be used as is for automatic new-word extraction. If you analyze a large amount of recent documents and collect the morphemes marked OUT_OF_VOCAB, you can pick out candidates for new words that are in no dictionary yet. If you register the words you discover in the custom dictionary, they are recognized accurately from the next analysis on.

The value of showing the source

Even for the same noun, telling whether it came from the embedding, from a custom dictionary, or from a model guess lets you gauge the reliability of the analysis result. Areas with many guesses (OUT_OF_VOCAB) are also a sign that the custom dictionary needs to be supplemented.

Frequently asked questions

Q. What values does out_of_vocab have?

There are seven: IN_WORD_EMBEDDING, OUT_OF_VOCAB, IN_CUSTOM_DICT, IN_BUILTIN_DICT, IN_URIMALSAEM, IN_WIKI_DICT, and IN_ON_DICT. They tell where each morpheme was recognized from.

Q. How are words not in a dictionary handled?

The transformer model looks at the surrounding context and automatically guesses the segmentation and POS. Such morphemes carry the OUT_OF_VOCAB marker.

Q. How can I find new words?

If you analyze recent documents and collect the morphemes marked OUT_OF_VOCAB, you can pick out new-word candidates. If you register them in the custom dictionary, they are recognized accurately from later analyses on.

Q. How are out-of-vocabulary words extracted?

Bareun extracts out-of-vocabulary words with word embeddings, and handles not only nouns but also particles and endings as form units. All low-frequency tokens that almost never appeared in training are treated as UNKNOWN, so even new words seen for the first time are recognized through the same path and analyzed from context.

Q. Why are low-frequency tokens treated as UNKNOWN?

If you take in even tokens that appeared only once or twice as vocabulary, the model memorizes unreliable signals and becomes unsteady. If low-frequency tokens are gathered into UNKNOWN, new words that did not exist at training time can be recognized and accepted.