Reducing OOV: out-of-vocabulary word monitoring and improvement cycle
No matter how rich your dictionary is, new sentences always contain words it does not know. Bareun marks such words as
out-of-vocabulary words (OOV). If you run a cycle of collecting these marks and registering the words in a custom dictionary,
you can steadily raise the quality of analysis in your domain.
This page explains the improvement loop of extracting the eojeol (space-delimited word units) whose out_of_vocab is OUT_OF_VOCAB and registering them in a dictionary.
What is an OOV word?
Each morpheme in a morphological analysis result carries out_of_vocab, which indicates its source.
If this value is OUT_OF_VOCAB, Bareun could not find the word in any dictionary and guessed it automatically.
out_of_vocab | Meaning |
|---|---|
OUT_OF_VOCAB | OOV word: not found anywhere, so it was guessed (target for improvement) |
IN_CUSTOM_DICT | Registered in the custom dictionary (the goal state) |
IN_BUILTIN_DICT / IN_URIMALSAEM, etc. | Recognized from a built-in dictionary |
Why is OOV a signal of quality?
Bareun extracts OOV words through word embeddings and treats all low-frequency tokens as UNKNOWN.
In other words, an OOV word is a part that Bareun guessed without confidence. If the same word keeps coming up as OUT_OF_VOCAB
in your domain documents, that word is a top candidate for registering in the custom dictionary.
Reducing OOV directly raises the accuracy of your domain analysis.
The improvement cycle
graph LR A[Analyze real sentences] --> B[Extract OUT_OF_VOCAB]; B --> C[Sort candidates by frequency]; C --> D[Classify and register in the 5 dictionaries]; D --> E[Re-analyze and verify]; E --> A;
- Analyze: Run morphological analysis on the real sentences coming in during operation.
- Extract: Collect the eojeol whose
out_of_vocabisOUT_OF_VOCAB. - Sort: Sort them from most frequent so you handle the highest-impact words first.
- Classify and register: Put names in the proper noun dictionary, established nouns in the compound noun dictionary, and conjugating words in the verb or adjective dictionary.
- Verify: Analyze again to confirm they changed to
IN_CUSTOM_DICT, and pass any new OOV words on to the next cycle.
Example: extracting OOV words
This example picks only the OOV words out of the analysis results to build a candidate list.
import json
from collections import Counter
from bareunpy import Tagger
my_tagger = Tagger('YOUR-API-KEY', 'localhost', port=5656)
sentences = [ '...', '...' ] # sentences from your domain documents
oov_counter = Counter()
for s in sentences:
res = json.loads(my_tagger.tags([s]).as_json_str())
for tok in res['sentences'][0]['tokens']:
for m in tok['morphemes']:
if m.get('outOfVocab') == 'OUT_OF_VOCAB':
oov_counter[m['text']['content']] += 1
# Start with the most frequent OOV words as dictionary registration candidates
for word, cnt in oov_counter.most_common(20):
print(f'{word}\t{cnt}')
여신심사 37
방카슈랑스 21
민식이법 14
Looking at this list, you can classify 여신심사 (loan screening) as a compound noun and 민식이법 (the Minsik Law) as a proper noun and register them.
It gets better the more you run the cycle
If you repeat OOV extraction, registration, and re-analysis, the words that appear most often in your domain disappear one after another. Make checking the OOV list regularly part of your operating routine.
Start with the most frequent words
You do not need to register every OOV word at once. Handling the words that appear most often gives you the biggest quality improvement for the least effort.
Frequently asked questions
Q. How do I find OOV words?
Collect the morphemes whose out_of_vocab is OUT_OF_VOCAB in the morphological analysis results.
These are the words Bareun could not find in a dictionary and guessed, and they are your candidates for custom dictionary registration.
Q. What improves when I register OOV words?
The word is recognized exactly as the part of speech you intended, not as a guess. After registration, the morpheme's
out_of_vocab changes to IN_CUSTOM_DICT. If you repeat this, the accuracy of your domain analysis keeps rising.
Q. Do I have to register every OOV word?
No. It is more efficient to handle the words that appear most often. There is no need to register a typo that appears once or a one-off expression.