The Urimalsaem dictionary: how Bareun uses the National Institute of Korean Language vocabulary

Bareun embeds the vocabulary of Urimalsaem (Open Korean Dictionary), an open Korean dictionary operated by the National Institute of Korean Language (NIKL), as the urimal-dict.fb dictionary. This dictionary determines whether a word is standard or non-standard, recognizes derived words with prefixes and suffixes attached, and is a solid foundation of Korean vocabulary that serves as the basis for spell checking.

This dictionary is included in builds that contain the spell checking feature.

FlatBuffer + mmap: reading a large dictionary quickly

Urimalsaem has a very large number of headwords, so loading it entirely into memory would slow startup and use a lot of memory. Bareun stores this dictionary in FlatBuffer format (.fb) and reads it by mapping it with mmap.

• FlatBuffer lets you read the file structure as is, without deserialization, so even a large dictionary can be queried immediately.
• With mmap, the operating system brings only the needed parts into memory, so memory use is efficient.
• Lookups go through a dictionary lookup structure keyed by headword.
• To make restarts fast, secondary indexes use gob caching.

Determining standard and non-standard words

Urimalsaem covers not only standard words but also dialect, archaic words, and non-standard spellings. Bareun uses this information to tell whether an input word is standard or not.

• Words listed as standard are accepted as they are.
• Words classified as non-standard are passed on as correction candidates, providing a basis for correcting them to the standard word.

Morphemes recognized this way get IN_URIMALSAEM as their out_of_vocab value, leaving the fact that "this word was confirmed in Urimalsaem" in the result.

Prefix and suffix checking

Korean is a language that endlessly creates new words by attaching affixes. A dictionary cannot hold every derived word, so Bareun performs prefix and suffix checking based on Urimalsaem to recognize derived words.

Thanks to this check, even words that are not listed whole in the dictionary can be split into components and recognized.

Frequently asked questions

Q. Which builds include the Urimalsaem dictionary?

urimal-dict.fb is included in builds that contain the spell checking feature. It is not included in the basic build that uses morphological analysis only.

Q. How are technical terms or new words that are not in Urimalsaem handled?

Technical terms and new words that Urimalsaem does not cover are supplemented by the Onterm dictionary (onterm-dict.fb; Onterm is a Korean terminology platform). Words that are still not found are analyzed by guessing as out-of-vocabulary words (OUT_OF_VOCAB).

Q. How are the standard/non-standard results used in correction?

Words determined to be non-standard are passed on as correction candidates and, together with the non-standard word dictionary and others, become the basis for correcting them to the standard word. The category of the correction result is shown as STANDARD (standard language).

Q. How can I tell in the result that a word was recognized from the Urimalsaem dictionary?

Morphemes confirmed in the Urimalsaem dictionary get IN_URIMALSAEM as their out_of_vocab value. However, this source value is shown only in builds that include the spell checker, and does not appear when you use morphological analysis alone.

Q. Are words with prefixes or suffixes recognized even if they are not in the dictionary?

Yes. A dictionary cannot hold every derived word, so Bareun performs prefix and suffix checking based on the Urimalsaem dictionary. Words with affixes such as 풋사과 and 사람들 are also split into components and recognized, so it can handle words that are not in the dictionary as a whole.