Word frequency dictionary: how corpus frequency is used in typo correction and candidate scoring
Bareun has a built-in word frequency dictionary, word-freq.gob, that records how often each word is actually used.
When there are several candidates for fixing a typo, it is used to treat the more frequent word as the more plausible answer and rank the candidates.
This dictionary is included in builds that include the spell check feature.
Why frequency is needed
When you fix a typo, several candidates that merely look similar will usually come up. To judge which is more plausible, it matters which words people actually use often.
Edit distance alone is not enough
There can be many words whose shape is similar to 학생 (student).
It is hard to rank them by character difference (edit distance) alone,
but giving priority to the word that appears far more often in the corpus produces a more natural correction.
The word frequency dictionary supplies exactly this measure of how much people actually use a word.
Typo correction combined with a BK-tree
To find typo candidates, Bareun uses a BK-tree (similar-word search based on jamo bigrams) to gather candidates with a small edit distance, then re-scores those candidates with the word frequency dictionary and ranks them.
graph TD TYPO[Typo eojeol] --> BK[BK-tree similar-word search]; BK --> CAND[Candidates with small edit distance]; CAND --> WF[word-freq.gob frequency scoring]; WF --> RANK[Rank by highest frequency]; RANK --> BEST[Best correction candidate];
| Step | Data used | Role |
|---|---|---|
| Collect candidates | BK-tree | Gather words with a small jamo edit distance |
| Score candidates | word-freq.gob | Rank candidates by frequency |
The BK-tree is built from the Urimalsaem dictionary (the Open Korean Dictionary) and quickly finds similar words by the edit distance of jamo bigrams. The frequency dictionary moves the words that are actually used more to the top among those candidates, which raises correction quality.
Phoneme-based dictionary search is covered separately
Separately from this frequency scoring, the spell checker also finds sets of words that fit phonological rules through an NFD index that breaks the dictionary down into phonemes (initial, medial, and final consonants and vowels). The principle of phoneme-based search is covered in the spelling rules documents.
The gob format and build cost
The word frequency dictionary is stored in Go's gob format (word-freq.gob).
It is made by counting frequencies over a large corpus, so building it is a separate process that takes more than 10 hours.
Once it is made, the server reads this file and uses it quickly.
Frequently asked questions
Q. What is the word frequency dictionary used for?
When typo correction has several candidates, it is used to rank them so that words used more often in the corpus come first.
Q. How do the BK-tree and the frequency dictionary work together?
The BK-tree gathers similar-word candidates with a small edit distance, and the word frequency dictionary scores those candidates by frequency and picks the most plausible correction.
Q. Why does it take so long to build the frequency dictionary?
It has to count word frequencies over an entire large corpus, so the build takes more than 10 hours.
That is why it is made in advance as a separate process and stored as word-freq.gob.
Q. Which builds include the word frequency dictionary?
word-freq.gob is shipped with builds that include the spell checker.
It is used to rank typo correction candidates, so it is not in the basic build that only does morphological analysis.
Q. Besides the frequency dictionary, is phoneme-level search used too?
Yes. Separately from frequency scoring, the spell checker also breaks the dictionary down into phonemes (NFD) and finds sets of words that fit phonological rules. The details of phoneme-based dictionary search are covered in the spelling rules documents.