Frequency-based typo correction: how the word frequency dictionary is used
Bareun quickly catches frequently occurring typos.
Bareun's frequency-based typo correction uses a mapping table of common typos,
a word frequency dictionary (word-freq.gob), and BK-tree similar-word search together.
Common typos are fixed directly from a mapping table
For common typos, there is no need to call a model or an LLM. They are corrected immediately from a mapping prepared in advance.
| Typo | Correction | Description |
|---|---|---|
스빈다 | 습니다 | A common sentence-ending typo from adjacent keys or mistyping |
마싯 | 맛있 | A typo from swapped final consonants or syllables |
These are classified in the TYPO category of RevisionCategory, and this is the lightest and fastest correction path.
If it is not in the mapping, look for similar-word candidates
For typos the mapping table does not catch, Bareun looks for candidates among similar words in the dictionary and corrects with them. Two data structures cooperate here.
graph TD
A[Eojeol to correct] --> B{In the frequency typo<br>mapping?};
B -- Yes --> C[Correct as mapped<br>e.g. 스빈다 → 습니다];
B -- No --> D[BK-tree similar-word search<br>jamo bigrams / edit distance];
D --> E[Similar-word candidates];
E --> F[word-freq.gob<br>sort candidates by frequency rank];
F --> G[Pick the most plausible candidate];
- BK-tree similar-word search: Based on jamo bigrams and edit distance, it quickly finds word candidates in the dictionary that are close to the input eojeol (a space-delimited word unit). The tree is built from Urimalsaem (the Open Korean Dictionary), and gob caching makes server restarts fast.
- Word frequency dictionary (
word-freq.gob): When there are several candidates, it gives priority to the word that is actually used more. Even among candidates with the same edit distance, the more frequent one is more likely to be what the typo was meant to be.
Why look at frequency too?
There are usually several words one or two characters away from a typo. Edit distance alone cannot tell which is the right one. People are more likely to have made the typo while trying to write the more common word, so ranking candidates with the frequency dictionary raises correction accuracy.
word-freq.gob is a separate artifact
The word frequency dictionary (word-freq.gob) is produced by a separate process that takes more than 10 hours to build.
It is a resource included only in builds that include the spell check feature.
Safeguard against over-correction
A high frequency does not mean a word is swapped in unconditionally. A candidate is accepted only when the jamo edit distance between the input and the candidate is close enough. Replacing it with a word too far from the original could change the meaning.
Turning off typo correction
If you do not need typo correction or want to keep the original spelling, turn on the disable_typo_correction option of RevisionConfig (default false).
{
"config": { "disable_typo_correction": true }
}
Frequently asked questions
Q. How are common typos like 스빈다 fixed so fast?
Frequently occurring typos are corrected immediately from a mapping table prepared in advance.
A typo in the mapping, such as 스빈다 → 습니다, is changed directly without going through a model or an LLM.
Q. What is the word frequency dictionary used for?
For typos not in the mapping, after finding similar-word candidates with the BK-tree, when there are several candidates it is used to give priority to
the more frequently used word by the frequency rank in word-freq.gob.
Q. Can I turn off typo correction?
Yes. If you set disable_typo_correction in RevisionConfig to true, frequency-based typo correction is disabled.
The default is false.
Q. Does a high frequency mean a word is always swapped in?
No. Even with a high frequency, a candidate is accepted only when the jamo edit distance between the input and the candidate is close enough. Replacing it with a word too far from the original could change the meaning, so only close candidates are allowed, which prevents over-correction.