Non-standard word correction: morpheme probability and particle-adaptive adjustment
Bareun converts expressions that are not standard Korean, such as dialect words, archaic words, and common usage errors, into standard Korean.
Bareun's non-standard word correction is grounded in the National Institute of Korean Language's Standard Korean Language Dictionary and Urimalsaem (the NIKL Open Korean Dictionary).
It does not stop at simply swapping words; its feature is that it automatically adjusts the particle that follows to match the final consonant (batchim) changed by the correction.
What is a non-standard word
A word or expression that is not standard Korean. It is classified into the STANDARD category of RevisionCategory and corrected.
| Non-standard word | Standard word | Basis |
|---|---|---|
글장이 | 글쟁이 | Standard Korean Language Dictionary |
알타리무 | 총각무 | Principle for Standard Language Selection, Article 22 |
낌 | 낌새 | Standard Korean Language Dictionary |
The non-standard word dictionary is built on the Standard Korean Language Dictionary's rule system
Bareun's non-standard word dictionary is built using the non-standard word rule system of the Standard Korean Language Dictionary. It follows exactly which spellings the Standard Korean Language Dictionary treats as non-standard and which standard words it maps them to, so correction results are consistent with the National Institute of Korean Language's rules.
How it is judged
Non-standard word correction looks at the dictionary and morpheme information together.
- Urimalsaem and Standard Korean Language Dictionary lookup: It finds whether a word is standard and its corresponding standard word in
urimal-dict.fb(Urimalsaem) and the non-standard to standard mapping (non-std-dict.fb). - Proper noun (NNP) filter: Proper nouns such as names of people, places, and businesses are normal even if they are not in the dictionary. To avoid wrongly correcting them as non-standard words, eojeols (space-delimited word units) analyzed as proper nouns are screened out of correction by applying a morpheme probability threshold.
Why filter out proper nouns?
Proper nouns such as 머스크 (Musk) and 엔비디아 (NVIDIA) may not be in a standard dictionary. Without the filter, they could be mistaken for non-standard words
and changed to something unrelated. So when morphological analysis sees an eojeol as a proper noun (NNP) with high enough probability,
it is excluded from correction to prevent over-correction.
Automatic adjustment of the following particle by final consonant
When a non-standard word is changed to a standard one, the last syllable of the word, especially whether it has a final consonant, often changes.
Korean particles change form depending on the final consonant of the preceding word (을/를, 이/가, 은/는, 과/와),
so if you change only the word and leave the particle, the sentence becomes awkward.
Bareun adjusts this automatically.
graph LR
A["낌을"] --> B[Non-standard word detected: 낌 → 낌새];
B --> C{Final consonant of the corrected word changed?};
C -- Final consonant gone: 낌새 --> D[Adjust the following particle<br>을 → 를];
D --> E["낌새를"];
낌을→낌새를: The을that attached to낌(final consonant ㅁ) becomes를after낌새, which has no final consonant.
By handling headword correction and particle adjustment together in this way, the corrected result is a natural sentence as is.
Returned with the grounds
Non-standard word corrections come with an explanation. ReviseHelp in the response's helps
provides the reason for the correction (comment) and the source of the rule (rule_article).
For example, the 알타리무 → 총각무 correction carries "Principle for Standard Language Selection, Article 22" as its basis.
To turn off non-standard word correction
There is no single option that turns off non-standard word correction itself, but if you do not need standard word replacement,
you can control the pipeline's behavior by adjusting other correction options in RevisionConfig.
For the meaning of each correction category, see the RevisionCategory documentation.
Frequently asked questions
Q. When a non-standard word is fixed, does the particle change too?
Yes. When the correction changes the final consonant of the word, the particle that follows (을/를, 이/가, etc.) is adjusted automatically.
For example, 낌을 becomes 낌새를, with the word and particle corrected together.
Q. Aren't proper nouns wrongly corrected as non-standard words?
Eojeols analyzed as proper nouns (NNP) are screened out of correction by applying a morpheme probability threshold.
This prevents names of people, places, and businesses that are not in the dictionary from being mistaken for non-standard words and changed.
Q. Where do the grounds for corrections come from?
They are based on the National Institute of Korean Language's Standard Korean Language Dictionary and Urimalsaem (urimal-dict.fb).
The reason for the correction and the source of the rule are provided in ReviseHelp (comment, rule_article) in the response's helps.
Q. How is the non-standard word dictionary built?
It is built using the non-standard word rule system of the Standard Korean Language Dictionary. It follows exactly which spellings the Standard Korean Language Dictionary treats as non-standard and which standard words it maps them to, so Bareun's correction results stay consistent with the National Institute of Korean Language's rules.