Adapting to colloquial speech and new words, compared with other analyzers
Korean constantly produces colloquial forms, contractions, and new words. Sentences that mix in spoken-style or shortened forms, such as "감사합니당", "그치만", and "ㅂ니당", are difficult input for analyzers that rely only on a dictionary.
Bareun handles these expressions well because its transformer-based segmentation and tagging models are trained on colloquial and augmented data, and because it guesses out-of-vocabulary words from context. This page explains how Bareun adapts to colloquial speech and new words.
Colloquial and new-word handling compared
| Item | Bareun | Dictionary-dependent analyzers |
|---|---|---|
| Colloquial and contracted forms | Guessed and restored through training on colloquial forms | Tend to break when the form is not in the dictionary |
| Out-of-vocabulary words (OOV) | Guessed from context, with the source marked | Weak at estimating |
| New words | Add to the custom dictionary with no downtime | Dictionary rebuild required |
| Form restoration | Restores conjugated, contracted, and omitted forms to their base form | Limited |
| Source distinction | out_of_vocab tells dictionary entries from guesses | Usually not provided |
Why it handles colloquial speech well
Before assigning parts of speech, Bareun runs a segmentation stage dedicated to deciding where to split. Colloquial and contracted forms also get their morpheme boundaries set in this stage, and conjugated, contracted, or omitted forms are returned to their base form. Even for expressions that are not in a dictionary, the AI model produces a plausible analysis from context. Where an ending (eomi, a verb or adjective suffix) should appear but speech alters it, Bareun is trained to recognize that an ending it has not learned has appeared. This makes it very flexible with varied ending changes.
The basis for colloquial adaptation: large-scale speech and conversation data
Bareun is strong on colloquial speech not by chance but because it was trained on a large amount of data close to how people actually talk.
On top of standard corpora, it was augmented with about 100 million eojeols (space-delimited word units) / 10 million sentences from the following speech and conversation datasets.
| Dataset | Nature |
|---|---|
| NIKL Daily Conversation 2020 | Everyday conversational speech |
| NIA emotional dialogue | Dialogue mixed with emotional expression |
| NIA free speech (general, speakers in their 10s to 50s / seniors, in their 60s) | Colloquial free speech by age group |
| NIA Korean conversation speech data | Transcripts of real conversational speech |
| NIA meeting speech data | Meeting utterances |
| NIA dialect speech data | Regional dialects |
Because it was trained on a large amount of real utterance data such as free speech, dialects, and meetings, it absorbs a wide range of contractions, dialect forms, and colloquial ending changes that are hard to handle with refined written text alone. The full composition of the training data is covered in Training data. The training process is also designed to treat all low-frequency tokens as UNKNOWN, so even when a new word appears for the first time, the model recognizes that an unknown word has shown up and tries to analyze it from context.
Effect of training data augmentation: before and after adding speech and conversation data (about 100 million eojeols / 10 million sentences)
Source of the graph
The figure above is taken from the "Effect of training data augmentation" slide in the presentation "How Bareun spell checking works inside" (바른 맞춤법 검사의 내부 원리).
Guessing out-of-vocabulary words and stating the source
When Bareun meets a word that is not in a dictionary, it does not simply fail. It attaches a source marker to the result the model guessed.
The side that receives the result can tell whether a word was judged OOV by guessing or is based on a dictionary entry with a known source. While many OOV words are checked against the built-in dictionaries, their source may change to a dictionary.
How OOV words are guessed and sourced is covered in more detail in OOV handling.
Words found in a dictionary have no probability value
When Bareun converts a result to a value found in a dictionary, it discards the probability that the AI model originally returned.
The probability value becomes 0.
Each morpheme carries an out_of_vocab value that shows whether it came from the training embeddings, was guessed automatically by the model, or came from one of the dictionaries (custom, default, Urimalsaem, Wiki, Onterm). The code that processes the response can therefore tell whether a new word is a provisional guess or a formally registered dictionary entry.
For the full list of values and how each is shown in the morphological analyzer and the spell checker (the Urimalsaem, Wiki, and Onterm sources appear only in the spell checker), see Morphological analysis API: words not learned.
Form restoration brings colloquial speech back
Spoken Korean often has phonological contractions and omissions. In post-processing, Bareun recalculates offsets so that restored morphemes have the exact position of the original syllable. For example, 킨데요 is restored to 키+이+ㄴ데+요, and 키, 이, and ㄴ데 are all made to point to the same start position.
graph LR A[Colloquial input: 킨데요] --> B[Segmentation and restoration]; B --> C[키 + 이 + ㄴ데 + 요]; C --> D[Restore offset to the original syllable position];
What the in-house corpus covers
Standard corpora alone cannot adequately cover colloquial speech, contractions, irregular conjugation, and unregistered nouns, so Bareun also trains on an in-house corpus (about 330,000 eojeols) that we built to fill those gaps. It covers the following.
- Irregular conjugation and phonological changes: cases where the form changes during conjugation, such as ㅂ, ㅅ, ㅎ, ㄹ, 러, and 르 irregulars, ㄹ deletion, '으' insertion, and glide combination
- Colloquial and contracted endings: ending variants that often appear in actual speech, such as '-잖아/-잖네', '-죠', '-네', '-셔/-셨', '-래/-랬', and '-쇼/-슈'
- Commonly misspelled contractions: trained to analyze easily confused contracted forms correctly, such as '되/돼', '뵈/봬', and '외/왜'
- Dialect endings: sentence-final and connective endings found in regional dialects (such as variants of the '-었-' family)
- Restoring the copula '이' after a noun: cases that restore the hidden '이' (copula, VCP) after a common noun (NNG) or proper noun (NNP), by final-consonant type (for example,
평갑니다→ 평가/NNG+이/VCP+ㅂ니다/EF) - POS tagging: easily confused tagging cases for connective and final endings (EC, EF), particles (subject JKS, object JKO, genitive JKG, adverbial JKB, auxiliary JX, and so on), and derivational affixes (XSN, XSV, XSA)
- Word spacing augmentation: strengthened so that eojeols written without spaces or with wrong spaces still get correct morpheme boundaries
- Numerals and special symbols: non-letter tokens such as Korean numerals (NR), special symbols, and English punctuation
- Ambiguity: trained to distinguish cases where the same surface form is analyzed differently depending on context (for example, whether '가' is a verb VV or an auxiliary verb VX)
Why add our own data?
Standard corpora center on refined written language, so they contain relatively few cases such as colloquial ending variants, irregular conjugation, and unregistered nouns. We fill in these low-frequency, high-difficulty cases with the in-house corpus so that segmentation and POS tagging behave more evenly on real input.
Much data is also missing when you look at how Korean is actually used
Real Korean contains many spellings and conjugations that standard corpora do not capture well. During development, whenever we found cases where the analysis was wrong, we filled them in with the in-house corpus. The table below lists representative cases that the standard corpora lacked and that we added separately.
| Case added | Examples |
|---|---|
| Low-frequency syllables and labels | Intensive reinforcement of syllable-level expressions whose low occurrence frequency (for example, 5 or fewer) hurt accuracy |
| Irregular conjugation | ㅂ, ㅅ, ㅎ irregulars, ㄹ irregular ('으' insertion), 르 irregular ('ㄹ' insertion on the stem) |
| Phonological contraction errors | Commonly misspelled contractions such as '되/돼', '뵈/봬', '외/왜', and '어었', and quotation contractions ('돼라는', '달라란') |
| Restoring the copula '이' after a noun | Forms where "noun+이+ending" is shortened, such as '걘·껍·난·낸'; vowel-final noun+이+시+오 ('쇼/슈'); 'particle+이+ending' ('에서+이+ㄴ지', '부터+이+ㄹ까') |
| Word spacing | Spacing of the genitive particle '의', spacing of numbers in units of 만 (ten thousand), and '란' after a space |
| Dialect endings | Sentence-final endings of the '-었-' family found in regional dialects |
| Non-standard and colloquial verb conjugation | '이카다/저카다' (progressive 'ㅣ' vowel assimilation), '않아하/않아해', '긇지·요렇케·시끄러' |
| Nominalizing endings | Terse bullet-style nominal forms that write the main verb and auxiliary verb together |
| Low-frequency vocabulary and expressions | '누차', '-을 뻔하-', '난이도', and so on |
| Special symbols and foreign-language notation | Sentences with uncommon symbols, English punctuation, and Roman-letter (SL) notation |
New words can be added immediately through the custom dictionary
New words and technical terms that were not part of training can be supplemented by registering them in the custom dictionary.
The custom dictionary does not use a database. It is written to the server's file system, so the server can detect right away when a new dictionary is added or a registered dictionary changes. Bareun monitors these events and applies them with no downtime, so every time a new term appears you can apply it to analysis immediately without stopping the server.
You can register proper nouns, compound nouns, verbs, adjectives, determiners, adverbs, and interjections, by dictionary type.
Notes on comparison
Other analyzers may also support adding dictionary entries or estimating some OOV words. However, the combination of colloquial training data, context-based guessing, source marking, and zero-downtime dictionary updates differs from tool to tool. Check each tool's version and documentation for specific behavior and numbers.
Related documents
- OOV handling: how words not in the dictionary are guessed
- Training data: composition of colloquial and augmented data
- Reducing OOV words with a custom dictionary: supplementing new words through the dictionary
Frequently asked questions
Q. Can colloquial text like "감사합니당" be analyzed?
Bareun is trained on colloquial and augmented data, so it tries to segment contracted and colloquial expressions like this and restore them to their base form.
Even when a form is not in the dictionary, it guesses from context and marks in the result whether the word is OOV.
Q. How do I add a newly coined word?
Register it in the custom dictionary. Bareun detects dictionary changes with no downtime and applies them from the next request.
No restart is needed.
Q. Can I tell whether a word is a guess or comes from a dictionary?
Yes. The out_of_vocab value of each morpheme tells you where the result came from: word embeddings, automatic guessing, the custom dictionary, the default dictionary, Urimalsaem (Open Korean Dictionary), and so on.
Q. What data did Bareun train on to handle colloquial speech?
On top of standard corpora, it was augmented with about 100 million eojeols / 10 million sentences of speech and conversation data, including NIKL daily conversation and NIA emotional dialogue, free speech, Korean conversation speech, meeting speech, and dialect speech. Because it learned real utterances such as free speech, dialects, and meetings at scale, it is strong on contractions, dialects, and colloquial ending changes that are hard to handle with written text alone.