Model training data: unified training on Sejong, Modu Corpus, and BigKinds, 100 million eojeol
The quality of a morphological analysis model ultimately depends on what it was trained on.
Bareun trains on a broad collection that ranges from standard corpora to large-scale news corpora and its own augmented data.
This article summarizes what data Bareun was trained on.
Training corpus composition
| Corpus | Size | Character |
|---|---|---|
| Sejong Corpus | About 12 million eojeol (space-delimited word units) | Standard corpus of the National Institute of Korean Language (NIKL), refined written and spoken text |
| Modu Corpus (NIKL "Corpus for Everyone") | About 3 million eojeol | NIKL modern Korean corpus |
| News (BigKinds) | About 100 million eojeol | Large-scale real news sentences |
| Own augmented data | (continuously expanding) | Error and exception cases tagged by hand |
graph TD S[Sejong 12M] --> MERGE[Unified training]; M[Modu Corpus 3M] --> MERGE; N[News 100M eojeol] --> MERGE; O[Own augmentation] --> MERGE; MERGE --> SEG[Segmentation model]; MERGE --> TAG[Tagging model];
Why combine several corpora
Each corpus has different strengths
- The Sejong and Modu corpora are refined data tagged precisely with a standard tag set, so they set the standard for analysis.
- The news corpus (BigKinds) consists of real sentences at the scale of 100 million eojeol, so it teaches diverse vocabulary, expressions, and proper nouns.
- The own augmented data fills in the error and exception cases that other corpora miss.
With standard corpora alone, the model cannot keep up with the variety of sentences in the real world, and with news alone, tagging consistency wavers. Bareun combines the two to get precision and diversity at the same time.
Reinforcing training data for spoken language, dialects, and neologisms
Corpora centered on written text make it hard to keep up with everyday conversation, dialects, and neologisms. So Bareun brought in additional speech and conversation-based datasets and reinforced the training data to a scale of about 100 million eojeol / 10 million sentences.
| Reinforcement dataset | Character |
|---|---|
| NIKL Everyday Conversation 2020 | Everyday spoken conversation |
| NIA emotional dialogue | Dialogue containing emotional expression |
| NIA free speech (general, ages 10 to 50) | Free speech across a broad age range |
| NIA free speech (elderly, 60s) | Speech characteristics of older speakers |
| NIA Korean conversational speech data | Transcripts of conversational speech |
| NIA meeting speech data | Speech in meeting situations |
| NIA dialect speech data | Regional dialect speech |
Effect of reinforcement: adaptability to spoken language, dialects, and neologisms
With this data added to the training data, Bareun became much more adaptable not only to written text but also to spoken language, dialects, and neologisms. It reliably identifies morpheme boundaries and parts of speech even in real conversational sentences, dialect expressions, and newly coined words.
Word spacing augmentation: robust to wrong spacing too
Word spacing augmentation is applied to the training data. It is a method of training on variants in which the word spacing of normal sentences has been deliberately altered in various ways.
Why augment
Real user input has uneven word spacing. People write words together, like 아름다운강산, or
space them wrongly, like 떨여 졌는데. Training with word spacing augmentation lets the model
find morpheme boundaries without being thrown off by such irregular input.
The unified corpus gathered this way and the augmentation strategy together produce accuracy of 99.6% for POS tagging and 99.7% for eojeol splitting/restoration.
Frequently asked questions
Q. What data was Bareun trained on?
It was trained on a combination of about 12 million eojeol of the Sejong Corpus, about 3 million eojeol of the Modu Corpus, about 100 million eojeol of BigKinds news, and its own augmented data.
Q. Why use a news corpus too?
Standard corpora alone cannot capture enough of the diverse vocabulary, expressions, and proper nouns of the real world. A news corpus at the scale of 100 million eojeol reinforces vocabulary diversity and real sentence patterns.
Q. What is word spacing augmentation?
It is a method of training together with variants in which the word spacing of normal sentences has been deliberately altered in various ways. This lets the model find morpheme boundaries accurately even when users get word spacing wrong.
Q. Does Bareun analyze spoken language and dialects well?
Yes. Bareun reinforced its training data to a scale of about 100 million eojeol / 10 million sentences by adding the NIKL Everyday Conversation 2020 and the NIA emotional dialogue, free speech (general and elderly), Korean conversational speech, meeting speech, and dialect speech datasets. As a result, its adaptability to spoken language, dialects, and neologisms has improved greatly.
Q. How large is the training data reinforcement?
By adding speech and conversation-based datasets, it was reinforced to a scale of about 100 million eojeol / 10 million sentences. This reinforcement improved the analysis of conversational style, dialects, and neologisms, which was weak with corpora centered on written text alone.