Our own training corpus: how we refined it
Good models come from good data. But even public corpora have tagging errors and consistency problems.
Bareun does not use the Sejong, Modu Corpus, and news corpora as they are; it went through a refinement process in which errors were corrected and
consistency was aligned with our own tagging guidelines. This article introduces that refinement work.
Why refinement is needed
Large corpora are tagged by hand, so problems like the following are mixed in.
| Problem type | Description |
|---|---|
| Tagging errors | Cases where the morpheme or POS was attached wrongly |
| Segmentation inconsistency | Cases where the same word is split differently from document to document |
| Guideline differences | The tagging criteria differ subtly from corpus to corpus |
When you train on data with errors mixed in
A model learns the patterns in the data as they are. If the same word is tagged as one morpheme in some places and as two morphemes in others in the corpus, the model learns a confusing signal and accuracy drops.
Aligning consistency with our own tagging guidelines
Bareun created its own tagging guidelines to apply consistently across the corpus, and polished the data to this standard.
A typical example is the handling of 누가.
누가 → 누 + 가
We settle by guideline how to segment and tag 누가 (a contracted form of the pronoun '누구' (who) combined with the subject particle),
and unify it in the same way across the whole corpus. Accumulating such decisions word by word produces consistent training data.
graph TD RAW[Original corpus] --> FIX[Fix tagging errors]; FIX --> GUIDE[Apply our tagging guidelines]; GUIDE --> UNIFY[Unify segmentation and POS consistency]; UNIFY --> CLEAN[Refined training corpus];
Refinement leads to accuracy
When you train on data from which errors were removed and which was polished with consistent guidelines, the model learns only clean, consistent signals. This refinement work underlies Bareun's 99.6% POS tagging and 99.7% eojeol (space-delimited word unit) splitting/restoration accuracy.
Refinement and data augmentation go as a pair
If refinement is work that raises the quality of the existing corpus, alongside it goes augmentation work that widens the breadth of the data. To fill in the colloquial speech, dialects, and new words that a corpus centered on written language was weak on, Bareun added speech- and conversation-based datasets and augmented the training data to a scale of about 100 million eojeols / 10 million sentences.
Quality (refinement) and diversity (augmentation) at the same time
Clean data with errors removed (refinement) and broad data containing real conversation, dialects, and new words (augmentation) complement each other. With refinement alone, you lean toward standard written language; with augmentation alone, the signal wobbles. Bareun does the two as a pair to get both precision and diversity.
Frequently asked questions
Q. Why don't you use public corpora as they are?
Large corpora have errors made during human tagging and guideline differences between corpora mixed in. If you train on such data as is, the model learns confusing signals and accuracy drops.
Q. What do the in-house tagging guidelines do?
They are the segmentation and POS criteria applied consistently across the corpus. For example, we decide to process 누가 as 누 + 가
and unify it across the whole corpus, so the same word is always tagged the same way.
Q. Does refinement affect model accuracy?
Yes. When you train on data with errors removed and consistency aligned, the model learns only clean signals. This is the basis of the high morphological analysis accuracy.
Q. How do corpus refinement and data augmentation differ?
Refinement is work that raises quality by fixing errors in the existing corpus and aligning consistency, and augmentation is work that widens breadth to a scale of about 100 million eojeols / 10 million sentences by adding colloquial, dialect, and new-word data. Bareun does the two as a pair to get both precision and diversity.