How the 99.6% accuracy was measured: evaluation sets and metrics
Bareun reaches 99.6% part-of-speech (POS) tagging accuracy and 99.7% eojeol (a space-delimited word unit) segmentation/restoration accuracy in Korean morphological analysis.
This page explains what these numbers mean and which evaluation sets and metrics were used to measure them.
Accuracy depends on "what data it was measured on and by what criteria," so it is important to state the measurement assumptions together with the numbers.
Training and evaluation corpora used for measurement
Bareun was trained and evaluated on the following large-scale Korean corpora.
| Corpus | Size | Notes |
|---|---|---|
| Sejong Corpus | About 12 million eojeols | Main corpus, based on written and spoken text |
| Modu Corpus (Everyone's Corpus) | About 3 million eojeols | Released by the National Institute of Korean Language (NIKL) |
| News (BigKinds) | About 100 million eojeols | Supplements POS tagging |
| In-house supplements and word spacing augmentation | About 330,000 eojeols | Supplements low-frequency data for segmentation rules, spoken language, and word spacing cases (training data) |
| Ambiguity data | 35,396 sentences / 8,285 surface forms | Public dataset |
Why we disclose the corpora
Accuracy figures cannot be meaningfully compared when the evaluation sets differ. Bareun bases its evaluation
on widely validated standard corpora such as the Sejong and Modu corpora, and states clearly which data each figure comes from.
Definitions of the two metrics
Bareun's accuracy consists of values measured separately for two different stages.
| Metric | What is measured | Meaning |
|---|---|---|
| Eojeol segmentation/restoration 99.7% | Segmentation (Segmenter) stage | Whether eojeols are split correctly at morpheme boundaries, and whether conjugated, contracted, or omitted forms are restored to their base forms |
| POS tagging 99.6% | POS tagging (PosTagger) stage | Whether each separated morpheme is assigned the correct one of the 47 POS tags defined by NIKL |
- Eojeol segmentation/restoration looks at "where the eojeol is split and how it is restored to its base form."
For example, it evaluates whether conjugated or contracted eojeols such as
구르+어 먹+어서are restored accurately to their base-form morphemes. - POS tagging looks at whether the tag attached to each morpheme produced by that split matches the gold tag. Tagging accuracy is only meaningful when segmentation is correct first, so the two metrics are presented separately.
Why segmentation and tagging are measured separately
Bareun performs analysis in two steps: segmentation, then POS tagging. Accuracy therefore has to be measured per stage
to show where the system is strong and where there is room to improve. Lumping the two metrics together would hide the quality of each stage.
graph LR A[Gold corpus] --> B[Compare segmentation results]; A --> C[Compare POS tagging results]; B --> D[Eojeol segmentation/restoration 99.7%]; C --> E[POS tagging 99.6%];
Each morpheme result also carries a probability (softmax confidence), so you can see how confident the model was.
What exactly is the 99.6% measuring?
The 99.6% POS tagging figure is a sentence-level accuracy measured on 150,000 sentences from the Modu Corpus, checking all 47 NIKL POS tags against the gold labels. The scoring is very strict: if even one morpheme has a wrong tag in a sentence, the whole sentence counts as wrong. A sentence is accepted only when it is entirely correct, not when most morphemes are, so the figure is conservative for the same model.
We also evaluate ambiguity cases separately, where the eojeol looks the same but the analysis differs.
| Evaluation item | Evaluation size | What is measured |
|---|---|---|
| Full 47-tag POS tagging | 150,000 sentences from the Modu Corpus | One wrong morpheme makes the whole sentence wrong |
| Ambiguity resolution | 35,396 sentences / 8,285 surface forms | Telling apart different analyses of the same surface form from context |
| VCP (copula '이') restoration | 12,129 sentences | Correctly restoring the copula in sentences where '이' is omitted |
- Ambiguity resolution: when the surface form is the same but the analysis differs (for example
트는→ 틀다 or 트다), the context decides. We built an evaluation set of 35,396 sentences / 8,285 surface forms by supplementing surface forms that were under-represented. The composition and public format of this set are covered in more detail in Ambiguity evaluation dataset. - VCP restoration: we extracted 12,129 sentences from the Modu Corpus in which the copula '이' is omitted, and evaluate whether the system restores
the '이' that does not appear on the surface as a morpheme, as in
평갑니다(평가/NNG+이/VCP+ㅂ니다/EF).
The key point is that 99.6% is achieved even under these demanding criteria.
Performance charts
The charts below summarize Bareun's training scale and evaluation accuracy based on presentation material.
They cover everything from segmentation and tagging model training statistics to ambiguity resolution, VCP restoration, and whole-sentence tagging accuracy.
Figure 1. Segmentation (Segmenter) model training statistics: training scale and segmentation/restoration performance
Figure 2. POS tagging (PosTagger) model training statistics: 47-tag training scale and performance
Figure 3. Ambiguity resolution accuracy: evaluation set of 35,396 sentences / 8,285 surface forms
Figure 4. VCP (copula '이') restoration accuracy: evaluation set of 12,129 sentences
Figure 5. Whole-sentence 47-tag POS tagging accuracy: 150,000 sentences, one wrong morpheme makes the sentence wrong
Source of the charts
The five figures above are taken from the training statistics and evaluation accuracy slides of the presentation "How Bareun spell checking works inside" (바른 맞춤법 검사의 내부 원리).
Figures depend on the measurement assumptions
The 99.6% and 99.7% figures are our own measurements under the evaluation sets and metric definitions stated above. If the evaluation set, tag set, gold criteria, or preprocessing differ, the same analyzer can produce different figures. When comparing with other analyzers, a fair comparison requires measuring on the same evaluation set, metrics, and criteria.
Related documents
- Looking into accuracy: internal factors behind the accuracy gains
- How accurate is it?: frequently asked questions about accuracy
- Bareun vs Mecab-ko: points to keep in mind when comparing with other analyzers
Frequently asked questions
Q. What is the difference between 99.6% and 99.7%?
99.7% is the eojeol segmentation/restoration accuracy of the segmentation stage, and 99.6% is the POS assignment accuracy of the POS tagging stage. Because the analysis has two stages, the metrics are measured separately for each.
Q. What data was used for measurement?
The system was trained and evaluated on the Sejong Corpus (about 12 million eojeols), the Modu Corpus (about 3 million eojeols), BigKinds news (about 100 million eojeols), and others. The ambiguity data was built at a scale of 35,396 sentences / 8,285 surface forms and is publicly available.
Q. Is 99.6% a morpheme-level or a sentence-level accuracy?
It is sentence-level. All 47 POS tags are scored on 150,000 sentences from the Modu Corpus, and if even one morpheme in a sentence has a wrong tag, the whole sentence counts as wrong. This is much stricter than morpheme-level scoring, so the figure is conservative.
Q. Does it also analyze sentences where the copula '이' is omitted?
Yes. We extracted 12,129 sentences from the Modu Corpus in which '이' is omitted and evaluate VCP (copula) restoration separately.
As with restoring 평갑니다 to 평가/NNG+이/VCP+ㅂ니다/EF, it brings back the '이' that does not appear on the surface as a morpheme.
Q. Will I get the same accuracy on my data?
Figures can differ when the evaluation set differs. If your data contains many domain-specific terms, you can improve accuracy by registering a custom dictionary, and custom dictionary changes take effect without downtime.