Transformer vs dictionary-based analyzers: what is different
Korean morphological analyzers fall broadly into two groups:
dictionary-based analyzers, which match words listed in a dictionary, and transformer-based analyzers, which train on large-scale data and infer from context.
Bareun belongs to the latter. This article lays out the differences between the two approaches.
How the two approaches work
graph TD
subgraph Dictionary-based
A1[Input] --> A2[Match words in the dictionary];
A2 --> A3{In the dictionary?};
A3 -->|Yes| A4[Analysis succeeds];
A3 -->|No| A5[Analysis fails: unregistered];
end
subgraph Transformer-based
B1[Input] --> B2[Context-trained model inference];
B2 --> B3[Guess segmentation and POS];
B3 --> B4[Out-of-vocabulary words also get a guessed result];
end
Key differences compared
| Item | Dictionary-based | Transformer-based (Bareun) |
|---|---|---|
| Basis of analysis | Whether the word is listed in the dictionary | Learned contextual patterns |
| Out-of-vocabulary words | Analysis tends to fail | Can be guessed from context |
| New words and colloquial speech | Weak until the dictionary is updated | Handled by learned patterns |
| Ambiguity | Depends on rules and priorities | Judged from context |
| Adaptability | Depends on dictionary maintenance | Improves by training on data |
Strengths of the transformer-based approach
Strong on out-of-vocabulary words and new words
A dictionary-based analyzer tends to get stuck when it meets a word that is not in the dictionary.
Even for a word that is in no dictionary, Bareun looks at the surrounding context, guesses the segmentation and POS, and marks it OUT_OF_VOCAB. In a Korean environment where new words and colloquial speech appear constantly, this adaptability makes a big difference.
Colloquial and informal input
Unrefined sentences such as social media posts, chats, and speech transcripts are hard to handle with dictionary matching. Bareun trains with word spacing augmentation and large amounts of real sentences (on the order of 100 million eojeol (space-delimited word units) and 10 million sentences), so it is not shaken by input with wrong word spacing or mixed-in colloquial speech.
Resolves ambiguity by context
Bareun embeds words that have the same surface form but different morphological forms as different tokens. The number of identical surface forms distinguished by embeddings that carry morphological information reaches 12,175, and where a dictionary-based analyzer relies on rules and priorities, Bareun judges from context.
It does not discard the dictionary
It uses both together
Bareun being transformer-based does not mean it does not use dictionaries. It uses the built-in dictionary, Urimalsaem (Open Korean Dictionary), and the custom dictionary together to adjust inference results and show their source. Bareun's way is to use the adaptability of the trained model together with the accuracy of dictionaries.
Frequently asked questions
Q. How do transformer-based and dictionary-based analyzers differ?
A dictionary-based analyzer analyzes by matching listed words, so it is weak on words that are not in the dictionary. Bareun, which is transformer-based, infers from learned contextual patterns, so it can guess even out-of-vocabulary words.
Q. Does Bareun not use dictionaries at all?
No. It infers with the trained model but also uses the built-in dictionary, Urimalsaem, and the custom dictionary to adjust the result and show the source of each morpheme. It uses the adaptability of learning together with the accuracy of dictionaries.
Q. Why is it strong on new words and colloquial speech?
Because it trained on real sentences on the order of 100 million eojeol and 10 million sentences, plus word spacing augmentation, and it guesses even words that are not in the dictionary from context. It also works stably on informal input such as social media and chat.
Q. Why is it stronger on ambiguity than dictionary-based analyzers?
Bareun embeds words that have the same surface form but different morphological forms as different tokens. The number of identical surface forms distinguished by embeddings that carry morphological information reaches 12,175, and where a dictionary-based analyzer relies on rules and priorities, Bareun judges from context.