How homonym disambiguation works: meaning beyond part of speech (beta)

Morphological analysis tells you that 다리 is a noun (NNG). But whether that 다리 is the bridge you cross (橋) or the leg of a body (脚) is not something part of speech (POS) can tell apart. Bareun adds a layer that picks the meaning from context on top of morphological analysis, so it can attach a dictionary sense number (superscript number) to each morpheme.

This document explains the principle, the training data, and the current level of quality. For how to use it, see the Homonym sense disambiguation API.

Why it is a hard problem

Telling Korean homonyms apart runs into three obstacles.

1. There are many candidates, and most share the same POS.

배 can be 배 (belly, 腹), 배 (ship, 船), 배 (pear, 梨), 배 (times, 倍), 배 (cup, 杯), and so on. The dictionary has nearly ten entries, and many of them are NNG. POS tagging gives no help with this distinction.

2. The distribution of meanings is extremely skewed.

In real text, some meanings are used thousands of times and others only a few times. So a simple rule that always picks "the most common meaning of that word" already reaches fairly high accuracy. For a model to be worthwhile, it must be clearly better than this most-frequent-sense rule.

3. There is not enough answer data.

Korean corpora with sense numbers attached by hand are limited, and each dictionary divides meanings by different criteria, so they cannot be merged as they are.

graph TD
  A["'다리' / NNG"] --> B{"Context"};
  B -->|"...를 건넜다 (crossed it)"| C["다리 5: structure (bridge)"];
  B -->|"...가 저리다 (is numb)"| D["다리 1: body (leg)"];
  B -->|"안경 ... (glasses ...)"| E["다리 3: supporting part of an object (temple/leg)"];

A dedicated model judges the meaning

Bareun's sense judgment is done by a dedicated model separate from morphological analysis. After the morphological analyzer splits the sentence into morphemes and attaches POS, the sense model takes that result and picks which meaning each morpheme is used in.

graph LR
  S["Sentence"] --> M["Morphological analysis<br/>(segmentation + POS)"];
  M --> W["Sense judgment<br/>(dedicated context model)"];
  W --> R["Morpheme + superscript number<br/>+ definition + probability"];

At first, we built a single model that produced POS and sense together. It had the advantage of low cost because the context representation was not built twice, but when we measured it on real sentences, its ability to tell senses apart was almost no different from the most-frequent-sense rule. So we separated out the sense judgment and rebuilt it as a dedicated model that reads the context representation more deeply. In 3.1.0, we changed that dedicated model once more so that it takes each candidate as a definition sentence rather than a number and compares it directly with the context (see "Current level and limits" below).

• It does not affect POS accuracy. The sense model does not change the morphological analysis result; it only adds the superscript number and definition. A call that does not request senses does not run this model at all.
• Requesting senses increases computation. Because it is a dedicated model, one more judgment runs for each sentence. If response time is critical on a path, we recommend turning it on only for the requests that need it.

For the structure of the context encoder on the morphological analysis side, see Transformer model architecture.

The key mechanism: limit candidates to that word's dictionary entries

The most important design in sense judgment is "narrowing down the possible answers in advance".

The full list of Korean senses has tens of thousands of items (currently about 58,000 entries). If the model picked one out of all of them, it would lean toward common meanings and effectively never choose rare ones. So at the moment of judgment, Bareun

1. finds dictionary candidates by the morpheme's headword and POS, and
2. calculates probabilities only among those candidates and picks the highest.

'다리' + NNG  →  candidates [1, 3, 5, 8, …]  →  calculate probabilities among the candidates  →  5

This restriction makes a big difference.

There are also a few side effects.

• A meaning from the wrong headword is never attached. The candidates are themselves the dictionary entries of that headword and POS, so there is structurally no room for another word's definition to be attached.
• The probability in the response is meaningful. sense.probability reflects how strongly the choice was made within this candidate set, combined with how commonly that sense is actually used. Because it looks at both, the candidates' probabilities do not sum to 1. When there is only one candidate, it is always 1.0, which does not mean confidence; it means "there is only one to choose from."
• If it is not reliable, it stays empty. Where no candidate is reliable enough, instead of attaching a wrong meaning, it leaves out sense.

We unified the superscript numbering on Urimalsaem

Sense numbers differ from dictionary to dictionary. Bareun's training materials also originally mixed different standards.

Bareun converted everything to the Urimalsaem standard and unified it into a single numbering space. We chose Urimalsaem because it has the most headwords and senses and so covers the words that appear in real text, it keeps being updated, and it can be connected to definitions through public data.

The effect of unification was also confirmed in accuracy. Moving from the Standard Korean Language Dictionary standard (about 6,600 senses) to the Urimalsaem standard (over 30,000), the senses to be distinguished become more than five times finer, so absolute accuracy goes down, but the gain the model produces over the most-frequent-sense rule actually grew. Even though the task got harder, the model's contribution increased. (This comparison is a measurement from when we were switching the numbering system; for the current level, see the independent test set figures in "Current level and limits" below.)

Training data: human-labeled answers come first

For sense disambiguation, the quality of the answer data nearly decides performance. Bareun set the priorities like this.

1. Human-annotated sense labels take top priority. The sense annotations of the Sejong corpus and the lexical semantic analysis data of NIKL (about 1.46 million morphemes of news sentences) are the baseline.
2. Where they are missing, fill them in automatically, but with verification. For sentences with no sense numbers, we attach candidates with a large language model, then compare example sentences grouped under the same sense and filter out the ones that don't agree. In the range that overlaps with human answers, automatic labeling showed a large number of errors, and the lesson was that we do not use automatic labeling where human answers exist.
3. Rare senses are found in the existing corpus and added. Rather than writing new sentences, we first look for usages of that sense in sentences that already exist and label them. This keeps the style of real text.
4. We do not touch the original corpus. Material with sense numbers is managed separately from the originals, so the baseline material used for POS and segmentation training is not contaminated.

Current level and limits

What the figures were measured on matters more than the figures. Measuring on sentences of the same kind as the corpus used for training gives high values, but the level you can expect on real service sentences is lower. So Bareun keeps an independent test set (sentences newly collected from the NIKL web corpus) that was never used in training and human-reviewed answer sentences, and judges quality with those.

The current level is better than a method that ignores context and picks the most common meaning of the word. The difference shows up especially in places where the most common meaning is not the answer, and for senses rarely seen in training. However, widening that gap is the current task, and because we do not consider it finished, we provide it as beta.

There is no effect on POS decisions. The sense model does not change the morphological analysis result; it only adds the superscript number and definition.

Separating sense judgment from morphological analysis into a dedicated model improved results once, and in 3.1.0 we changed it again to take candidates as definition sentences and compare them with the context. For a sense that has a definition in the dictionary, it can choose with a basis even a sense seen rarely in training.

Even so, there are still noticeable weaknesses on real sentences. This is why we release it as beta.

The cause is mostly on the data side. The automatically attached sense annotations have no small amount of error, and the current model's accuracy has already come close to the level of the judge that produced those annotations. In other words, making more automatic annotation the same way will not raise it much. The next work is to use agreement among several judges of different kinds, or to have people attach answers directly, and to find and add usages of rare senses in the actual corpus.

Summary

• Sense judgment is done by a dedicated model separate from morphological analysis. It does not run unless requested, and it does not change the morphological analysis result itself.
• Judgment happens only among the dictionary candidates of that headword and POS. So nonsense meanings are not attached, and the response probability becomes an interpretable value.
• Sense numbers are unified on the Urimalsaem standard, and definitions are contained in the result in advance.
• Human answers come first, and automatic labeling is used only as a supplement after verification.
• Quality is judged with an independent test set not used in training and human-reviewed answer sentences, and repeated occurrences, short sentences, and rare senses remain as open tasks.

For comparison with other tools and resources, see Comparison of Korean homonym analysis resources.

Frequently asked questions

Q. Do you run an additional model for sense disambiguation?

A. Yes. Sense judgment is done by a dedicated model separate from morphological analysis. A call that doesn't turn on with_sense doesn't run this model at all and costs exactly the same as before; when it is turned on, one more judgment runs for each sentence.

Q. Can a meaning that isn't in the dictionary come out?

A. No. The judgment candidates are themselves limited to the dictionary entries of that headword and POS, so a number that is not in the dictionary or another word's meaning is never attached.

Q. Why is Urimalsaem the standard for superscript numbers?

A. It has the most headwords and senses and so covers the words in real text, and it can be connected to definitions through public data. We also unified the numbering standards, which differed across training materials, into Urimalsaem alone.

Q. How is this different from picking the most frequent sense?

A. On an independent test set not used in training, Bareun's model beats the most-frequent-sense rule. The difference is largest in places where the most common meaning is not the answer. The most-frequent-sense rule, by definition, gets none of those right, but a model that reads context gets a good share of them. However, in short sentences with little contextual clue, results still tend to drift toward the most frequent sense.