Automatic compound noun decomposition: why it improves search and LLM tokenizer accuracy
Bareun automatically decomposes compound nouns made by joining several words.
It splits 정보통신기술 (information and communication technology) into meaning units such as 정보 + 통신 + 기술,
and this feature plays a big part in improving the indexing accuracy of search engines and the tokenizer quality of LLM and RAG systems.
How compound noun decomposition works
Compound noun decomposition happens in a post-processing step after morphological analysis. It looks again at the eojeol (space-delimited word unit) that morphological analysis recognized as a single noun, and if it is a registered compound noun, splits it into its component nouns.
graph TD IN[한국어자연어처리] --> SCAN[Compound noun decomposition]; SCAN --> A[한국어]; SCAN --> B[자연어]; SCAN --> C[처리];
Split points are marked with the ^ symbol in the result. This way you can see at a glance
where a compound noun that was originally one piece was split.
한국어자연어처리 → 한국어^자연어^처리
Why show it both whole and split
If you split a compound noun into small pieces unconditionally, the information that it was originally one concept is lost.
With the ^ mark, Bareun also keeps the fact that "this was originally one compound noun, split at these points,"
so the consumer can choose to use it whole or split.
Why accuracy goes up in search and RAG
Splitting compound nouns brings the following benefits to search and LLM pipelines.
Search engines
If you index only 정보통신기술, documents will not be found when a user searches for 통신 or 기술.
If you decompose it into 정보^통신^기술 before indexing, partial-word searches match exactly as well.
LLM and RAG tokenizers
LLM tokenizers tend to break Korean compound nouns into small pieces unrelated to meaning. If Bareun decomposes them into meaning units first, meaning is preserved at the chunking and embedding stages, and retrieval accuracy goes up.
| Use | Before decomposition | After decomposition | Effect |
|---|---|---|---|
| Search index | 정보통신기술 | 정보 / 통신 / 기술 | Matches partial-word searches |
| RAG chunking | One lump token | Meaning-unit tokens | Preserves embedding meaning |
| Keyword extraction | One long compound word | Several key terms | Richer keywords |
Controlling split points with the custom dictionary
Compound noun decomposition works together with the compound noun split dictionary (CP^) in the custom dictionary.
If you register a compound noun with split points that you mark yourself with ^, that word is decomposed
at the points you want whenever it appears. This is useful when you want to handle domain-specific terms accurately.
Frequently asked questions
Q. What does the ^ in a compound noun decomposition result mean?
It marks a split point: the word was originally one compound noun and is split into component nouns at this point.
For example, 정보통신기술 is shown as 정보^통신^기술.
Q. Why does compound noun decomposition help search and RAG?
Splitting compound nouns into meaning units makes partial-word searches match, and meaning is preserved at the chunking and embedding stages of a RAG pipeline, which raises retrieval accuracy.
Q. Can I specify the split points myself?
Yes. If you register a word in the compound noun split dictionary (CP^) of the custom dictionary with its split points marked by ^,
that word is decomposed at the points you want.