Fine-tune tokenization with the compound noun split (^) dictionary

If you analyze 형태소분석 (morphological analysis) as is, it may come out as a single noun chunk. But for search indexing or RAG embedding, 형태소 (morpheme) and 분석 (analysis) must be captured as separate tokens for search to work well. The compound noun split dictionary (cp_caret_set) in Bareun lets you specify split points with the ^ symbol, and fine-tunes tokenization so that a closed-up word is split into two or more parts at the positions you intend.

This document explains the ^ notation, how the split dictionary works, and how to use it in search and RAG.

Mark split points with `^`

Unlike entries in ordinary dictionaries, entries in the compound noun split dictionary use the ^ symbol to mark where to cut. It suits words that the rules of the National Institute of Korean Language say should be written with spaces, but that people often write closed up in practice.

Input text	Registered entry	Tokenization result
`형태소분석`	`형태소^분석`	`형태소` + `분석`
`디지털인문학`	`디지털^인문학`	`디지털` + `인문학`
`데이터산업`	`데이터^산업`	`데이터` + `산업`
`코로나백신`	`코로나^백신`	`코로나` + `백신`

Difference between compound nouns (CP) and compound noun splitting (CP^)

The compound noun dictionary (cp_set) binds a term such as 코로나19 as a single word (NNG). The compound noun split dictionary (cp_caret_set) does the opposite: it splits closed-up characters at the points you intend. Remember it this way: "I want it treated as one word → CP", "I want to split it into words for indexing → CP^".

Registration example

The ^ symbol is an exception that is allowed only in the compound noun split dictionary. You cannot put the symbol in any other dictionary entry.

from bareunpy import Tagger

my_tagger = Tagger('YOUR-API-KEY', 'localhost', port=5656)

cust_dic = my_tagger.custom_dict("search_index")
cust_dic.copy_cp_caret_set({'형태소^분석', '디지털^인문학', '데이터^산업'})
cust_dic.update()

How the split looks

If you analyze after registering, the word splits into two common nouns, and each morpheme is marked as coming from the custom dictionary.

디지털인문학 강좌에서 형태소분석을 배웠다.

(Roughly: "I learned morphological analysis in a digital humanities course.")

디지털/NNG, IN_CUSTOM_DICT
인문학/NNG, IN_CUSTOM_DICT
형태소/NNG, IN_CUSTOM_DICT
분석/NNG, IN_CUSTOM_DICT

Use in search and RAG tokenization

Search engines and RAG pipelines usually index and embed noun-level tokens. If a compound word is bound into one chunk, a search for "형태소" may not match a document containing "형태소분석". If you cut compounds with the compound noun split dictionary, partial-word search works naturally.

Use the compound noun split dictionary when

You want internal document search to find "데이터산업" with either "데이터" or "산업" alone
You want to split RAG embedding input into meaningful noun units
You want to cut domain-specific compound words consistently, which is hard to leave to automatic word spacing alone

Choose split points carefully

Put ^ only at boundaries where the meaning is still alive. Forced splits can even lower search quality. If a newly coined word has settled into a single word, the compound noun dictionary (cp_set) is the right choice, not the split dictionary.

Frequently asked questions

Q. In which dictionary can I use `^`?

It is an exception symbol that works only in the compound noun split dictionary (cp_caret_set). You cannot put the symbol in any other dictionary entry, including the proper noun, compound noun, verb, and adjective dictionaries.

Q. Can I split into three or more parts?

Yes. Add more split points by putting ^ in several times. Still, marking only noun boundaries where the meaning is alive is better for search quality.

Q. How is this different from automatic word spacing?

Automatic word spacing (auto_spacing) adjusts eojeol (space-delimited word unit) boundaries with general rules. The compound noun split dictionary is more deterministic, because it guarantees that a specific domain compound word is always cut at the same point.