Korean chunking in RAG with Bareun morphological analysis
Half of the quality of a RAG (retrieval-augmented generation) pipeline is decided by how you split documents (chunking).
If you cut blindly by character count, Korean text gets cut in the middle of noun phrases or right through particles (josa),
and search embeddings fail to capture the meaning. If you build Bareun morphological analysis into the chunk boundary decision,
you get chunking that preserves meaning units.
The problem
Fixed-length splitting causes the following problems in Korean.
- A compound noun such as
인공지능기반자연어처리(AI-based natural language processing) gets cut at a chunk boundary and is missed by search. - Particles and endings (eomi) are scattered across neighboring chunks, so the embedding vector is affected by unnecessary function morphemes.
- A chunk cut in the middle of a sentence loses its context when passed to an LLM.
How to apply Bareun
Bareun breaks an eojeol (a space-delimited word unit) into the eight major morphological units, such as nominals (N), particles (J), predicates (V), and endings (E), and splits registered compound nouns into smaller nouns (compound noun decomposition). You can use this in two ways.
graph TD DOC[Source document] --> SPLIT[Sentence splitting]; SPLIT --> TAG[Bareun morphological analysis]; TAG --> NOUN[Extract nouns and compound nouns]; NOUN --> BOUND[Decide chunk boundaries by noun density]; BOUND --> CHUNK[Meaning-preserving chunks]; CHUNK --> EMB[Embeddings and vector DB];
- Noun-based chunk boundaries: Split by sentence, but attach short sentences with almost no nouns to the previous chunk so that chunks have a similar "content density".
- Decomposed compound noun indexing: Keep the chunk body as it is, and also store the decomposed nouns as search keywords
so that queries for both
자연어처리(natural language processing) and자연어(natural language) or처리(processing) match.
from bareunpy import Tagger
# Get an API key at https://bareun.ai ("koba-...")
tagger = Tagger("koba-XXXX-...", "localhost")
def chunk_keywords(sentence: str):
"""Extract noun keywords for chunk indexing from a sentence."""
res = tagger.tags([sentence])
# nouns() returns only nominals (NNG, NNP, and so on).
return res.nouns()
text = "인공지능기반 자연어처리 기술은 검색 품질을 크게 높입니다."
print(chunk_keywords(text))
['인공지능', '기반', '자연어', '처리', '기술', '검색', '품질']
The compound noun 자연어처리 is decomposed into 자연어 and 처리, so it matches whichever one the user searches for.
Results and benefits
| Method | Chunk boundary | Search recall |
|---|---|---|
| Fixed-length splitting | By character count (ignores meaning) | Low |
| Sentence splitting only | By sentence end | Medium |
| Bareun noun-based | Noun density + compound noun decomposition | High |
Summary
If you extract nouns with Bareun, set chunk boundaries, and index with compound nouns decomposed, the search stage can find the relevant chunks more accurately and pass them to the LLM.
Frequently asked questions
Q. Do I need to convert the chunk body into morphemes too?
No. It is best to leave the body as the original text. An LLM answers better from natural sentences. Use the morphological analysis result only for search keywords and the index.
Q. What if compound nouns get split too finely?
For a term that should be treated as one word, like 자연어처리, register it in the compound noun dictionary of the custom dictionary
and it stays as a single noun. Only where decomposition is needed, mark the split point with ^ in the split compound noun dictionary.
Q. Can it handle long documents?
Yes. Split into sentences and pass them to tags(). If you need position information, use the
AnalyzeSyntaxList family, which preserves input order.
Related documents
- Morphological analysis API (AnalyzeSyntax): the analysis API used for chunk keyword extraction
- Tokenize vs AnalyzeSyntax: the difference between tokenization for indexing and morphological analysis
- Keyword extraction and text mining: applying the noun extraction workflow
- How compound noun decomposition works: how compound nouns are split into smaller nouns
Q. Do I really need a morphological analyzer for Korean RAG? Won't an LLM tokenizer do?
An LLM tokenizer splits text into frequency-based subwords, not by meaning, so the splits do not line up with the boundaries of Korean noun phrases and compound nouns. Bareun morphological analysis breaks an eojeol into the eight major morphological units and splits compound nouns, so you can align chunk boundaries and search keywords to meaning units. Using both together gives the best search quality.