Applying Bareun to Korean tokenization in Elasticsearch
Korean search quality in Elasticsearch depends on tokenization (the analyzer).
The built-in analyzer, nori, is fast, but it often misanalyzes neologisms, proper nouns, and compound nouns.
If you put Bareun morphological analysis in the indexing preprocessing stage, you can index with more accurate tokens and improve search accuracy.
The problem
- nori makes custom dictionary management awkward, and each time a new proper noun appears, you often have to restart the nodes.
- Program names such as
걸어서세계속으로get split in the wrong places and are not found by search. - Documents with messy word spacing (
아름다운강산) end up with shifted token boundaries.
A plugin makes this simpler
There is a plugin that plugs Bareun directly into Elasticsearch as an analyzer. You do not need to build a separate indexing pipeline, and the same analyzer runs at index time and at search time, so the token rules match automatically.
Use the approach below in environments where you cannot install the plugin (such as managed Elasticsearch), or when you want your application to decide the rules for producing tokens.
How to apply Bareun
In the indexing pipeline, analyze the document body with Bareun first to produce an array of morpheme tokens,
and then index it as is with a whitespace or keyword analyzer.
Using Bareun's custom dictionary and word spacing correction (auto_spacing) makes operations simpler than with nori.
graph TD DOC[Original document] --> BAREUN[Bareun morphological analysis]; BAREUN --> NORM[Base-form tokens of nouns and predicates]; NORM --> ES[Elasticsearch indexing]; Q[Search query] --> BAREUN2[Bareun analysis]; BAREUN2 --> ES;
If you produce index tokens and query tokens with the same Bareun analyzer, the token rules on both the index and search sides match, so matching is stable.
from bareunpy import Tagger
tagger = Tagger("koba-XXXX-...", "localhost")
def to_tokens(text: str):
"""Convert the body text into an array of index tokens (nouns + base forms of verbs and adjectives)."""
res = tagger.tags([text])
tokens = res.nouns() # Nominals
for morph, tag in res.pos():
if tag in ("VV", "VA"): # Also index base forms of verbs and adjectives
tokens.append(morph)
return tokens
print(to_tokens("걸어서세계속으로 다시 방송됩니다."))
['걸어서세계속으로', '방송', '걷', '되']
If you register 걸어서세계속으로 in the proper noun dictionary of the custom dictionary, it is kept as a single token.
Results and benefits
| Item | nori | With Bareun |
|---|---|---|
| Proper nouns and neologisms | Dictionary management is cumbersome | Custom dictionary updates in real time |
| Compound nouns | Often over-split | Kept intact if registered, or split as intended |
| Word spacing errors | Token boundaries shift | Tokenized after auto_spacing correction |
| Accuracy | Average | 99.6% POS tagging |
Operations tip
Bareun's custom dictionary detects file changes in real time and applies them, so changing the dictionary does not require a server restart. This suits search services that add new proper nouns often.
Frequently asked questions
Q. Do I have to replace nori completely?
It depends. A mixed setup is also possible, where only the key fields that need accuracy are preprocessed with Bareun before indexing, and the rest use nori.
Q. How do I keep index and search token rules consistent?
Call Bareun with the same custom dictionary and the same options at index time and at search time.
Tokenize the query with tagger.tags() too before searching.
Q. Is real-time indexing performance okay?
Bareun runs inference quickly on ONNX Runtime. For bulk indexing, you can raise throughput by grouping documents into batches and passing
several sentences at once to tags([...]).
Q. How do I integrate Bareun with Elasticsearch?
Place Bareun in the preprocessing stage of the indexing pipeline. Analyze the document body with Bareun to produce an array of morpheme tokens,
index it as is with a whitespace or keyword analyzer, and tokenize search queries with the same Bareun analyzer.
When you call it over REST, use the Connect path (POST /bareun.LanguageService/AnalyzeSyntax).
Related documents
- Tokenize API versus morphological analysis: helps with index token design
- Getting started with the custom dictionary: the flow for registering proper nouns and neologisms
- Real-time custom dictionary updates (hot reload): update the dictionary without a restart
- Calling over REST (HTTP+JSON): integrate indexing and search through the Connect path