The difference between Tokenize and AnalyzeSyntax, and when to use each
Bareun's LanguageService has both Tokenize, which focuses on segmentation,
and AnalyzeSyntax, which also performs part-of-speech tagging.
Both methods take the same input document, but they differ in "how far they analyze before returning."
This document summarizes the difference between the two methods and when to use each.
What is different
Bareun's analysis flows through the stages segmentation (Segmenter) → POS tagging (PosTagger) → post-processing.
Tokenize focuses on splitting words only, not on morphological analysis, so instead of the 47-POS system it marks
tokens with the 8-component system (N substantive / V predicate / A adverb / M determiner / J particle / E ending / S symbol / I interjection).
AnalyzeSyntax, on the other hand, returns everything: 47-POS tagging and morpheme-level results.
graph LR IN[Input sentence] --> SEG[Segmentation: Segmenter] SEG --> TK[Tokenize: eojeol/token] SEG --> TAG[POS tagging: PosTagger] TAG --> AS[AnalyzeSyntax: morphemes + 47 POS]
| Item | Tokenize | AnalyzeSyntax |
|---|---|---|
| Core result | Eojeol and token segmentation (word splitting) | Sentence → eojeol → morpheme (47 POS) |
| Classification system | 8 components (N/V/A/M/J/E/S/I) | 47 POS |
| POS tag | Not the focus | A tag is assigned to each morpheme |
Confidence (probability) | — | Provided per morpheme |
Word source (out_of_vocab) | — | Provided |
| Processing cost | Relatively light | Performs tagging as well |
When to use which
- When Tokenize fits: Work where "you don't need POS and just need to split finely," such as tokenization for a search index, checking eojeol boundaries, and quick preprocessing.
- When AnalyzeSyntax fits: Most serious NLP work, such as extracting only nouns, analyzing verbs and adjectives, POS-based filtering, and analysis that needs the position and confidence of each morpheme.
from bareunpy import Tagger
tagger = Tagger("koba-XXXX-...", host="api.bareun.ai", port=443)
res = tagger.tags(["오늘 날씨가 참 좋네요."])
# Pick out only nouns (NNG/NNP).
for sent in res.msg().sentences:
for token in sent.tokens:
for m in token.morphemes:
if m.tag in ("NNG", "NNP"):
print(m.text.content, m.tag, m.probability)
from bareunpy import Tagger
tagger = Tagger("koba-XXXX-...", host="api.bareun.ai", port=443)
# Get the segmentation result in eojeol/token units.
res = tagger.tokenize_list(["오늘 날씨가 참 좋네요."])
print(res)
Both use the same segmentation model
Tokenize and AnalyzeSyntax share the same segmentation model (seg-model.onnx).
AnalyzeSyntax runs the POS tagging model (tag-model.onnx) one step further after that.
So the segmentation result is consistent, and you only need AnalyzeSyntax when you need POS.
Frequently asked questions
Q. If I don't need POS, is Tokenize faster?
It does not go through the POS tagging stage, so it is generally lighter. For work that does not need
POS information, such as search indexing and preprocessing, Tokenize fits.
Q. Which method do I use to extract only nouns?
Use AnalyzeSyntax and pick out only the morphemes whose tag is NNG (common noun) or NNP (proper noun).
POS tags are provided only by AnalyzeSyntax.
Q. Can the segmentation results of the two methods differ?
Segmentation uses the same model, so it is consistent. The difference is whether POS tagging is performed afterward.
Q. Does Tokenize return the 47 POS as is?
No. Tokenize splits words only, rather than doing morphological analysis, so it classifies tokens with the
8-component system (N substantive / V predicate / A adverb / M determiner / J particle / E ending / S symbol / I interjection)
instead of the 47 POS. If you need 47-POS morpheme tags, use AnalyzeSyntax.