LanguageService methods at a glance (Tokenize/AnalyzeSyntax/AnalyzeSyntaxList/AnalyzeSyntaxRaw)

All of Bareun's morphological analysis lives in a single LanguageService. This service provides four methods: Tokenize, AnalyzeSyntax, AnalyzeSyntaxList, and AnalyzeSyntaxRaw. The four methods use the same model, but they differ slightly in analysis depth, input form, and whether post-processing is applied.

This document summarizes the input, output, and purpose of the four methods in one table to help you quickly pick the right one for your situation.

Comparing the four methods

Method	Input	Core output	Sentence splitting	Post-processing	Typical use
`Tokenize`	Single document (`Document`)	Eojeol and token segmentation result	Optional	None	When you need only segmentation or tokenization
`AnalyzeSyntax`	Single document (`Document`)	Sentence → eojeol → morpheme (47 POS tags)	`auto_split_sentence`	Compound noun and verb decomposition, automatic word spacing and joining	Full morphological analysis
`AnalyzeSyntaxList`	Sentence array (`sentences[]`)	Per-sentence analysis in input order	No splitting	Compound noun and verb decomposition, automatic word spacing and joining	Batches, tracking order and position
`AnalyzeSyntaxRaw`	Single document (`Document`)	Pure model inference result	`auto_split_sentence`	None	Model performance evaluation, BigKinds re-tagging, WSD

Common request options

These are the options each method supports.

Option	`AnalyzeSyntax`	`AnalyzeSyntaxList`	`AnalyzeSyntaxRaw`	`Tokenize`
`encoding_type`	✔	✔	✔	✔
`auto_split_sentence`	✔	—	✔	—
`auto_spacing`	✔	✔	—	✔
`auto_jointing`	✔	✔	—	—
`custom_dict_names`	✔	✔	—	—

Custom dictionary and word spacing options in AnalyzeSyntaxRaw

To guarantee pure model output, AnalyzeSyntaxRaw does not support custom_dict_names, auto_spacing, or auto_jointing. Only auto_split_sentence works.

Call flow

graph TD
  D[Input text] --> C{What do you need?}
  C -->|Segmentation/tokens only| TK[Tokenize]
  C -->|Up to POS tags, single document| AS[AnalyzeSyntax]
  C -->|Sentence array, keep order| AL[AnalyzeSyntaxList]
  C -->|Pure model output, no post-processing| AR[AnalyzeSyntaxRaw]
  AS --> R[Sentence → Token → Morpheme]
  AL --> R
  AR --> R

Response structure

The responses of AnalyzeSyntax, AnalyzeSyntaxList, and AnalyzeSyntaxRaw have the same hierarchical structure. A sentence (Sentence) is an array of eojeol (Token), and an eojeol is an array of morphemes (Morpheme). Each piece is represented by a TextSpan (content, begin_offset, length) that carries position information.

service LanguageService {
  rpc AnalyzeSyntax(AnalyzeSyntaxRequest) returns (AnalyzeSyntaxResponse);
  rpc AnalyzeSyntaxList(AnalyzeSyntaxListRequest) returns (AnalyzeSyntaxListResponse);
  rpc AnalyzeSyntaxRaw(AnalyzeSyntaxRawRequest) returns (AnalyzeSyntaxResponse);
  rpc Tokenize(TokenizeRequest) returns (TokenizeResponse);
}

Morpheme holds the form (text), the tag (tag, 47 POS tags), the confidence (probability), the word source (out_of_vocab), and the name of the dictionary used (custom_dict_name).
Sentence.refined holds the result sentence with word spacing and similar issues corrected. (AnalyzeSyntaxRaw does no post-processing, so this value may be empty.)

The four methods use the same model

All four methods use the same models: segmentation (seg-model.onnx), segmentation-linked POS tagging (slot-tag-model.onnx), and POS tagging (tag-model.onnx). The difference is "how the input is received and how far the result is returned." So analysis quality is the same whichever method you switch to, but AnalyzeSyntaxRaw does no compound noun or compound verb decomposition, so it may produce fewer tokens than AnalyzeSyntax.

Frequently asked questions

Q. What is the difference between AnalyzeSyntax and AnalyzeSyntaxList?

AnalyzeSyntax takes a single document and can split it into sentences internally (auto_split_sentence). AnalyzeSyntaxList takes an array of sentences that are already split and keeps the input order as is. If tracking the order and position of input sentences matters, AnalyzeSyntaxList is the better fit.

Q. Can I get parts of speech from Tokenize alone?

Tokenize focuses on segmentation and tokenization. If you need the 47-POS morphological analysis result, use AnalyzeSyntax or AnalyzeSyntaxList.

Q. Can the custom dictionary be used in every method?

custom_dict_names can be specified only in AnalyzeSyntax and AnalyzeSyntaxList requests. AnalyzeSyntaxRaw aims for pure model output, so it does not support the custom dictionary.

Q. When do I use AnalyzeSyntaxRaw?

Use it when you need the model output exactly as it is, without compound noun or compound verb decomposition or automatic word spacing. Typical cases are model performance evaluation, re-tagging the large BigKinds corpus, and WSD (word sense disambiguation) pipelines.