Design philosophy of the 8 major form units (N/J/V/E/A/M/I/S)

Before it attaches any of the 47 detailed POS tags, Bareun first groups Korean morphemes into 8 major form units. They are substantive (N), particle (J), predicate (V), ending (E), adverbial (A), determiner (M), interjection (I), and symbol (S). These 8 units are the backbone of the segmentation stage, and a design meant to tackle the Korean tokenizer problem head on.

The 8 major form units

Symbol	Major unit	Korean category name	Examples of the 47 POS it includes
N	Substantive (chewon, 체언)	Nouns, pronouns, numerals	NNG common noun, NNP proper noun, NNB dependent noun, NP pronoun, NR numeral
J	Particle (josa, 조사)	Case particles, auxiliary particles, conjunctive particles	JKS subject, JKO object, JKB adverbial, JX auxiliary, JC conjunctive
V	Predicate (yongeon, 용언)	Verbs, adjectives, copulas	VV verb, VA adjective, VX auxiliary predicate, VCP positive copula, VCN negative copula
E	Ending (eomi, 어미)	Pre-final, final, connective, transformative endings	EP pre-final, EF final, EC connective, ETN nominalizing, ETM adnominalizing
A	Adverbial (buesa-eo, 부사어)	Adverbs	MAG general adverb, MAJ conjunctive adverb
M	Determiner (gwanhyeong-eo, 관형어)	Determiners	MMA descriptive determiner, MMD demonstrative determiner, MMN numeral determiner
I	Interjection (gamtansa, 감탄사)	Independent words	IC interjection
S	Symbol (giho, 기호)	Punctuation, foreign words, numbers	SF period, SP comma, SL foreign word, SH Chinese character, SN number

From major units to detailed POS

Analysis follows a flow that first decides the major unit and then narrows down to a detailed POS within it.

graph TD
  EOJ[학교에서] --> SEG[Segmentation];
  SEG --> N[학교 → N substantive];
  SEG --> J[에서 → J particle];
  N --> NNG[NNG common noun];
  J --> JKB[JKB adverbial particle];

In the example above, 학교에서 ("at school") first splits into the substantive (N) 학교 and the particle (J) 에서. Then the substantive is refined to the common noun NNG, and the particle to the adverbial particle JKB. The final result is 학교/NNG + 에서/JKB.

Why group the 8 units first

Agglutinative language and the tokenizer

In transformer models, the Korean tokenizer is always a thorny problem. Korean is an agglutinative language, so many morphemes attach within one eojeol (a space-delimited word unit), and inflection, contraction, and omission happen constantly. It is routine for one eojeol such as 먹었습니다 ("ate") to break down into 먹/VV + 었/EP + 습니다/EF.

Instead of getting all 47 POS right at once, Bareun first sets the broad framework of substantive, predicate, particle, and ending, which is the natural classification in Korean grammar. This lets the segmentation model learn simpler, clearer signals such as "here a substantive ends and a particle begins". The 47 detailed POS are decided on top of that, so the whole analysis is stable.

Strength of the broad framework

The 8 units also line up naturally with the 9-part-of-speech system of Korean school grammar, so people can understand the analysis results intuitively. Another key to its robustness is that if the major unit is right, a wrong detailed POS does little harm to the interpretation of meaning.

Tokenize uses 8 components, not 47 POS

Unlike morphological analysis (AnalyzeSyntax), which attaches one of 47 detailed POS to each morpheme, Tokenize, which only separates words, uses the 8-component system, not the 47-POS system. It does not go down to detailed POS. It groups only by the 8 major form units above and cuts out words.

Symbol	8 component	Symbol	8 component
N	Substantive	J	Particle
V	Predicate	E	Ending
A	Adverb	S	Symbol
M	Determiner	I	Interjection

For uses such as search indexing and preprocessing, where you do not need detailed POS and only need to know "where words break", the 8-component system, which is lighter than the 47 POS, is more practical.

Frequently asked questions

Q. What are the 8 major form units?

They are substantive (N), particle (J), predicate (V), ending (E), adverbial (A), determiner (M), interjection (I), and symbol (S). They group the 47 detailed POS into these 8 broad branches.

Q. How do the major units relate to the 47 POS?

The major units are the parent classification of the detailed POS. For example, the substantive (N) contains common noun (NNG), proper noun (NNP), dependent noun (NNB), pronoun (NP), and numeral (NR).

Q. Why divide by major unit first?

Korean is agglutinative, so many morphemes attach in one eojeol. Setting the broad framework of substantive, predicate, particle, and ending first makes segmentation boundaries clear, and the detailed POS are decided on top of it, so the analysis is stable.

Q. Does Tokenize use the 47 POS?

No. Tokenize, which only separates words, uses the 8-component system (N substantive, V predicate, A adverb, M determiner, J particle, E ending, S symbol, I interjection), not the 47-POS system. It is lighter and more practical for uses such as search indexing and preprocessing, where detailed POS are not needed.