Sentiment analysis preprocessing: eojeol units versus morpheme units

The accuracy of sentiment analysis depends heavily on the unit in which you represent the text. If you treat Korean in eojeol (space-delimited word unit) form, the same sentiment word scatters into dozens of forms because of particles and endings. If you build morpheme-level features with Bareun, the key expressions that carry sentiment gather in one place, which makes classification easier.

The problem

• "좋아요", "좋은데", "좋습니다", and "좋았어" all express the sentiment of 좋다 (to be good), but as eojeols they are completely different tokens.
• Negation ("안 좋다", "좋지 않다") is hard to capture at the eojeol level.
• When you have little data, eojeol features are sparse, which makes training difficult.

How to apply Bareun

Break each sentence into morphemes and use adjective and verb base forms and negation morphemes as features. Even when a predicate is inflected, it is unified to its base form (좋) and becomes the same feature.

graph TD
  S[Review sentence] --> CMP{Preprocessing unit};
  CMP -->|Eojeol| W[좋아요 / 좋은데 / 좋습니다 ...];
  CMP -->|Morpheme| M[좋 + ending];
  W --> SPARSE[Sparse features];
  M --> DENSE[Unified sentiment feature];
  DENSE --> CLF[Sentiment classification];

from bareunpy import Tagger

tagger = Tagger("koba-XXXX-...", "localhost")

def sentiment_features(text: str):
    """Build sentiment analysis features (adjective and verb base forms + negation)."""
    res = tagger.tags([text])
    feats = []
    for morph, tag in res.pos():
        if tag in ("VA", "VV"):       # Adjective and verb base forms
            feats.append(morph)
        if tag == "VCN":              # Negative copula (아니다)
            feats.append("NEG")
        if morph == "안" and tag == "MAG":  # Negation adverb
            feats.append("NEG")
    return feats

for s in ["정말 좋아요", "좋은데 비싸요", "별로 안 좋다"]:
    print(s, "->", sentiment_features(s))

정말 좋아요 -> ['좋']
좋은데 비싸요 -> ['좋', '비싸']
별로 안 좋다 -> ['NEG', '좋']

All three sentences share the unified feature 좋, and negation is made explicit as NEG.

Eojeol units versus morpheme units

Frequently asked questions

Q. Should I use nouns as sentiment features too?

In some cases a noun itself carries sentiment, as with 최악 (the worst) and 명작 (masterpiece). Including sentiment nouns in the features along with predicate base forms improves performance.

Q. How do I capture negation more accurately?

If you recognize negation morphemes such as 안 (negation adverb), 못, 없 (adjective), and 아니 (negative copula) and group a NEG token with the nearby predicate, you can tell "안 좋다" and "좋다" apart as opposite sentiments.

Q. What about emoji and expressions like ㅋㅋ?

These expressions are not targets of morphological analysis, so handle them with separate rules. Using morpheme features together with emoji and repeated-jamo features works well for social media reviews.

Related documents

• POS tag table: predicate and negation tags such as VA, VV, and VCN
• Morphological analysis API (AnalyzeSyntax): the analysis API that returns predicate base forms
• Chatbot intent classification preprocessing: the same morpheme feature normalization applied elsewhere
• Keyword extraction and text mining: useful for counting sentiment nouns

Q. Which morphological analyzer is good for review sentiment analysis preprocessing?

Sentiment is mainly carried by predicates and they take many inflected forms, so an analyzer with accurate base-form restoration is best. Bareun, with 99.6% POS tagging accuracy, unifies both 좋아요 and 좋습니다 to 좋 and also captures negation morphemes accurately. It lets you build sentiment features that are robust to inflection, which suits preprocessing for review and social media sentiment analysis.