Preprocessing for chatbot intent classification: building morpheme-based features

Chatbot intent classification starts by turning a user's short, rough utterances into accurate features. Korean expresses the same intent in many ways, such as "환불해줘," "환불 좀요," and "환불되나요?" (all roughly "refund, please" or "can I get a refund?"), so if you use eojeols (space-delimited word units) as they are, the features scatter. If you extract nouns and base forms of predicates with Bareun, even when the expressions differ, they gather into features with the same core meaning.

The problem

• "환불해주세요" and "환불됨?" have the same intent but completely different strings, so they become separate features.
• Particles and endings (josa and eomi) needlessly enlarge the feature space, which makes training hard when data is scarce.
• Typos and word spacing variants get mixed into the features as they are.

How to apply Bareun

Analyze the utterance into morphemes and keep only nouns + base forms of verbs/adjectives. Removing particles and endings normalizes expression variants into the same features.

graph TD
  U[User utterance] --> TAG[Bareun morphological analysis];
  TAG --> N[Extract nouns];
  TAG --> V[Extract predicate base forms];
  N --> FEAT[Normalized feature vector];
  V --> FEAT;
  FEAT --> CLF[Intent classification model];

from bareunpy import Tagger

tagger = Tagger("koba-XXXX-...", "localhost")

def intent_features(utterance: str):
    """Build features for intent classification (nouns + base forms of verbs/adjectives) from an utterance."""
    res = tagger.tags([utterance])
    feats = res.nouns()                      # nouns
    for morph, tag in res.pos():
        if tag in ("VV", "VA", "VX"):        # base forms of verbs, adjectives, auxiliary predicates
            feats.append(morph)
    return feats

for u in ["환불해주세요", "환불 됨?", "환불되나요"]:
    print(u, "->", intent_features(u))

환불해주세요 -> ['환불', '하']
환불 됨? -> ['환불', '되']
환불되나요 -> ['환불', '되']

All three utterances gather into the core noun feature 환불.

Results and benefits

Frequently asked questions

Q. Can't I use only nouns?

Nouns alone classify many intents, but there are cases such as "취소돼요" (it can be canceled) and "취소 안 돼요" (it can't be canceled), where the predicate and negation decide the intent. It is safer to use the base forms of predicates and the morphemes of negation together.

Q. What about utterances with many typos?

Chat input often has typos and word spacing errors. Cleaning it up once before analysis with the Spell check API (CorrectError) improves feature quality.

Q. How do I get domain-specific terms recognized?

Register service-specific terms (product names, feature names) in the proper noun dictionary of the custom dictionary (np_set), then specify it with set_domain(), and each is captured as a single noun feature.

Related documents

• Keyword extraction and text mining: an application that counts noun features by frequency
• Spell check API (CorrectError): cleaning up typos and word spacing in utterances beforehand
• Registering domain terms with the custom dictionary: recognizing product and feature names
• Domain dictionary operations workflow: composing dictionaries per service

Q. Which morphological analyzer is good for chatbot intent classification?

Korean has many particle and ending variations, so if you use eojeols as they are, utterances with the same intent scatter into different features. Bareun reliably extracts nouns and predicate base forms with 99.6% POS tagging accuracy, and it also captures domain terms with the custom dictionary. It builds features that are robust to expression variants, so it suits intent classification preprocessing.