Keyword extraction and text mining workflow

To quickly grasp "what is this text about" from a pile of documents, you need keyword extraction. Most keywords in Korean are nouns, so extracting nouns accurately with Bareun and counting their frequency is the most robust starting point for text mining.

The problem

• If you split words by whitespace only, particles (josa) stick to words, as in 데이터를 and 데이터가, and the same word is counted separately.
• Proper nouns (person and organization names) cannot be told from common nouns, so keywords get lumped together.
• Compound nouns are split or joined against your intent, which skews frequencies.

How to apply Bareun

Run morphological analysis on the documents, extract only the substantives (nominals), and count proper nouns (NNP) and common nouns (NNG) separately. Particles and endings are removed, so the same word gathers under a single key.

graph TD
  DOCS[Document collection] --> TAG[Bareun morphological analysis];
  TAG --> POS[Extract by POS];
  POS --> NNG[Common noun NNG];
  POS --> NNP[Proper noun NNP];
  NNG --> CNT[Count frequency];
  NNP --> CNT;
  CNT --> TOP[Top keywords];

from collections import Counter
from bareunpy import Tagger

tagger = Tagger("koba-XXXX-...", "localhost")

def extract_keywords(docs, top_n=5):
    """Extract the top keywords by noun frequency from a collection of documents."""
    counter = Counter()
    for doc in docs:
        res = tagger.tags([doc])
        # Count only common nouns and proper nouns as keywords
        for morph, tag in res.pos():
            if tag in ("NNG", "NNP"):
                counter[morph] += 1
    return counter.most_common(top_n)

docs = [
    "바른은 한국어 형태소 분석기입니다.",
    "형태소 분석으로 키워드를 추출합니다.",
    "한국어 자연어처리에 형태소 분석이 쓰입니다.",
]
print(extract_keywords(docs))

[('형태소', 3), ('분석', 3), ('한국어', 2), ('키워드', 1), ('바른', 1)]

형태소 (morpheme) and 분석 (analysis) rise to the top as the main keywords, and the variants with particles attached are merged into one.

Results and benefits

Frequently asked questions

Q. How do I handle stopwords?

Among nouns there are dependent nouns (NNB) with weak meaning, such as 것 (thing), 수 (way, ability), and 때 (time). Filter such words with a separate stopword list, or exclude dependent nouns (NNB) from the count.

Q. I want to extract proper nouns only.

If you collect only the morphemes whose tag is NNP from the pos() result, you get only proper noun keywords such as person, place, and organization names.

Q. How do I weight keywords?

Besides plain frequency, applying TF-IDF brings out the keywords specific to each document. Here too, using Bareun nouns as the token unit makes the weight calculation stable.

Related documents

• Morphological analysis API (AnalyzeSyntax): the analysis API used for noun extraction
• POS tag table: meanings of tags such as NNG, NNP, and NNB
• News keyword extraction case: an application centered on proper nouns
• Korean chunking for RAG: using noun extraction for a search index

Q. Is a morphological analyzer really necessary for Korean text mining?

Yes. In Korean a particle attaches to a noun to form one eojeol (a space-delimited word unit), so if you split by whitespace only, 데이터를 and 데이터가 are counted as different words. If you extract only the substantives with Bareun morphological analysis, particles and endings are removed and the same word gathers into one, and by distinguishing proper nouns (NNP) from common nouns (NNG) you get more accurate keywords and frequency statistics.