Extracting and analyzing only Korean sentences in a multilingual environment

Logs, reviews, and customer inquiries of global services contain a mix of languages. If English, Chinese, or Japanese sentences go straight into a Korean analysis pipeline, the results get polluted. Bareun is a Korean-only (ko_KR) analyzer, and you can use the morpheme tags to filter out foreign-language tokens.

The problem

• You want to pick out and analyze only the Korean sentences in text with mixed languages.
• When English words are mixed into a sentence (code-switching), you need to tell the foreign-language parts apart.
• Running morphological analysis on non-Korean sentences gives meaningless results.

How to apply Bareun

Bareun classifies foreign-language text with the SL (foreign language) tag (for the meaning of each tag, see the POS tag table). After morphological analysis of a sentence, you can tell whether it is Korean or mixed with foreign text by looking at the share of SL and SH (Hanja). The only analysis language Bareun supports is ko_KR. If you just want to quickly split into morpheme units, also see Tokenization vs morphological analysis.

graph TD
  MIX[Multilingual text] --> SPLIT[Sentence splitting];
  SPLIT --> TAG[Bareun morphological analysis];
  TAG --> RATIO[Compute share of foreign SL];
  RATIO -->|Mostly Korean| KO[Korean analysis result];
  RATIO -->|Mostly foreign| SKIP[Exclude or send to another pipeline];

from bareunpy import Tagger

tagger = Tagger("koba-XXXX-...", "localhost")

def is_korean(text: str, threshold=0.5) -> bool:
    """Decide whether a sentence is Korean by the share of Korean morphemes."""
    res = tagger.tags([text])
    pos = res.pos()
    if not pos:
        return False
    # Ratio excluding SL (foreign language), SH (Hanja), SN (number), and symbols
    non_ko = sum(1 for _, t in pos if t in ("SL", "SH", "SN", "SW"))
    return (len(pos) - non_ko) / len(pos) >= threshold

print(is_korean("오늘 회의는 3시에 시작합니다."))   # "The meeting starts at 3 today."
print(is_korean("Please join the meeting at 3."))

True
False

You can pick out only the Korean sentences and pass them to later analysis.

Results and benefits

Frequently asked questions

Q. How are English words mixed into Korean handled?

English words such as API and OK are classified with the SL tag. If the whole sentence is Korean, you can collect only the SL tokens and use them as a list of English terms.

Q. Does Bareun also analyze English and Japanese?

No. Bareun's only analysis language is ko_KR. Send other languages to an analyzer dedicated to that language, and apply Bareun only to Korean sentences.

Q. What if I want to separate languages sentence by sentence?

Split into sentences first, then compute the share of foreign-language tokens in each sentence and keep only the Korean sentences. Adjust the threshold to fit your data. When processing a large number of sentences at once, see Analyzing several sentences at once.

Q. How do I build a pipeline that analyzes only Korean in multilingual logs and reviews?

Split the mixed-language text into sentences, pass each sentence to Bareun morphological analysis, and compute the share of non-Korean tokens such as SL (foreign language) and SH (Hanja). If the share is below the threshold, treat it as a Korean sentence and pass it on for later analysis; if it is high, exclude it or send it to an analyzer dedicated to that language. Bareun's only analysis language is ko_KR.

Related documents

• POS tag table: meanings of the SL (foreign language), SH (Hanja), and SN (number) tags
• Tokenization vs morphological analysis: deciding which API to use
• Analyzing several sentences at once: bulk processing of multilingual logs