Korean processing pipeline for academic papers

Academic papers are text dense with technical terms. A general morphological analyzer treats technical terms that were not in its training data as out-of-vocabulary words (OOV) or splits them incorrectly. Bareun combines a technical term dictionary (Onterm, a Korean terminology platform) with a custom dictionary to recognize a paper's technical terms as exactly one word.

The problem

• Technical terms such as 합성곱신경망 (convolutional neural network) and 유전자발현 (gene expression) get split into several nouns, which throws off search and counting.
• Field-specific abbreviations and neologisms fall through as OOV words.
• If terms are not captured accurately, the quality of paper indexing and similar-paper recommendation drops.

How to apply Bareun

Bareun has a built-in Onterm dictionary (onterm-dict.fb) that supplements the technical terms Urimalsaem (Open Korean Dictionary) does not cover. On top of that, register field-specific terms in the proper noun (np_set) and compound noun (cp_set) custom dictionaries to keep them as one word. For the procedure to turn field terms into a dictionary, see Domain dictionary building workflow.

graph TD
  PAPER[Paper text] --> TAG[Bareun morphological analysis];
  TAG --> ON[Onterm technical term dictionary];
  TAG --> CUST[Field custom dictionary];
  ON --> TERM[Technical terms recognized accurately];
  CUST --> TERM;
  TERM --> INDEX[Indexing, keywords, recommendation];

from bareunpy import Tagger

tagger = Tagger("koba-XXXX-...", "localhost")

# Register field technical terms in the custom dictionary (kept as compound nouns)
cust = tagger.custom_dict("paper_terms")
cust.copy_cp_set({"합성곱신경망", "유전자발현", "강화학습"})
cust.update()
tagger.set_domain("paper_terms")

res = tagger.tags(["합성곱신경망으로 유전자발현을 예측한다."])  # ("Predict gene expression with a convolutional neural network.")
print(res.nouns())

['합성곱신경망', '유전자발현']

The technical terms are kept as one word instead of being split.

Results and benefits

Frequently asked questions

Q. What if a term is not in the Onterm dictionary?

Register field-specific terms directly in the custom dictionary. Put terms to be treated as a single noun in the proper noun dictionary (np_set), and terms that combine several nouns in the compound noun dictionary (cp_set). For why OOV words arise and how to reduce them, see How out-of-vocabulary (OOV) words are handled.

Q. I want to split a term on purpose.

In the compound noun split dictionary (cp_caret_set), mark the split point with ^, and the term splits at the position you registered. This is useful when you index the split form and display the combined form. For details, see Registering compound noun splits.

Q. Can I tell which dictionary a term was recognized from?

Check the out_of_vocab value of the morpheme in the analysis result. It shows IN_CUSTOM_DICT for the custom dictionary, IN_ON_DICT for the Onterm dictionary, and IN_URIMALSAEM for Urimalsaem. (Some sources, such as IN_URIMALSAEM and IN_ON_DICT, are shown only in the spell checker build.)

Q. How do I solve the problem of technical terms falling through as OOV words?

Papers contain many technical terms that were not in the training data, so a general analyzer treats them as OOV words. Bareun has a built-in Onterm technical term dictionary to supplement terms Urimalsaem does not cover, and field-specific terms are registered in the custom dictionary to keep them as one word. That way, terms like 합성곱신경망 are not chopped up.

Related documents

• Domain dictionary building workflow: Collect field terms and turn them into a dictionary
• Onterm dictionary: How the built-in technical term dictionary works
• Out-of-vocabulary (OOV) handling: Why technical terms end up as OOV words and how to respond