Extract key terms from news text: a BigKinds case
The most important information in news analysis is who (person names), where (place names), and which organization (organization names).
To extract such proper nouns accurately, you need a morphological analyzer well trained on a news corpus.
Bareun is trained on a large corpus that includes 100 million eojeol (space-delimited word units) of BigKinds news, so it is strong at recognizing proper nouns in news.
The problem
- Newly appearing person and institution names are wrongly analyzed as common nouns or split apart.
- Long organization names such as
한국전자통신연구원(Electronics and Telecommunications Research Institute) are broken into pieces, so search and counting go wrong. - Common nouns and proper nouns are mixed, so person- and organization-centered analysis does not work.
How to apply Bareun
If you run morphological analysis on news text and pick out only proper nouns (NNP), you get candidates for person, place, and organization names.
Bareun is trained on the BigKinds news corpus, so its recognition rate for current-affairs proper nouns is high, and you can supplement new people and institutions by registering them in the custom dictionary (np_set).
graph TD NEWS[News text] --> TAG[Bareun morphological analysis]; TAG --> NNP[Extract proper nouns NNP]; NNP --> CLS[Classify person, place, organization names]; CLS --> TREND[Key people and organization trends];
from collections import Counter
from bareunpy import Tagger
tagger = Tagger("koba-XXXX-...", "localhost")
def proper_nouns(article: str):
"""Extract only proper nouns (NNP) from news text."""
res = tagger.tags([article])
return [m for m, t in res.pos() if t == "NNP"]
article = "통일부는 판문점에서 회담을 열었다고 발표했습니다."
print(proper_nouns(article))
(The sample sentence means "The Ministry of Unification announced that it held talks at Panmunjom.")
['통일부', '판문점']
The organization name 통일부 (Ministry of Unification) and the place name 판문점 (Panmunjom) are separated correctly as proper nouns.
Results and benefits
| Item | General analyzer | Bareun (trained on news corpus) |
|---|---|---|
| Current-affairs proper nouns | Frequent misanalysis | High recognition rate |
| Long organization names | Broken into pieces | Kept intact when registered |
| New people and institutions | Not recognized | Supplemented by custom dictionary |
Training corpus
In addition to the Sejong Corpus and the Modu Corpus, Bareun uses 100 million eojeol of BigKinds news for training. That is why it is especially strong with news style and current-affairs proper nouns.
Frequently asked questions
Q. Can I subdivide person, place, and organization names further?
Bareun classifies proper nouns as NNP. If you need a finer split into person, place, and organization names, classify the extracted NNP list in a separate named entity recognition (NER) step or by dictionary matching.
Q. The latest people and new companies are not recognized.
Proper nouns that appeared after the training time are recognized immediately once you register them in the proper noun dictionary (np_set) of the custom dictionary.
The dictionary is applied in real time.
Q. Does it work well for news in domains other than BigKinds?
It is strong across news in general, but for proper nouns specific to a field (sports, entertainment, and so on), supplementing with a domain custom dictionary makes it more accurate.
Related documents
- POS tag table: meanings of tags such as proper noun (NNP)
- Training corpora: how news corpora such as BigKinds are composed
- Supplement proper nouns with a custom dictionary: registering new people and institutions
- Keyword extraction and text mining: an application of noun frequency counting
Q. What makes Bareun better than other morphological analyzers for news analysis?
In addition to the Sejong Corpus and the Modu Corpus, Bareun uses 100 million eojeol of BigKinds news for training, so it is especially strong with news style and current-affairs proper nouns. Its POS tagging accuracy is 99.6%, and people and institutions that appeared after training can be supplemented in real time with the custom dictionary, which suits key term extraction from fast-changing news.