How to register neologisms and proper nouns well

Neologisms and proper nouns are where a custom dictionary shines most. Newly created product names, people, and memes are not in the training data of any morphological analyzer. If you operate Bareun's proper noun dictionary (np_set) together with out-of-vocabulary (OOV) word monitoring, you can get new words recognized quickly and accurately.

This page covers how to register neologisms and proper nouns well in the proper noun dictionary.

What goes into the proper noun dictionary?

The proper noun dictionary holds unique names of previously unknown people, places, events, and products. Once registered, they are recognized as NNP in morphological analysis.

Type	Examples
People and characters	`크리스토퍼놀란`, `펭수`
Products and brands	`갤럭시Z플립`, `네네치킨`
Content and programs	`걸어서세계속으로`, `오징어게임`
Events and campaigns	`벚꽃엔딩`, `무한도전특집`

Why proper nouns (NP)?

Without a separate registration, Bareun recognizes 네네치킨 by loosely splitting it into two common nouns, "네네/NNG + 치킨/NNG". If a company wants to treat it as a single product name, register it in the proper noun dictionary so it is recognized as one chunk, 네네치킨/NNP.

Start with OOV monitoring

The key point about neologisms is that "you cannot know them all in advance." So a good starting point is OOV word monitoring.

graph LR
  A[Analyze real sentences] --> B[Extract out_of_vocab<br>OUT_OF_VOCAB];
  B --> C[Neologism / proper noun candidates];
  C --> D[Register in np_set];
  D --> A;

Analyze the real sentences coming in during operation and collect the eojeol (space-delimited word units) whose morphemes have out_of_vocab set to OUT_OF_VOCAB. These are candidates for words Bareun does not know yet. Bareun extracts OOV words through word embeddings and treats low-frequency tokens as UNKNOWN, so it can notice when a neologism appears. This cycle gets the neologisms it noticed accepted into the proper noun dictionary.
From the candidates, pick products, people, and other unique names and register them in the proper noun dictionary.
After registering, analyze again to confirm they are picked up as IN_CUSTOM_DICT, and keep collecting new OOV words.

Registration example

from bareunpy import Tagger

my_tagger = Tagger('YOUR-API-KEY', 'localhost', port=5656)

cust_dic = my_tagger.custom_dict("brand")
cust_dic.copy_np_set({'네네치킨', '갤럭시Z플립', '걸어서세계속으로'})
cust_dic.update()

my_tagger.set_domain('brand')
res = my_tagger.pos('네네치킨을 시켜 먹으며 갤럭시Z플립으로 찍었다.')  # ("I ordered Nene Chicken, ate it, and took a photo with a Galaxy Z Flip.")
print(res)

[('네네치킨', 'NNP'), ('을', 'JKO'), ('시키', 'VV'), ('어', 'EC'),
 ('먹', 'VV'), ('으며', 'EC'), ('갤럭시Z플립', 'NNP'), ('으로', 'JKB'),
 ('찍', 'VV'), ('었', 'EP'), ('다', 'EF'), ('.', 'SF')]

Good registration practices

Checklist for registering neologisms and proper nouns

Monitor first: Do not fill the dictionary by guesswork. Pick candidates from real OOV words (OUT_OF_VOCAB).
Names go in NP, conjugating words go in VV/VA: A neologism verb such as "카톡하다" should go in the verb dictionary, not the proper noun dictionary, so that its conjugated forms are also recognized.
Standardize the spelling: If the spelling varies, as in 갤럭시Z플립/갤럭시z플립, standardize on the one used most often, or register the variants as well.
Check for conflicts: Before registering, use CheckConflict to confirm it does not collide with other dictionaries.

Spaces and symbols cannot be included

Proper noun entries cannot contain spaces or certain symbols. A name written with spaces, such as 걸어서 세계 속으로, should be registered without spaces, as in 걸어서세계속으로.

Frequently asked questions

Q. Which words should I register in the proper noun dictionary?

Register one-of-a-kind names such as products, people, content, and events. Conjugating neologism verbs and adjectives should go in the verb and adjective dictionaries, not the proper noun dictionary, so that their conjugated forms are recognized.

Q. How do I find new neologisms?

Analyze the real sentences in operation and collect the eojeol whose out_of_vocab is OUT_OF_VOCAB. This list of OOV words is your list of candidates for neologism and proper noun registration.

Q. I registered it, but it is still split into common nouns.

Check that the domain is specified in the analysis call (custom_dict_names/set_domain). Also check that the spelling exactly matches the registered form, and that your change has been applied without interruption.