Concepts

Bareun provides a custom dictionary feature. The morphological analysis engine handles sentences under a wide range of conditions on its own, but for certain words you sometimes need to decide from the outside how they should be processed. For example, when Bareun analyzes "네네치킨을 먹습니다." ("I eat Nene Chicken."), it splits the name reasonably well into "네네/NNG" and "치킨/NNG".

A company that makes or distributes 네네치킨 may want to see it as a single word.

For that, you need a way to register "네네치킨" as one compound noun.

Bareun calls this a custom dictionary and provides related APIs.

Five kinds of dictionaries

The custom dictionaries that Bareun provides fall into five kinds.

Proper noun dictionary : The proper noun dictionary holds, literally, nouns that name previously unknown people, places, events, products, and so on.

Compound noun dictionary : For compound nouns, Bareun already includes, by default, a dictionary of more than 1 million entries registered in Urimalsaem (Open Korean Dictionary) of the National Institute of Korean Language (NIKL). However, new compound nouns that do not appear there can emerge. "압수수색" (search and seizure) is such a case. Its abbreviation "압색" can be seen as another. In the end, a compound noun can represent a neologism that you want to express as a single common noun.

Compound noun split dictionary : The compound noun split dictionary is used for sequences of words that look like a single word but should be spaced according to NIKL rules. For example, even if "형태소분석" (morphological analysis) is written without a space, it is hard to regard it as an established single word. If it is better to express it split into "형태소" and "분석", write it as "형태소^분석". The same goes for "데이터^산업" (data industry).

Verb dictionary : The verb dictionary holds verbs that are made up of one or more words, as well as neologism verbs. For example, you may need to register a new verb such as 카톡하다 ("to KakaoTalk"). These are mostly cases where a word borrowed from elsewhere combines with a verb-deriving suffix to express a single verb. In most of these cases, the two combined words take on a meaning different from the original. 흘낏하다 is a verb derived from the adverb 흘낏. 흘낏거리다 and 흘낏대다 are likewise all registered in Urimalsaem as verbs. Bareun includes all such verbs in its default verb dictionary. In the morphological analysis training data, quite a few of these were tagged in some cases as "adverb + verb-deriving suffix". To tag them correctly, Bareun keeps a verb dictionary. Of course, users can also define and register new verbs.

Adjective dictionary : The adjective dictionary is used in the same way as the verb dictionary. Words registered in Urimalsaem, such as 소녀스럽다 ("girlish"), are already in the default adjective dictionary.

Domains

Internally, Bareun treats the five custom dictionaries above as a single bundle, and it can hold multiple bundles of custom dictionaries. For example, for a financial company, "banking (deposits and loans)", "credit card business", and "investment business" are very different in nature. Suppose this company uses Bareun and has built custom dictionaries. If it wants to register the three dictionaries separately and keep them permanently, would it have to set up three separate products?

Bareun lets you define an unlimited number of custom dictionaries on a single server. To tell the dictionaries apart and mark their areas, it uses the concept of a domain.

In short, Bareun lets you define a new dictionary for each domain.

Suppose about 100 custom dictionaries are defined on the server. If you make a call without any domain setting, you simply get the default result. If you specify the domain "은행" (bank) and make a call, "영업점" (branch) and "공인인증서" (certified digital certificate) may appear registered as common nouns. If you specify the domain "법률" (law), "민식이법" (the Minsik Law) may be identified as a proper noun.

Domain name restrictions

Domain names follow rules similar to variable names. You can use only letters (A-Z), digits, _, and -.

Dictionary item restrictions

• The symbols below cannot be added to dictionary words.
• Dictionary entries cannot contain spaces.
• Dictionary entries cannot be empty strings.
• Other symbols:

'\"()[]｀{}˝‘’“”′″〈〉《》｢「｣」『』【】〔〕<＜>＞≪≫!,.:;·˙‧∙，/／：；․…‥!.?．？

Try it out

Click Authorize to register your API key, and then use Try it out.

The five APIs

The custom dictionary service does not run as a separate service. It is built into Bareun. So you connect to it directly through the designated port, 5656. The custom dictionary service has five main functions.

• List all registered dictionaries
• Get the dictionary data registered for a specific domain
• Update or create the dictionary data of a specific domain
• Delete a specific domain or all custom dictionaries
• Check dictionary conflicts

Creating and updating a custom dictionary

Of these, we will focus on the third and most frequently used one, the UpdateCustomDictionary API.

This API takes an UpdateCustomDictionaryRequest message as the request and returns an UpdateCustomDictionaryResponse message as the response.

Create request: UpdateCustomDictionaryRequest

This message carries all of the dictionary data. If a dictionary with the same domain name already exists, it is replaced.

message UpdateCustomDictionaryRequest {
  string domain_name = 1;
  CustomDictionary dict = 2;
}

// Data transfer format of a custom dictionary
message CustomDictionary {
  string domain_name = 1;
  // Dictionary for proper nouns
  DictSet np_set = 2;
  // Dictionary for compound nouns
  DictSet cp_set = 3;
  // Dictionary for splitting compound nouns
  DictSet cp_caret_set = 4;
  // Verb dictionary
  DictSet vv_set = 5;
  // Adjective dictionary
  DictSet va_set = 6;
}

message DictSet {
  map<string, int32> items = 1;
  DictType type = 2;
  string name = 10;
}


// Dictionary types
enum DictType {
  // Format used for the default distributed dictionary.
  // Not used for transfer.
  TOKEN_INDEX = 0;
  // Used to represent proper nouns or compound nouns.
  WORD_LIST = 1;
  // # Used only for transfer; not actually used during inference.
  WORD_LIST_COMPOUND = 2;
}

The proper noun, compound noun, compound noun split, verb, and adjective dictionaries are all updated together in one request. There is no way to submit them individually. Providing separate individual APIs would only add complexity when you use them.

You only need to provide the domain name of the dictionary and the word list for each dictionary.

The most important thing is to build the DictSet. There are five DictSets in total. They correspond to the proper noun, compound noun, compound noun split, verb, and adjective dictionaries described above. Inside the message, they are named np_set, cp_set, cp_caret_set, vv_set, and va_set. A DictSet is an ordinary Set data structure. Google's protobuf does not support sets, so it is implemented as a map.

Use DictType for the dictionary type. Use WORD_LIST.

Create response: UpdateCustomDictionaryResponse

The response message is very simple, and you can only check whether the request succeeded or failed.

Failures are uncommon, and most are probably system errors. Note that when the system cannot bear the load and returns an error in a call over gRPC, it is delivered as an exception rather than as an update-failed message, so you are more likely not to receive a normal failure message.

API usage example

Once you understand these data structures, you can use the bareunpy Python client library to use a custom dictionary right away.

import json

from bareunpy import Tagger

# Connect to your own installed host.
# Enter below the API KEY ("koba-...") issued after email verification at "https://bareun.ai/". See "Log in - My info".
my_tagger = Tagger('YOUR-API-KEY', 'localhost', port=5656) # <- Replace with your own API KEY

# Save a custom dictionary
# The dictionary name may use only letters, digits, and the symbols '-' and '_'
# Symbols cannot be added to dictionary words (except the ^ symbol in the compound noun split dictionary).
cust_dic = my_tagger.custom_dict("my_dict_01") 
cust_dic.copy_np_set({'내고유명사', '걸어서세계속으로', '크리스토퍼놀란'})
cust_dic.copy_cp_set({'코로나19', '바나나우유'})
cust_dic.copy_cp_caret_set({'코로나^백신', '디지털^인문학'})
cust_dic.copy_vv_set({'카톡하다', '맑내하다'})
cust_dic.copy_va_set({'로맨틱하다', '신박하다', '판타스틱하다'})
cust_dic.update()

# Load a previously saved custom dictionary
cust_dict2 = my_tagger.custom_dict("my_dict_01")
cust_dict2.load()

text = '내고유명사로 존재하는 "크리스토퍼놀란"은 코로나19 상황 속에서도 걸어서세계속으로라는\
        TV 프로그램을 통해 바나나우유를 마시며 카톡하다가 \
        떠오른 신박한 아이디어로 판타스틱한 로맨틱 영화를 만들 계획이며, \
        그 과정에서 디지털인문학의 도움을 받아 코로나백신에 대한 사회적 이슈도 함께 다루려 한다.'

# Print only the POS information
my_tagger.set_domain('my_dict_01')
response = my_tagger.pos(text) 
print(f'Morphological analysis result: {response}')

# Store the morphological analysis result as a dictionary
response2 = my_tagger.tags([text]) 
res_dict = json.loads(response2.as_json_str())
idx = 1
for token in res_dict['sentences'][0]['tokens']:
    custom_token = [f"{t['text']['content']}/{t['tag']}, {t['outOfVocab']}" \
                    for t in token['morphemes'] if t['outOfVocab'] == 'IN_CUSTOM_DICT']
    if len(custom_token) != 0:
        print(f"{idx}: {custom_token}")
        idx += 1

Morphological analysis result: [('내고유명사', 'NNP'), ('로', 'JKB'), ('존재하', 'VV'), ('는', 'ETM'), ('"', 'SS'), ('크리스토퍼놀란', 'NNP'), ('"', 'SS'), ('은', 'JX'), ('코로나19', 'NNG'), ('상황', 'NNG'), ('속', 'NNG'), ('에서', 'JKB'), ('도', 'JX'), ('걸어서세계속으로', 'NNP'), ('라는', 'ETM'), ('TV', 'SL'), ('프로그램', 'NNG'), ('을', 'JKO'), ('통하', 'VV'), ('아', 'EC'), ('바나나우유', 'NNG'), ('를', 'JKO'), ('마시', 'VV'), ('며', 'EC'), ('카톡하', 'VV'), ('다가', 'EC'), ('떠오르', 'VV'), ('ㄴ', 'ETM'), ('신박하', 'VA'), ('ㄴ', 'ETM'), ('아이디어', 'NNG'), ('로', 'JKB'), ('판타스틱하', 'VA'), ('ㄴ', 'ETM'), ('로맨틱', 'NNG'), ('영화', 'NNG'), ('를', 'JKO'), ('만들', 'VV'), ('ㄹ', 'ETM'), ('계획', 'NNG'), ('이', 'VCP'), ('며', 'EC'), (',', 'SP'), ('그', 'MMD'), ('과정', 'NNG'), ('에서', 'JKB'), ('디지털', 'NNG'), ('인문학', 'NNG'), ('의', 'JKG'), ('도움', 'NNG'), ('을', 'JKO'), ('받', 'VV'), ('아', 'EC'), ('코로나', 'NNP'), ('백신', 'NNG'), ('에', 'JKB'), ('대하', 'VV'), ('ㄴ', 'ETM'), ('사회', 'NNG'), ('적', 'XSN'), ('이슈', 'NNG'), ('도', 'JX'), ('함께', 'MAG'), ('다루', 'VV'), ('려', 'EC'), ('하', 'VX'), ('ㄴ다', 'EF'), ('.', 'SF')]

1: ['내고유명사/NNP, IN_CUSTOM_DICT']
2: ['크리스토퍼놀란/NNP, IN_CUSTOM_DICT']
3: ['코로나19/NNG, IN_CUSTOM_DICT']
4: ['걸어서세계속으로/NNP, IN_CUSTOM_DICT']
5: ['바나나우유/NNG, IN_CUSTOM_DICT']
6: ['카톡하/VV, IN_CUSTOM_DICT']
7: ['신박하/VA, IN_CUSTOM_DICT']
8: ['판타스틱하/VA, IN_CUSTOM_DICT']
9: ['디지털/NNG, IN_CUSTOM_DICT', '인문학/NNG, IN_CUSTOM_DICT']