Libraries
To use Bareun from your development environment, use one of its client libraries. Four languages are supported, and each library follows the standard package distribution channel of its language.
| Language | Package | Latest version | Repository |
|---|---|---|---|
| Python | PIP bareunpy | 2.1.0 | bareunpy |
| Java | Maven ai.bareun:bareun-client | 2.0.0 | bareun-java |
| JavaScript / TypeScript | npm bareun | 2.0.0 | bareun-js |
| R | RBareun | 1.3.0 | RBareun |
There is also a plugin that plugs the morphological analyzer directly into Elasticsearch. See the Elasticsearch plugin page.
All of these are published on GitHub.
Java 2.0.0 and JavaScript 2.0.0 are being prepared for release
Both libraries are already in their source repositories but have not yet been published to Maven Central or npm. Until then, clone the repository and build it yourself. Build instructions are in each repository's README.
The previously released versions (Maven ai.bareun.tagger:bareun 1.4.3, npm bareun 1.1.0)
still work, but they do not support spell correction, homograph sense disambiguation,
or Urimalsaem phoneme search.
Installation for each language:
pip install bareunpy
<dependency>
<groupId>ai.bareun</groupId>
<artifactId>bareun-client</artifactId>
<version>2.0.0</version>
</dependency>
Java 17 or later is required.
npm install bareun
Works on Node 18 or later and in the browser. Type definitions are included.
install.packages(c("curl", "httr", "jsonlite"))
library(devtools)
devtools::install_github("bareun-nlp/RBareun")
What can I use in each language?
| Feature | Python | Java | JavaScript | R |
|---|---|---|---|---|
| Morphological analysis | O | O | O | O |
| Eojeol segmentation (Tokenize) | O | O | O | - |
| Custom dictionary | O | O | O | O |
| Spell correction | O | O | O | O |
| Homograph sense disambiguation | O | O | O | O |
| Urimalsaem phoneme search | O | O | O | O |
| Streaming correction | O | - | - | - |
Spell correction and Urimalsaem phoneme search work only on servers that include the spell correction feature. A request to a morphological-analysis-only server fails with a "feature not provided" error.
How do the libraries work?
To use a library, you must be connected to a TCP/IP network. Every library acts as a client and must be able to reach the server. Bareun connects to the server over TCP port 5656. If the server is installed on your own PC, this is not an issue.
The Bareun libraries talk to the server using the Connect protocol. The server accepts Connect, gRPC, and HTTP+JSON on a single port, so no separate proxy is needed. The interface is defined in protobuf.
You need an API key to use a library. The API key is sent in the api-key header.
- An environment with TCP/IP or internet access
- A running server with your API key registered
- An API key, used whenever you call the library
In short, to use a library you connect to a running Bareun server with your API key. When the library sends a request, the server checks the API key and, if it is valid, returns the response. The library provides a variety of functions for working with the response.
sequenceDiagram
Bareun Library ->>+ Bareun Service: Send request
Bareun Service ->>+ Bareun Interceptor: Check request
alt Authorization error
Note right of Bareun Interceptor: Check API key
Bareun Interceptor --x Bareun Library: ERROR
else Authorized
Bareun Interceptor ->>- Bareun Service: User info
Bareun Service ->>- Bareun Library: Return response
end
Bareun Library -->+ Bareun Library: Print or use the response
Sample code in each language
The libraries are designed to be used in a similar way across languages. Below is a brief look at sample code for each language.
Morphological analysis
For morphological analysis, you simply pass in a sentence. Besides the sentence itself, three options are available.
auto_split_sentence: Splits sentences automatically. The default isfalse.auto_spacing: Fixes word spacing automatically. The default istrue.auto_jointing: Joins words that should be written together automatically. The default isfalse.
import sys
import google.protobuf.text_format as tf
from bareunpy import Tagger
#
# you can API-KEY from https://bareun.ai/
# Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
API_KEY="koba-ABCDEFG-1234567-LMNOPQR-7654321" # <- replace with your own API KEY
# If you have your own localhost bareun.
tagger = Tagger(API_KEY, 'localhost')
# or if you have your own bareun which is running on 10.8.3.211:15656.
tagger = Tagger(API_KEY, '10.8.3.211', 15656)
# print results.
res = tagger.tags(["안녕하세요.", "반가워요!"])
# get protobuf message.
m = res.msg()
tf.PrintMessage(m, out=sys.stdout, as_utf8=True)
print(tf.MessageToString(m, as_utf8=True))
print(f'length of sentences is {len(m.sentences)}')
## output : 2
print(f'length of tokens in sentences[0] is {len(m.sentences[0].tokens)}')
print(f'length of morphemes of first token in sentences[0] is {len(m.sentences[0].tokens[0].morphemes)}')
print(f'lemma of first token in sentences[0] is {m.sentences[0].tokens[0].lemma}')
print(f'first morph of first token in sentences[0] is {m.sentences[0].tokens[0].morphemes[0]}')
print(f'tag of first morph of first token in sentences[0] is {m.sentences[0].tokens[0].morphemes[0].tag}')
## Advanced usage.
for sent in m.sentences:
for token in sent.tokens:
for m in token.morphemes:
print(f'{m.text.content}/{m.tag}:{m.probability}:{m.out_of_vocab}')
# get json object
jo = res.as_json()
print(jo)
# get tuple of pos tagging.
pa = res.pos()
print(pa)
# another methods
ma = res.morphs()
print(ma)
na = res.nouns()
print(na)
va = res.verbs()
print(va)
import static org.junit.Assert.assertTrue;
import java.nio.file.Path;
import java.util.List;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.gson.Gson;
import ai.bareun.tagger.*;
import groovy.ui.Console;
import ai.bareun.protos.AnalyzeSyntaxRequest;
import ai.bareun.protos.AnalyzeSyntaxResponse;
import ai.bareun.protos.CustomDictionary;
import ai.bareun.protos.CustomDictionaryMeta;
import ai.bareun.protos.Document;
import ai.bareun.protos.LanguageServiceGrpc;
/**
* Unit test for simple App.
*/
public class AppTest
{ // Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
private static Logger logger = LoggerFactory.getLogger(AppTest.class.getSimpleName());
String API_KEY="koba-ABCDEFG-1234567-LMNOPQR-7654321" // <- replace with your own API KEY
void log(Object str){
if( str instanceof String ) {
logger.info(str.toString());
} else {
Gson gson = new Gson();
logger.info(gson.toJson(str));
}
}
/**
* Rigorous Test :-)
*/
@Test
public void shouldAnswerWithTrue()
{
LanguageServiceClient conn = new ai.bareun.tagger.LanguageServiceClient("localhost", API_KEY);
AnalyzeSyntaxResponse response = conn.analyze_syntax("아버지가 방에 들어가신다.");
String str = conn.toJson();
assertTrue( !str.isEmpty() );
logger.info(str);
}
static String TestString = "아버지가 방에 들어가신다.";
// "윤석열 대통령이 취임 220일차인 오는 15일 대국민 소통의 일환으로 ‘국정과제 점검회의’를 진행한다.\n 취임 후 처음 국민과 대면해 생방송으로 대화를 나눈다는 점에서 취지에 걸맞는 소통이 얼마나 이뤄질지 관심이다.";
@Test
public void pos() {
Tagged tag = new Tagger("localhost", API_KEY).tag(TestString);
List<?> ret;
Boolean flatten = true, join = true, detail = true;
ret = tag.pos(flatten, join, detail);
assertTrue(!ret.isEmpty());
log(String.format("flatten = %B, join = %B, detail = %B", flatten , join , detail));
log(ret);
flatten = true;
join = true;
detail = false;
ret = tag.pos(flatten, join, detail);
assertTrue(!ret.isEmpty());
log(String.format("flatten = %B, join = %B, detail = %B", flatten , join , detail));
log(ret);
flatten = true;
join = false;
detail = true;
ret = tag.pos(flatten, join, detail);
assertTrue(!ret.isEmpty());
log(String.format("flatten = %B, join = %B, detail = %B", flatten , join , detail));
log(ret);
flatten = false;
join = true;
detail = true;
ret = tag.pos(flatten, join, detail);
assertTrue(!ret.isEmpty());
log(String.format("flatten = %B, join = %B, detail = %B", flatten , join , detail));
log(ret);
flatten = false;
join = false;
detail = true;
ret = tag.pos(flatten, join, detail);
assertTrue(!ret.isEmpty());
log(String.format("flatten = %B, join = %B, detail = %B", flatten , join , detail));
log(ret);
}
@Test
public void morphs() {
Tagged tag = new Tagger("localhost", API_KEY).tag(TestString);
List<String> ret = tag.morphs();
assertTrue(!ret.isEmpty());
log("morphs()");
log(ret);
ret = tag.nouns();
assertTrue(!ret.isEmpty());
log("nouns()");
log(ret);
ret = tag.verbs();
assertTrue(!ret.isEmpty());
log("verbs()");
log(ret);
}
static public void main() {
AppTest theApp = new AppTest();
theApp.shouldAnswerWithTrue();
theApp.pos();
theApp.morphs();
theApp.testCustomDict();
}
}
let host="nlp.bareun.ai"
// Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
let API_KEY="koba-ABCDEFG-1234567-LMNOPQR-7654321" // <- replace with your own API KEY
let {LanguageServiceClient, Tagger, CustomDict} = require("bareun");
let language_service_client = new LanguageServiceClient(host, API_KEY);
language_service_client.AnalyzeSyntax("아버지가 방에 들어가신다.",
(error, res) => {
console.log('result : language_service_client.AnalyzeSyntax("아버지가 방에 들어가신다.")');
if( error ) {
throw error;
return;
}
console.log(JSON.stringify(res));
}
);
(async () => {
try {
let res = await language_service_client.asyncAnalyzeSyntax("아버지가 방에 들어가신다.")
console.log(JSON.stringify(res));
} catch(e) {
console.log(e);
}
})();
let tagged = await tagger.tag("미친 세상에서 맨정신으로 산다는 건 힘든 일이다.");
(async () => {
const t=true, f=false;
let obj;
obj = tagged.pos(t, t, t);
console.log("pos(t, t, t)"+JSON.stringify(obj));
obj = tagged.pos(t, t, f);
console.log("pos(t, t, f)"+JSON.stringify(obj));
obj = tagged.pos(t, f, t);
console.log("pos(t, f, t)"+JSON.stringify(obj));
obj = tagged.pos(f, t, t);
console.log("pos(f, t, t)"+JSON.stringify(obj));
obj = tagged.morphs();
console.log("morphs()"+JSON.stringify(obj));
obj = tagged.nouns();
console.log("nouns()"+JSON.stringify(obj));
obj = tagged.verbs();
console.log("verbs()"+JSON.stringify(obj));
})();
> library(RProtoBuf)
> library(bareun)
# Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
> set_key("YOUR_API_KEY") # <- replace with your own API KEY
> t <- tagger()
> text <- "문장을 입력합니다.\n여러 문장을 넣습니다."
> pos(t, text)
[[1]]
[1] "문장/NNG" "을/JKO" "입력하/VV" "ㅂ니다/EF" "./SF"
[[2]]
[1] "여러/MMN" "문장/NNG" "을/JKO" "넣/VV" "습니다/EF" "./SF"
Print the 4th morpheme of the 1st sentence
> postag(t, text)[[1]][[4]]
$morpheme
[1] "ㅂ니다"
$tag
[1] "EF"
Print eojeol, nouns, and verbs (an empty string array is returned if there are none)
> morphs(t)
[[1]]
[1] "문장" "을" "입력하" "ㅂ니다" "."
[[2]]
[1] "여러" "문장" "을" "넣" "습니다" "."
> nouns(t)
[[1]]
[1] "문장"
[[2]]
[1] "문장"
> verbs(t)
[[1]]
[1] "입력하"
[[2]]
[1] "넣"
Custom dictionary
A custom dictionary can be looked up, modified, and deleted. Custom dictionaries are managed separately for each API key. Even if you use the same dictionary name as another user, there is no conflict.
You only need to define the entries you want to specify.
A custom dictionary is accessed through the Tagger class.
Once saved, a dictionary can be reused.
import json
from bareunpy import Tagger
# Connect to the host where you installed the server.
# Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
my_tagger = Tagger('YOUR-API-KEY', 'localhost', port=5656) # <- replace with your own API KEY
# Save a custom dictionary
# Dictionary names may contain only English letters, digits, and the symbols '-' and '_'
# Symbols cannot be added to dictionary entries (except the ^ symbol in the compound noun splitting dictionary).
cust_dic = my_tagger.custom_dict("my_dict_01")
cust_dic.copy_np_set({'내고유명사', '걸어서세계속으로', '크리스토퍼놀란'})
cust_dic.copy_cp_set({'코로나19', '바나나우유'})
cust_dic.copy_cp_caret_set({'코로나^백신', '디지털^인문학'})
cust_dic.copy_vv_set({'카톡하다', '맑내하다'})
cust_dic.copy_va_set({'로맨틱하다', '신박하다', '판타스틱하다'})
cust_dic.update()
# Load a previously saved custom dictionary
cust_dict2 = my_tagger.custom_dict("my_dict_01")
cust_dict2.load()
text = '내고유명사로 존재하는 "크리스토퍼놀란"은 코로나19 상황 속에서도 걸어서세계속으로라는\
TV 프로그램을 통해 바나나우유를 마시며 카톡하다가 \
떠오른 신박한 아이디어로 판타스틱한 로맨틱 영화를 만들 계획이며, \
그 과정에서 디지털인문학의 도움을 받아 코로나백신에 대한 사회적 이슈도 함께 다루려 한다.'
# Print POS information only
my_tagger.set_domain('my_dict_01')
response = my_tagger.pos(text)
print(f'Morphological analysis result: {response}')
# Store the morphological analysis result as a dictionary
response2 = my_tagger.tags([text])
res_dict = json.loads(response2.as_json_str())
idx = 1
for token in res_dict['sentences'][0]['tokens']:
custom_token = [f"{t['text']['content']}/{t['tag']}, {t['outOfVocab']}" \
for t in token['morphemes'] if t['outOfVocab'] == 'IN_CUSTOM_DICT']
if len(custom_token) != 0:
print(f"{idx}: {custom_token}")
idx += 1
import static org.junit.Assert.assertTrue;
import java.nio.file.Path;
import java.util.List;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.gson.Gson;
import ai.bareun.tagger.*;
import groovy.ui.Console;
import ai.bareun.protos.AnalyzeSyntaxRequest;
import ai.bareun.protos.AnalyzeSyntaxResponse;
import ai.bareun.protos.CustomDictionary;
import ai.bareun.protos.CustomDictionaryMeta;
import ai.bareun.protos.Document;
import ai.bareun.protos.LanguageServiceGrpc;
/**
* Unit test for simple App.
*/
public class AppTest
{
// Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
private static Logger logger = LoggerFactory.getLogger(AppTest.class.getSimpleName());
String API_KEY="koba-ABCDEFG-1234567-LMNOPQR-7654321" // <- replace with your own API KEY
void log(Object str){
if( str instanceof String ) {
logger.info(str.toString());
} else {
Gson gson = new Gson();
logger.info(gson.toJson(str));
}
}
/**
* Rigorous Test :-)
*/
@Test
public void shouldAnswerWithTrue()
{
LanguageServiceClient conn = new ai.bareun.tagger.LanguageServiceClient("localhost", API_KEY);
AnalyzeSyntaxResponse response = conn.analyze_syntax("아버지가 방에 들어가신다.");
String str = conn.toJson();
assertTrue( !str.isEmpty() );
logger.info(str);
}
static String TestString = "아버지가 방에 들어가신다.";
// "윤석열 대통령이 취임 220일차인 오는 15일 대국민 소통의 일환으로 ‘국정과제 점검회의’를 진행한다.\n 취임 후 처음 국민과 대면해 생방송으로 대화를 나눈다는 점에서 취지에 걸맞는 소통이 얼마나 이뤄질지 관심이다.";
@Test
public void testCustomDict() {
CustomDict dict = new CustomDict("game", "localhost", undefined, API_KEY);
log("testCustomDict()");
log("read file.");
String curdir = System.getProperty("user.dir");
Path path = Path.of( curdir, "testdict.txt");
if( dict.read_np_set_from_file(path.toString()) <= 0 ) {
log("file io error : " + path.toString());
return ;
}
log(dict.getSet("np_set"));
Boolean r = dict.update();
assertTrue(r);
dict.load();
log("update and load.");
log(dict.getSet("np_set"));
log("get list ");
List<CustomDictionaryMeta> list = dict.get_list();
for(CustomDictionaryMeta meta: list) {
log(meta.getDomainName());
}
}
static public void main() {
AppTest theApp = new AppTest();
theApp.shouldAnswerWithTrue();
theApp.testCustomDict();
}
}
// Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
let host="nlp.bareun.ai"
let API_KEY="koba-ABCDEFG-1234567-LMNOPQR-7654321" // <- replace with your own API KEY
let {LanguageServiceClient, Tagger, CustomDict} = require("bareun");
let language_service_client = new LanguageServiceClient(host);
let dict = new CustomDict("game", host, undefined, API_KEY);
(async () => {
let set = new Set(["지지", "캐리", "던전", "현피", "세계관", "만렙","어그로","치트키","퀘스트","본캐","로밍","방사","딜러","버스","사플" ] );
dict.copy_cp_set(set);
let success = await dict.update();
console.log("result : dict.update() - " + success);
await dict.read_np_set_from_file(__dirname + "/game_dict.txt");
console.log("result : dict.load() - " + JSON.stringify([...dict.word_sets.np_set]));
let success = await dict.update();
console.log("result : dict.update() - " + success);
let res = await dict.client.async_get_list();
console.log("async_get_list() : " + JSON.stringify(res, null, 2));
await dict.load();
let res = await dict.clear();
console.log("clear() : " + JSON.stringify(res, null, 2));
})();
> library(RProtoBuf)
> library(bareun)
# Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
> set_key("YOUR_API_KEY") # <- replace with your own API KEY
> t <- tagger()
> text <- "문장을 입력합니다.\n여러 문장을 넣습니다."
> pos(t, text)
> np <- c("청하", "트와이스", "티키타카", "TIKITAKA", "오마이걸")
> cp <- c("자유여행", "방역당국", "코로나19", "주술부", "완전주의")
> caret <- c("주어^역할", "주어^술어^구조", "하급^공무원")
> vv <- c("카톡하다", "인스타하다")
> va <- c("혜자스럽다", "창렬하다")
> make_custom_dict(t, "sample", np, cp, caret, vv, va)
[1] "sample : 업데이트 성공" # "sample : update succeeded"
Word segmentation
Word segmentation performs only the segmentation step, without morphological tagging.
It is implemented as the tokenize function in the bareunpy library.
Currently this feature is implemented only in Python.
Documentation for tokenize is still incomplete
We have not yet written proper REST documentation for this feature. We will add it when time allows. In the meantime, you can try it out by clicking Tokenizer / Word segmentation on [bareun.ai]. This feature is also useful for improving the Korean performance of large language models.
import sys
import google.protobuf.text_format as tf
from bareunpy import Tokenizer
# Enter the API KEY ("koba-...") you received after email verification at "https://bareun.ai/" below. ("Log in" - "My info")
API_KEY = 'koba-ABCDEFG-1234567-LMNOPQR-7654321' # <- replace with your own API KEY
# If you have your own localhost bareun.
my_tokenizer = Tokenizer(API_KEY, 'localhost')
# or if you have your own bareun which is running on 10.8.3.211:15656.
my_tokenizer = Tagger(API_KEY, '10.8.3.211', 15656)
# print results.
tokenized = my_tokenizer.tokenize_list(["안녕하세요.", "반가워요!"])
# get protobuf message.
m = tokenized.msg()
tf.PrintMessage(m, out=sys.stdout, as_utf8=True)
print(tf.MessageToString(m, as_utf8=True))
print(f'length of sentences is {len(m.sentences)}')
## output : 2
print(f'length of tokens in sentences[0] is {len(m.sentences[0].tokens)}')
print(f'length of segments of first token in sentences[0] is {len(m.sentences[0].tokens[0].segments)}')
print(f'tagged of first token in sentences[0] is {m.sentences[0].tokens[0].tagged}')
print(f'first segment of first token in sentences[0] is {m.sentences[0].tokens[0].segments[0]}')
print(f'hint of first morph of first token in sentences[0] is {m.sentences[0].tokens[0].segments[0].hint}')
## Advanced usage.
for sent in m.sentences:
for token in sent.tokens:
for m in token.segments:
print(f'{m.text.content}/{m.hint}')
# get json object
jo = tokenized.as_json()
print(jo)
# get tuple of segments
ss = tokenized.segments()
print(ss)
ns = tokenized.nouns()
print(ns)
vs = tokenized.verbs()
print(vs)
# postpositions
ps = tokenized.postpositions()
print(ps)
# Adverbs
ass = tokenized.adverbs()
print(ass)
ss = tokenized.symbols()
print(ss)