Dictionary update cycle and quality control: how freshness is maintained
The built-in dictionaries of Bareun are not created by the server itself.
They are generated in the training track and delivered to the server through a deployment process.
This page explains how dictionaries are updated and what mechanisms are in place for fast startup and quality.
Generate → deploy: how the two tracks divide the work
Bareun is split into a training track that builds dictionaries and a service (server) that reads dictionaries to run inference. To reflect new vocabulary, the dictionaries must be rebuilt in the training track and deployed.
graph LR SRC[Corpus and dictionary sources] --> BUILD[Generate dictionaries in the training track]; BUILD --> DEPLOY[Register in the deployment list]; DEPLOY --> FETCH[Server downloads at startup]; FETCH --> SERVER[Server loads the dictionaries]; SERVER --> INFER[Morphological analysis and correction];
Dictionaries are not in the installer
Models and dictionaries are not packed into the installer. The server downloads the ones that match its version the first time it runs and stores them in the data area. So you do not need to reinstall when a new dictionary is released. To check what is currently in use or to switch to a new model, see Model update.
| Dictionary | Source of generation |
|---|---|
dict.res | Trained token index |
ext-dict.fb | Extended vocabulary such as surnames, quantities, and wiki entries |
urimal-dict.fb | Urimalsaem (Open Korean Dictionary) vocabulary |
onterm-dict.fb | Onterm (Korean terminology platform) technical terms |
word-freq.gob | Corpus frequency counts (takes 10+ hours) |
Why doesn't the server build them itself?
Building dictionaries is heavy work that handles large corpora. The frequency dictionary alone takes more than 10 hours to build. If the inference server took on this work, the service would slow down, so the training track does the heavy generation and the server only reads the finished dictionaries quickly.
Storage format for fast startup
Dictionaries are large, so reading them from scratch and building data structures every time would make server startup slow. Bareun speeds up startup with its storage format and caching.
- FlatBuffer + mmap: Built-in dictionaries are stored in
.fb(FlatBuffer) format and mapped with mmap. There is no step that reads the whole file in, so startup time does not grow even as dictionaries get bigger. The extended (ext-dict.fb), Hanja (hanja-dict.fb), loanword (foreign-exam-dict.fb), non-standard word (non-std-dict.fb), Urimalsaem (urimal-dict.fb), and Onterm (onterm-dict.fb) dictionaries all use this method. - Preparation at build time: Preparation needed for lookups, such as sorting and indexing, is finished at the stage where the dictionary is built. The server only looks up the prepared structures as they are.
- gob caching: Auxiliary data derived from dictionaries, such as the BK-tree used to find typo candidates, is cached as
gob, and loaded at the next startup instead of being rebuilt.
BK-tree versioning
The BK-tree used to find typo candidates is built from Urimalsaem and is versioned (for example, v1.2.0).
When the dictionary or the tree structure changes, the version is bumped so that the cached old tree is replaced with the new one.
Balancing freshness and stability
Vocabulary stays current because the training track updates the corpus and source dictionaries, while the server guarantees fast and consistent behavior through caching and versioning. Separating generation from inference makes this balance possible.
Frequently asked questions
Q. Who updates the dictionaries, and how?
The training track rebuilds dictionaries from the corpus and dictionary sources, and they reach the server through a deployment process. The inference server only reads the finished dictionaries.
Q. Aren't server starts slow because the dictionaries are large?
Built-in dictionaries are in FlatBuffer (.fb) format, so they are only mapped with mmap and looked up right away. There is no step of reading the whole file and
rebuilding data structures, so startup time does not grow even as dictionaries get bigger. Auxiliary data derived from dictionaries
(such as the BK-tree) is loaded from the gob cache.
Q. How does the BK-tree stay up to date?
The BK-tree is built from the Urimalsaem dictionary and managed by version (for example, v1.2.0).
When the structure changes, the version is bumped so that the new tree is used instead of the cached old one.
Q. How do I add new vocabulary or neologisms to the dictionary?
To expand the vocabulary of the built-in dictionaries, the training track must update the corpus and source dictionaries, rebuild the dictionaries, and deploy them.
If you need a specific word to be recognized right now, registering it in the custom dictionary through the CustomDictionaryService API is the
quick way. Custom dictionary files are binary, so apply changes with UpdateCustomDictionary.
Q. When a dictionary is updated, are both morphological analysis and the spell checker affected?
Dictionaries included in every build, such as dict.res and ext-dict.fb, affect morphological analysis,
while the Urimalsaem, non-standard word, loanword, Hanja, Onterm, and frequency dictionaries are used only in builds that include the spell checker,
so they affect correction behavior.