dict.res: the trained vocabulary and index resource
dict.res is the base built-in dictionary that Bareun creates during training.
It holds the token index the model uses to refer to vocabulary, and it is the most fundamental
resource for morphological analysis. It is always included, even in the default build that offers only morphological analysis without spelling correction.
The token index: token_index
The core of dict.res is the token index (token_index).
The model handles characters and morphemes as numeric indices, and the token index is the table that connects these indices to the actual vocabulary.
graph LR T["Token (morpheme)"] --> IDX[token_index]; IDX --> NUM[Integer index]; NUM --> MODEL[Model input tensor];
- The segmentation and POS tagging models take their input as an array of integer indices.
- The token index converts vocabulary into those integers and converts the indices the model outputs back into vocabulary.
- You need this index to exchange the trained vocabulary with the model consistently.
Why is it shipped together with the trained resources?
The token index is the vocabulary-to-index mapping fixed when the model was trained.
If the model and the index are out of sync, the same index points to a different word and the analysis breaks.
So dict.res is shipped as one bundle with the models (seg-model.onnx and tag-model.onnx).
Base resources included in every build
| Dictionary | Default build | Spell-check build |
|---|---|---|
dict.res | Included | Included |
ext-dict.fb | Included | Included |
| Correction dictionaries (Urimalsaem, non-standard words, and so on) | Not included | Included |
Unlike the correction dictionaries, dict.res is a base resource that is included in every build.
This is because morphological analysis itself cannot run without this index.
A dictionary that goes hand in hand with the model
dict.res is not so much a dictionary you update on its own as a vocabulary index paired with the trained model.
When a new model is trained, the matching dict.res is created and shipped with it.
Frequently asked questions
Q. What does dict.res contain?
It contains the trained token index (token_index). It is the table connecting the integer indices the model works with to
the actual vocabulary, and it serves as the base vocabulary for morphological analysis.
Q. Which builds include dict.res?
dict.res is always included in both the default build and the build that includes the spell checking feature.
This is because morphological analysis does not work without this index.
Q. Can I update dict.res separately?
dict.res is a resource paired with the trained model, so it is trained and shipped together with the model.
If the model and the index are out of sync, the analysis breaks, so they are managed as one bundle.
Q. How is dict.res different from a custom dictionary?
dict.res is the base built-in vocabulary index created together with the model during training,
while a custom dictionary is a separate resource that holds vocabulary specific to a domain or organization.
You manage custom dictionaries through the CustomDictionaryService API, and dict.res is shipped as one bundle with the model.
Q. What role does the token index play in morphological analysis?
The segmentation and POS tagging models take their input as an array of integer indices. The token index converts vocabulary into those integers and converts the indices the model outputs back into vocabulary. You need this index to exchange the trained vocabulary with the model consistently and perform morphological analysis.