How word spacing and joining decisions work

Korean word spacing includes cases with clear rules and ambiguous cases where the same string can be written with or without a space depending on context. Bareun handles this in two stages: it first fixes what is certain by rules, and uses an LLM to decide only the ambiguous cases.

Stage 1: rule-based word spacing correction

Cases that must definitely be joined or spaced are handled by rules.

• 아름다운강산 → 아름다운 강산: a noun after a modifier is written with a space.
• 떨여졌는데 → 떨어졌는데: eojeol (space-delimited word unit) boundaries are fixed by segmentation and form restoration.

This stage is corrected under the SPACING (word spacing) category of RevisionCategory, and can carry a rule basis such as "Hangul Spelling Rules (한글맞춤법) Article 2: each word of a sentence is, in principle, written separated by spaces."

Stage 2: LLM decision for ambiguity

The problem is the case where the same characters are spaced differently depending on context.

For this kind of ambiguity, rules alone can hardly settle the answer. Bareun marks such eojeols as ambiguous in Stage 1, and at the final step of the pipeline, where word spacing ambiguity is decided by an LLM, it gives the LLM the context and has it decide whether to space or join.

graph TD
  A[Review eojeol boundary] --> B{Can rules<br>decide it?};
  B -- Yes --> C[Rule correction<br>e.g. 아름다운강산 → 아름다운 강산];
  B -- No: ambiguous --> D[Mark as ambiguous];
  D --> E[Word spacing ambiguity<br>LLM context decision];
  E --> F{Join compound<br>noun or not?};
  F --> G[Compound noun joining decision];
  G --> H[Final spacing decision];

LLM decision results are stored in the word spacing decision cache (key = MD5(spaced form | joined form | context)), so the same input is not sent to the LLM again.

Word spacing options

You can adjust word spacing behavior with RevisionConfig (all default to false).

Frequently asked questions

Q. How are 그다음 and 그 다음 told apart?

This is ambiguous word spacing that is hard to settle by rules alone. Bareun marks such eojeols as ambiguous, and at the final step of the pipeline, where word spacing ambiguity is decided by an LLM, it gives the LLM the context and has it decide which reads more naturally.

Q. Are compound nouns written joined or spaced?

Compound nouns are decided by a process that judges whether to join them. Boundaries that rules cannot settle with confidence are handed to the LLM decision, which chooses the one that fits the context.

Q. Can I turn off or adjust word spacing correction?

Yes. You can adjust the behavior with disable_caret_spacing, disable_vx_spacing, enable_cleanup_whitespace, and treat_as_title in RevisionConfig.

Q. Are wrong eojeol boundaries also fixed at the word spacing stage?

Yes. Corrections that fix eojeol boundaries through segmentation and form restoration, such as 떨여졌는데 → 떨어졌는데, are handled together in the Stage 1 rule-based word spacing. Eojeols that were wrongly joined or split morphologically are restored to the correct form.