SLA and response time design tips
To use Bareun reliably in a service that handles heavy traffic, you need to consider response time and throughput together at the design stage. Bareun gives you several levers, such as the execution environment (executionProvider), batch calls, caching, and concurrency.
This document collects design tips for making response time predictable and raising throughput.
What determines response time
| Factor | Description | Lever |
|---|---|---|
| Execution environment | Choice of CPU or GPU inference | executionProvider (cpu/gpu) |
| Call unit | Single call vs batch | AnalyzeSyntaxList |
| Concurrency | Handling parallel requests | gRPC multiplexing |
| Repeated input | Avoid recomputing the same input | Application cache |
| Sentence length | Segmentation processes 160 characters at a time | Input cleanup |
Choosing the execution environment
A self-hosted Bareun installation lets you choose the execution provider in its settings.
Set executionProvider to cpu or gpu to match your inference environment.
{
"executionProvider": "gpu"
}
- If throughput is high and latency matters, consider a GPU environment.
- If the inference accelerator (EP, execution provider) fails, Bareun automatically falls back to CPU to protect availability.
Reduce round trips with batches
Instead of calling once per sentence, group several sentences into one request with AnalyzeSyntaxList. This cuts network round trips and lowers overall latency. Decide the batch size by measuring, watching memory and latency, to find a suitable value.
graph TD L[Latency target] --> B[Adjust batch size] T[Throughput target] --> C[Adjust number of concurrent requests] B --> M[Measure and monitor] C --> M M --> A[Readjust settings]
Concurrency and caching
- Concurrency: Sending several batches in parallel raises throughput through server-side multiplexing. However, too many concurrent requests can increase latency, so set the upper limit by measuring.
- Caching: If the same input repeats often, caching analysis results in the application is an effective way to reduce duplicate calls.
- Cloud auto-scaling: The cloud (
api.bareun.ai:443) adjusts capacity automatically with usage, so traffic spikes are absorbed without any extra action.
Set goals first, then measure
Set goals first, such as "p95 latency under X ms" or "at least Y sentences per second", then measure while changing the batch size, number of concurrent requests, and execution environment. Bareun maintains 99.6% (POS tagging) and 99.7% (eojeol splitting/restoration) accuracy, so you can focus tuning on balancing throughput and latency rather than accuracy.
Frequently asked questions
Q. Where should I start to reduce response time?
First check that the execution environment (executionProvider) fits your load, and group single calls into AnalyzeSyntaxList batches to reduce round trips. Then adjust the number of concurrent requests and caching.
Q. Does the service stop if the inference accelerator fails?
No. If the execution provider (EP) fails, Bareun automatically falls back to CPU and keeps the service available.
Q. How does the cloud respond when traffic suddenly increases?
The cloud service is configured so that capacity adjusts automatically with usage, so it handles requests during traffic spikes without any extra action.