GPU environment setup

Bareun uses deep learning models for morphological analysis. A GPU has more processing cores than a CPU and can run many computations at the same time, so if you use a GPU, Bareun runs faster.

Starting with Bareun 3.0, ONNX Runtime is used (1.29.0 since 3.2.0), supporting a variety of hardware acceleration options. You can choose a CPU or GPU execution environment through the executionProvider item in the configuration file (bareun.json).

Supported execution providers

Bareun supports the following execution providers:

CPU: Default CPU execution (all platforms)
CUDA: NVIDIA GPU acceleration (Linux, Windows)
TensorRT: NVIDIA TensorRT optimization engine (Linux, Windows)
DirectML: DirectX 12-based GPU acceleration (Windows)
CoreML: Apple Neural Engine acceleration (macOS)
DNNL: Intel CPU optimization (all platforms)
OpenVINO: Intel GPU/CPU/NPU acceleration (all platforms)
ROCm: AMD GPU acceleration (Linux)
MIGraphX: AMD GPU optimization (Linux)

Support by platform

The following table shows platform support for each execution provider:

Execution Provider	Linux	Windows	macOS	Description
CPU	✓	✓	✓	Default CPU execution (all platforms)
CUDA	✓	✓	✗	NVIDIA GPU acceleration
TensorRT	✓	✓	✗	NVIDIA TensorRT optimization
DirectML	✗	✓	✗	DirectX 12-based GPU acceleration (Windows only)
CoreML	✗	✗	✓	Apple Neural Engine acceleration (macOS/iOS only)
DNNL	✓	✓	✓	Intel CPU optimization (all platforms)
OpenVINO	✓	✓	✓	Intel GPU/CPU/NPU acceleration (all platforms)
ROCm	✓	✗	✗	AMD GPU acceleration (Linux only)
MIGraphX	✓	✗	✗	AMD GPU optimization (Linux only)

Recommendations by platform

Linux: CUDA/TensorRT (NVIDIA), ROCm/MIGraphX (AMD), OpenVINO (Intel)
Windows: CUDA/TensorRT (NVIDIA), DirectML (general-purpose GPU), OpenVINO (Intel)
macOS: CoreML (Apple Silicon), DNNL (CPU optimization). Intel Macs are not supported since 3.2.0

Configuration examples

The config/ folder in the installation directory contains various example configuration files:

bareun.cpu-example.json: CPU execution example
bareun.cuda-example.json: CUDA GPU execution example
bareun.tensorrt-example.json: TensorRT optimized execution example
bareun.directml-example.json: DirectML (Windows) execution example
bareun.coreml-example.json: CoreML (macOS) execution example
bareun.dnnl-example.json: Intel DNNL CPU optimization example
bareun.openvino-example.json: Intel OpenVINO execution example
bareun.rocm-example.json: AMD ROCm GPU execution example
bareun.migraphx-example.json: AMD MIGraphX optimization example

You can modify bareun.json as needed by referring to the example files.

Explanations

The config/ folder in the installation directory also contains files that explain the configuration in detail:

README_EXAMPLES.md: Purpose and usage of each example file
EXECUTION_PROVIDER.md: Detailed explanation of the execution provider options

1. CPU mode (default)

This is the default setting, running on the CPU only. No separate GPU driver installation is needed.

{
  "executionProvider": {
    "provider": "cpu"
  }
}

If you omit the executionProvider item, it runs in CPU mode automatically.

2. CUDA mode (NVIDIA GPU)

This mode accelerates with an NVIDIA GPU. It is available on Linux and Windows.

Prerequisites

To use CUDA, you need:

An NVIDIA GPU (Compute Capability 3.5 or higher)
CUDA Toolkit 11.8 or higher
cuDNN library 8.x or higher
The latest NVIDIA driver

Configuration example

{
  "executionProvider": {
    "provider": "cuda",
    "cudaOptions": {
      "deviceId": 0,
      "gpuMemLimit": "2G",
      "arenaExtendStrategy": "kNextPowerOfTwo",
      "cudnnConvAlgoSearch": "HEURISTIC",
      "doCopyInDefaultStream": true,
      "enableCudaGraph": false
    }
  }
}

CUDA options in detail

Option	Type	Default	Description
`deviceId`	Integer	0	Number of the GPU device to use (starting from 0). If you have several GPUs, choose one of 0, 1, 2, etc.
`gpuMemLimit`	String/integer	"2G"	Limit on GPU memory use. Supports "2G", "512M", "1024K", "1T" or a number of bytes (units: K/KB, M/MB, G/GB, T/TB, case-insensitive)
`arenaExtendStrategy`	String	"kNextPowerOfTwo"	Memory allocation strategy. "kNextPowerOfTwo" (default) or "kSameAsRequested" (case-sensitive)
`cudnnConvAlgoSearch`	String	"DEFAULT"	How to search for cuDNN convolution algorithms. "EXHAUSTIVE", "HEURISTIC", "DEFAULT" (case-sensitive)
`doCopyInDefaultStream`	Boolean	true	Whether to perform copy operations in the default stream (true or false)
`enableCudaGraph`	Boolean	false	Enables CUDA Graph optimization (experimental, true or false)

Memory size units

gpuMemLimit supports the following formats:

Kilobytes (KB): "1024K", "1024k", "1024KB", "1024kb"
Megabytes (MB): "512M", "512m", "512MB", "512mb"
Gigabytes (GB): "2G", "2g", "2GB", "2gb"
Terabytes (TB): "1T", "1t", "1TB", "1tb"
Bytes: "2147483648" (a number alone is interpreted as bytes)

Units are case-insensitive and all are calculated in base 1024 (binary).

Example: "2G" = "2048M" = "2097152K" = "2147483648"

Watch the case of string options

The arenaExtendStrategy and cudnnConvAlgoSearch options are case-sensitive. For example, "knextpoweroftwo" or "KNEXTPOWEROFTWO" will not work; you must enter exactly "kNextPowerOfTwo".

Single GPU use

Currently Bareun uses only one GPU device per session. deviceId is for choosing one of the several GPUs installed in the system.

3. TensorRT mode (NVIDIA optimization)

This mode performs additional optimization with NVIDIA TensorRT. It can deliver higher performance than CUDA.

Prerequisites

To use TensorRT, you need the following in addition to the CUDA requirements:

TensorRT 8.6 or higher

Configuration example

{
  "executionProvider": {
    "provider": "tensorrt",
    "tensorrtOptions": {
      "deviceId": 0,
      "maxWorkspaceSize": "1G",
      "maxPartitionIterations": 1000,
      "minSubgraphSize": 1,
      "fp16Enable": true,
      "int8Enable": false,
      "trtEngineCachePath": "${BAREUN_ROOT}/var/trt_cache",
      "trtEngineCacheEnable": true,
      "trtDlaCore": -1,
      "trtLayerNormFp32Fallback": false
    }
  }
}

TensorRT options in detail

Option	Type	Default	Description
`deviceId`	Integer	0	Number of the GPU device to use (starting from 0). Chooses one of the system's GPUs
`maxWorkspaceSize`	String/integer	"1G"	Maximum TensorRT workspace size. Supports "1G", "512M", "1024K", "1T" or a number of bytes (units: K/KB, M/MB, G/GB, T/TB, case-insensitive)
`maxPartitionIterations`	Integer	1000	Maximum number of graph partitioning iterations
`minSubgraphSize`	Integer	1	Minimum subgraph size to convert with TensorRT
`fp16Enable`	Boolean	true	Enables FP16 half-precision arithmetic (faster, slight loss of accuracy, true or false)
`int8Enable`	Boolean	false	Enables INT8 quantization (fastest, possible loss of accuracy, true or false)
`trtEngineCachePath`	String	-	Cache path for compiled TensorRT engines. The `${BAREUN_ROOT}` variable can be used
`trtEngineCacheEnable`	Boolean	true	Whether to use the engine cache (if true, faster loading on restart, true or false)
`trtDlaCore`	Integer	-1	DLA (Deep Learning Accelerator) core number. -1 means not used
`trtLayerNormFp32Fallback`	Boolean	false	Falls back to FP32 for LayerNorm (improves accuracy, true or false)

Memory size units

maxWorkspaceSize supports various units:

Kilobytes: "1024K", "1024k", "1024KB", "1024kb"
Megabytes: "512M", "512m", "512MB", "512mb"
Gigabytes: "1G", "1g", "1GB", "1gb"
Terabytes: "1T", "1t", "1TB", "1tb"
Bytes: "1073741824" (enter a number alone)

Units are case-insensitive and all are calculated in base 1024.

TensorRT engine cache

If you set trtEngineCacheEnable to true, on the first run the model is converted into a TensorRT engine and saved to the cache. Later runs load the cached engine, so startup time is shortened greatly.

4. DirectML mode (Windows GPU)

This is a general-purpose GPU acceleration mode that uses DirectX 12 on Windows. It works not only on NVIDIA but also on AMD and Intel GPUs.

Prerequisites

Windows 10 version 1903 or higher
A DirectX 12-capable GPU
The latest graphics driver

Configuration example

{
  "executionProvider": {
    "provider": "directml",
    "directmlOptions": {
      "deviceId": 0,
      "enableMetacommands": true,
      "enableGraphCapture": true
    }
  }
}

DirectML options in detail

Option	Type	Default	Description
`deviceId`	Integer	0	Number of the GPU device to use (starting from 0). Chooses one of the system's GPUs
`enableMetacommands`	Boolean	true	Enables DirectML metacommand optimization (true or false)
`enableGraphCapture`	Boolean	true	Enables graph capture optimization (true or false)

5. CoreML mode (macOS acceleration)

This mode uses the Neural Engine of Apple Silicon. It is macOS only.

Prerequisites

macOS 13.4 (Ventura) or higher
Apple Silicon (M1 or later)

Configuration example

{
  "executionProvider": {
    "provider": "coreml",
    "coremlOptions": {
      "useMlprogram": true,
      "enableOnSubgraph": false,
      "enableModelCaching": true,
      "modelCacheDir": "${BAREUN_ROOT}/var/coreml_cache"
    }
  }
}

CoreML options in detail

Option	Type	Default	Description
`useMlprogram`	Boolean	true	Uses the ML Program format (iOS 15+, macOS 12+, true or false). If false, the legacy Neural Network format
`enableOnSubgraph`	Boolean	false	Enables CoreML on subgraphs too (true or false)
`enableModelCaching`	Boolean	true	Enables the compiled model cache (true or false)
`modelCacheDir`	String	-	Path of the model cache directory. The `${BAREUN_ROOT}` variable can be used

Apple Silicon optimization

When you use CoreML on Apple Silicon (M1/M2/M3), the Neural Engine is used automatically, enabling power-efficient, high-performance inference.

macOS version requirements

The macOS build of ONNX Runtime requires macOS 13.4 (Ventura) or higher. The useMlprogram option can be used on macOS 12.0 or higher, but ONNX Runtime itself works on 13.4 or higher.

CoreML support limits

The transformer-based ONNX models that Bareun uses are currently not fully supported by CoreML.

CoreML does not support some operators (multi_head_attention, Tensordot, etc.) of transformer models with dynamic shapes
If you set CoreML and an error occurs during inference, it automatically falls back to CPU mode
Recommendation: On macOS, use CPU mode or DNNL mode

We plan to improve CoreML support through future model optimization.

6. DNNL mode (Intel CPU optimization)

Intel DNNL (Deep Neural Network Library, formerly MKL-DNN) provides optimized performance on Intel CPUs.

Configuration example

{
  "executionProvider": {
    "provider": "dnnl",
    "dnnlOptions": {
      "numThreads": 0,
      "useArena": true
    }
  }
}

DNNL options in detail

Option	Type	Default	Description
`numThreads`	Integer	0	Number of OpenMP threads (0 = automatic)
`useArena`	Boolean	true	Whether to use a memory arena (true or false)

Intel CPU optimization

On Intel Xeon and Core i series CPUs, it provides better performance than the regular CPU mode.

7. OpenVINO mode (Intel GPU/NPU)

Intel OpenVINO supports a variety of Intel hardware, including Intel GPUs, CPUs, and NPUs.

Prerequisites

An Intel GPU (Arc, Iris Xe, UHD Graphics, etc.)
OpenVINO Runtime installed

Configuration example

{
  "executionProvider": {
    "provider": "openvino",
    "openvinoOptions": {
      "deviceType": "GPU",
      "precision": "FP16",
      "enableDynamicShapes": false,
      "cacheDir": "${BAREUN_ROOT}/var/openvino_cache",
      "numThreads": 0
    }
  }
}

OpenVINO options in detail

Option	Type	Default	Description
`deviceType`	String	"CPU"	Device type: "CPU", "GPU", "GPU.0", "GPU.1", "NPU", etc.
`precision`	String	"FP32"	Precision: "FP32", "FP16", "U8" (INT8)
`enableDynamicShapes`	Boolean	false	Enables dynamic shape support (true or false)
`cacheDir`	String	-	Compiled model cache directory. The `${BAREUN_ROOT}` variable can be used
`numThreads`	Integer	0	Number of threads to use in CPU mode (0 = automatic)

Using an Intel GPU

If you have an Intel Arc or Iris Xe GPU, you can use GPU acceleration by setting deviceType: "GPU".

8. ROCm mode (AMD GPU)

AMD ROCm is AMD's counterpart to CUDA, usable on AMD Radeon GPUs.

Prerequisites

An AMD Radeon GPU (Linux only)
ROCm platform installed

Configuration example

{
  "executionProvider": {
    "provider": "rocm",
    "rocmOptions": {
      "deviceId": 0,
      "gpuMemLimit": "2G",
      "arenaExtendStrategy": "kNextPowerOfTwo",
      "miopenConvAlgoSearch": "HEURISTIC",
      "doCopyInDefaultStream": true
    }
  }
}

ROCm options in detail

Option	Type	Default	Description
`deviceId`	Integer	0	Number of the GPU device to use
`gpuMemLimit`	String/integer	"2G"	Limit on GPU memory use. Supports "2G", "512M", "1024K", "1T" or a number of bytes (units: K/KB, M/MB, G/GB, T/TB, case-insensitive)
`arenaExtendStrategy`	String	"kNextPowerOfTwo"	Memory allocation strategy. "kNextPowerOfTwo" (default) or "kSameAsRequested" (case-sensitive)
`miopenConvAlgoSearch`	String	"DEFAULT"	How to search for MIOpen convolution algorithms. "EXHAUSTIVE", "HEURISTIC", "DEFAULT" (case-sensitive)
`doCopyInDefaultStream`	Boolean	true	Whether to perform copy operations in the default stream (true or false)

Memory size units

gpuMemLimit supports various units:

Kilobytes: "1024K", "1024k", "1024KB", "1024kb"
Megabytes: "512M", "512m", "512MB", "512mb"
Gigabytes: "2G", "2g", "2GB", "2gb"
Terabytes: "1T", "1t", "1TB", "1tb"
Bytes: "2147483648" (enter a number alone)

Units are case-insensitive and all are calculated in base 1024.

Watch the case of string options

The arenaExtendStrategy and miopenConvAlgoSearch options are case-sensitive.

Linux only

ROCm is currently supported only on Linux. It cannot be used on Windows.

9. MIGraphX mode (AMD optimization)

AMD MIGraphX is a graph compiler that provides additional optimization on AMD GPUs.

Prerequisites

An AMD Radeon GPU (Linux only)
ROCm and MIGraphX installed

Configuration example

{
  "executionProvider": {
    "provider": "migraphx",
    "migraphxOptions": {
      "deviceId": 0,
      "fp16Enable": true,
      "int8Enable": false,
      "cachePath": "${BAREUN_ROOT}/var/migraphx_cache",
      "enableCache": true
    }
  }
}

MIGraphX options in detail

Option	Type	Default	Description
`deviceId`	Integer	0	Number of the GPU device to use
`fp16Enable`	Boolean	false	Enables FP16 (half-precision) mode (true or false)
`int8Enable`	Boolean	false	Enables INT8 quantization mode (true or false)
`cachePath`	String	-	Compiled graph cache path. The `${BAREUN_ROOT}` variable can be used
`enableCache`	Boolean	false	Enables the cache feature (true or false)

AMD GPU optimization

Setting fp16Enable: true can improve performance significantly.

Applying the configuration file

Open the config/bareun.json file in an editor
Add or modify the settings for the execution provider you want
Save your changes
Restart the Bareun service

sudo systemctl restart bareun

net stop bareunsvc
net start bareunsvc

Or restart the 바른 한국어 형태소분석기 (Bareun Korean morphological analyzer) service in Services Manager.

# Check the service (the text ai.baikal.bareun is printed.)
sudo launchctl list | grep bareun

# Stop the service (shuts the service down completely and unloads it from memory.)
sudo launchctl bootout system/ai.baikal.bareun

# Start the service (turns a stopped service back on, or loads it manually after boot.)
sudo launchctl bootstrap system /Library/LaunchDaemons/ai.baikal.bareun.plist

# Restart the service (force-terminates the process and restarts it immediately when you changed settings or the service misbehaves.)
sudo launchctl kickstart -k system/ai.baikal.bareun

docker restart bareun

# Terminate the process
pkill bareun

# Restart the server
export BAREUN_ROOT=$HOME/bareun-v3.0.0
nohup $BAREUN_ROOT/bin/bareun &

Checking GPU use

How to check that the GPU is being used properly:

# Monitor GPU utilization with nvidia-smi
watch -n 1 nvidia-smi

You can see GPU memory usage increase when Bareun runs.

# Monitor GPU utilization with rocm-smi
watch -n 1 rocm-smi

You can see GPU utilization increase when Bareun runs.

# Use the Intel GPU Top tool (Linux)
intel_gpu_top

Or you can check GPU utilization in a system monitor.

Troubleshooting

If the GPU is not recognized

Check that the GPU driver is up to date
Check that the CUDA/cuDNN versions are compatible
Check for error messages in the log files (var/logs/)
Switch to CPU mode to confirm basic operation

Out-of-memory errors

Reduce the gpuMemLimit (CUDA) or maxWorkspaceSize (TensorRT) value
Reduce the number of requests processed at the same time
Close other programs that use the GPU

If performance is lower than expected

Check that fp16Enable: true is set in TensorRT mode
Check that the cache feature is enabled (trtEngineCacheEnable, enableModelCaching)
The first run may be slow because of model compilation; later runs are faster

Notes

Experimental feature

GPU acceleration is currently at an experimental stage. Keep the following in mind:

Some options may not work properly on certain hardware or driver versions
Performance gains can vary with hardware specifications and model characteristics
We recommend sufficient testing before applying it to a production environment
Not all of the GPU acceleration features in this document have been tested yet

More information