When GPU execution falls back to CPU

You set it up to use the GPU, but it seems to be running on the CPU? Bareun's ONNX inference automatically falls back to the CPU if initializing the execution provider (EP) fails. Thanks to that, the server does not stop, but you do not get the GPU acceleration you expected. Use this page to find the cause.

Symptoms

• You set executionProvider to gpu, but inference speed is about the same as on the CPU.
• GPU utilization is almost 0.
• The startup log shows an execution provider initialization warning, and the server keeps working normally after that.

Cause

Bareun first tries a GPU-class execution provider (for example, CUDA or CoreML) in the ONNX Runtime, and if initialization fails, it automatically falls back to the CPU. The server does not die, so it is hard to notice, and the cause is usually in the environment.

Solution

1. Check the setting first. See whether executionProvider in bareun.json is gpu. If it is set to cpu, it naturally runs on the CPU.

   Copy

   {
     "executionProvider": "gpu"
   }
   

2. Check the startup log. If GPU EP initialization fails, a warning is left right before the fallback. Which provider failed and why is the key to diagnosing the cause.

3. Check the driver and runtime. In a CUDA environment, confirm that the GPU driver, CUDA, and cuDNN are all installed and compatible with each other. For details on matching them, see CUDA/cuDNN version mismatch.

4. Check that the GPU libraries are present. Confirm that you installed a package that includes the GPU ONNX Runtime library under libPath. You cannot get GPU acceleration with a CPU-only package.

Prevention and tips

• Before putting it into production, start it once in the GPU environment and check that the log has no fallback warning.
• If an environment does not need a GPU, explicitly setting executionProvider to cpu also helps reduce confusion.

Frequently asked questions

Q. Why does it run on the CPU when I set it to GPU?

Because Bareun automatically falls back to the CPU when initializing the GPU execution provider fails. A missing driver or CUDA/cuDNN, a version mismatch, and a missing GPU library are the common causes, and you can find the reason in the fallback warning in the startup log.

Q. How do I check whether it fell back to the CPU?

Look at the server startup log. If GPU EP initialization fails, a warning is left and it then switches to the CPU. If GPU utilization is almost 0, it has most likely fallen back.

Q. It should be on the GPU but runs on the CPU. What should I fix first?

First check that executionProvider in bareun.json is gpu, and look in the startup log to see which execution provider failed and why. Then check that the GPU driver, CUDA, and cuDNN are compatible, and that libPath contains the GPU ONNX Runtime library. You cannot get GPU acceleration with a CPU-only package.

Q. Are the results the same after a CPU fallback?

Yes. Only the execution provider differs, and the morphological analysis results themselves are identical. The only difference is speed, so in an environment that does not need acceleration, you can keep running in the fallback state. However, if you need acceleration and it fell back, use the log to correct the GPU environment.