Diagnosing and fixing CUDA/cuDNN version mismatches

Did you turn on GPU acceleration, but inference falls back to the CPU or fails at initialization? In GPU environments, the most common cause is a version mismatch among the driver, CUDA, cuDNN, and the ONNX Runtime. Let's go through it with this page.

Symptoms

• You set executionProvider to gpu, but initialization of the GPU execution provider fails.
• The server works by falling back to the CPU, but it does not get GPU acceleration.
• The startup log shows errors loading CUDA/cuDNN-related libraries.

Cause

Bareun GPU inference runs on the GPU execution provider of the ONNX Runtime. The layers of this stack below have to be compatible with each other.

graph TD
  DRV[GPU driver] --> CUDA[CUDA runtime];
  CUDA --> CUDNN[cuDNN];
  CUDNN --> ORT[ONNX Runtime GPU EP];
  ORT --> BAREUN[Bareun inference];

If even one layer's version is off, GPU EP (execution provider) initialization fails, and the CPU fallback occurs as described in When GPU execution falls back to the CPU.

Fix

1. Check the log to see which library failed. What you need to fix depends on whether it is the CUDA runtime or cuDNN. The warning right before the fallback is the clue.

2. Make them compatible in the order driver → CUDA → cuDNN → ONNX Runtime. Check from top to bottom in the diagram above. If the driver is too old, all the layers above it break, so look at the driver first.

3. First confirm that it works normally on the CPU. If you change executionProvider to cpu and the server starts normally, the problem is limited to the GPU stack. This helps narrow down the cause.

   Copy

   {
     "executionProvider": "cpu"
   }
   

4. After making them compatible, switch back to gpu and check the log. If the fallback warning disappears and GPU utilization goes up, it is solved.

Prevention and tips

• When you set up a new GPU node, pinning the stack versions in a document makes it easy to reproduce the same combination in the next deployment.
• For workloads that do not really need a GPU, running on cpu means less troubleshooting burden.

Frequently asked questions

Q. What happens if the CUDA/cuDNN versions do not match?

GPU execution provider initialization fails, and Bareun automatically falls back to the CPU. The server works but does not get GPU acceleration, and a library load error is left in the startup log.

Q. Where should I start checking?

Check from top to bottom in the order GPU driver → CUDA → cuDNN → ONNX Runtime. If the driver is old, all the layers above it break, so check the driver first, and limiting the problem to the GPU stack by running on the CPU makes diagnosis easier.

Q. Will upgrading to the latest version fix it?

Not necessarily. What matters is compatible versions, not the latest. The driver, CUDA, cuDNN, and ONNX Runtime must be matched in a combination verified to work together, and upgrading only one layer arbitrarily can actually break GPU execution provider initialization.

Q. What should I look at in the CUDA error log?

Look at the warning right before the fallback to see which library failed. What you need to fix depends on whether it is the CUDA runtime or cuDNN. Use that clue as the basis for making everything from the driver to the ONNX Runtime compatible.