Tracking down the cause of Docker container OOM kills
Did you start Bareun with Docker and the container dies from an OOM kill? Most of the time it is because the container memory limit is set lower than the amount needed to load the models and dictionaries. Let's track down the cause.
Symptoms
- The container exits from OOM (Out Of Memory) during startup or right after startup.
dockerstatus showsOOMKilled.- It happens more often with images that include the spell check feature.
Cause
Bareun loads models and dictionaries into memory ahead of time at startup. If the container memory limit is lower than this loading amount, the kernel forcibly kills the process (OOM kill) during loading.
| Cause | Description |
|---|---|
| Memory limit too low | The container --memory limit is too low |
| Loading correction data | The spell check image also loads word-freq.gob and the BK-tree (.fb dictionaries are mmap'd) |
| Concurrent load | Competing for memory with other containers |
The spell check image in particular loads more dictionaries than the morphological-analysis-only image, so it needs more memory. For details, see Finding the cause of rising memory usage.
Solution
-
Set a generous memory limit. The container memory limit must be larger than the amount needed to load models and dictionaries. After raising the limit, check that it starts.
docker run --memory 4g --memory-swap 4g bareunai/bareunAdjust the numbers to your environment
The
4gabove is an example. The memory needed differs depending on whether the image you use is morphological-analysis-only or a correction build. Look at the actual memory usage at startup and leave some headroom. -
Check the image type. If you only need morphological analysis, an image that does not load the correction dictionaries uses less memory. Check whether you really need the correction feature.
-
Use the startup log to see where it dies. Checking whether it ends during model loading or dictionary loading helps you estimate the memory needed.
-
Check host resources. If several containers compete for memory on one host, OOM happens. Look at the host's total available memory and other workloads too.
Prevention and tips
- Before putting it into production, start the image you will actually use once, measure the memory usage at the stable point, and set the limit with headroom above it.
- A big jump in memory right after startup is normal behavior from loading models and dictionaries ahead of time, so set the limit so it can withstand that peak.
Frequently asked questions
Q. My container dies from an OOM kill. What is the most common cause?
The container memory limit is set lower than the amount needed to load models and dictionaries. Bareun loads models and dictionaries into memory ahead of time at startup, so you need to raise the limit enough to withstand that peak.
Q. Why does OOM happen more often on the correction image?
The spell check image also loads word frequencies (word-freq.gob) and the BK-tree for similar-word search, so it needs more memory than the morphological-analysis-only image.
Q. How much should I set --memory to?
There is no fixed value, because the memory needed differs depending on whether it is morphological-analysis-only or a correction build.
Start the image you will actually use once, measure the memory usage at the stable point, and leave enough headroom to withstand the loading peak right after startup. The 4g in the example is only a starting point, not a value to use as is.
Q. How can I tell at which stage it dies?
The startup log shows whether it is in the middle of loading models or loading dictionaries. Knowing where it ends helps you estimate the memory needed. Several containers may also compete for memory on one host, so check the host's total available memory too.