OOM Killer is a mechanism for Linux systems to terminate processes when memory is exhausted to prevent crashes. It can protect critical processes from being killed by adjusting the oom_score_adj value, and optimize system-level memory management through overcommit settings. It is scored based on factors such as process memory usage and importance. The higher the score, the easier it is to be killed; to prevent key processes from being killed, the oom_score_adj can be set to -1000; the overcommit_memory policy can be adjusted at the system level, limit memory and swap usage, and monitor OOM events through dmesg. In Kubernetes or Docker environments, you need to pay attention to the interaction effect of resource quotas and cgroups.

When the Linux system is insufficient, a mechanism called OOM Killer (Out-of-Memory Killer) will be enabled. Its role is to selectively terminate certain processes to free up memory resources and prevent the entire system from crashing. While this mechanism can serve as the "last line of defense", the default behavior is not always the most reasonable - sometimes it may kill processes you don't want to be killed.

If you are running a high-load service, a container environment, or have specific processes that need protection, it is necessary to understand and properly adjust the behavior of OOM Killer.

What is OOM Killer?

OOM Killer is part of the Linux kernel and is triggered when the system can no longer allocate memory (including swap space). It will score each process according to a set of scoring mechanisms. The higher the score, the easier it is to be selected and killed.

This set of ratings mainly consider several factors:

• The memory size used by the process
• The importance of the process (such as whether it is started by root)
• Is it running for a long time
• OOM score adj value (I'll talk about it later)

You can view and modify the adjustment value of a process in /proc/<pid>/oom_score_adj</pid> , and control the overall memory allocation strategy in /proc/sys/vm/overcommit_memory , etc.

How to prevent key processes from being killed by OOM Killer?

If you have a service that cannot be interrupted easily (such as the database main process), you can reduce the probability of being killed by adjusting its OOM score.

The method is as follows:

• Modify the oom_score_adj value of the process, the range is -1000 to 1000. The lower the value, the less likely it is to be killed.
• For example, if you want to protect a process with PID 1234, you can execute:

   echo -500 > /proc/1234/oom_score_adj

• If you want to completely ban OOM Killer from killing this process, you can set it to -1000:

   echo -1000 > /proc/1234/oom_score_adj

?? Note: Only root permissions can modify these values. In addition, do not set all processes to -1000 casually, as this may cause the system to be killed without processes and get stuck.

How to tune system-level OOM behavior?

In addition to a single process, some adjustments can be made from the system level to affect the triggering timing and behavior of OOM Killer:

• Adjust overcommit settings
  /proc/sys/vm/overcommit_memory controls memory allocation policy:

  • 0: Default mode, allowing moderate memory over-allocation.
  • 1: Always allow excessive distribution, suitable for testing environments.
  • 2: Excessive distribution is prohibited, it is more conservative and suitable for production environments.

• Limit the maximum amount of available swaps
  Setting /proc/sys/vm/overcommit_ratio and swappiness can indirectly affect the memory pressure performance of the system before OOM occurs.

• Monitoring system log
  When OOM Killer is actually triggered, the system will record relevant information in dmesg or syslog. You can check whether any process has been killed by:

   dmesg | grep -i 'oom\|kill'

Practical application suggestions

• In Kubernetes or Docker environments, the behavior of OOM Killer may interact with the memory limit of cgroups. At this time, pay special attention to the setting of resource quota.
• For development environments, you can temporarily close OOM Killer for debugging (not recommended for long-term use):

   echo 1 > /proc/sys/vm/panic_on_oom

  The system will crash directly in OOM instead of killing the process, which is suitable for certain test scenarios.

• Regularly check the oom_score_adj values ??of critical services to make sure they have not returned to the default values ??due to restarts or other reasons.

Basically that's it. OOM Killer is a useful guarantee mechanism, but in actual deployment, it often requires certain adjustments based on business needs. Understanding its working principle and configuration will help improve the stability and controllability of the system.

The above is the detailed content of What is the Out-of-Memory (OOM) Killer and how to tune it?. For more information, please follow other related articles on the PHP Chinese website!