The auditd events cause ‘backlog limit exceeded’ messages¶
If auditd generates a lot of events, some of them may be lost with the following numerous messages in dmesg or kernel logs:
auditd: backlog limit exceeded
You may also observe high or growing values of the lost
counter in the
auditctl output. For example:
auditctl -s
...
lost 1351280
...
To resolve the issue, you may need to update the rules loaded to auditd and adjust the size of the backlog buffer.
Update the rules loaded to auditd¶
If auditd contains a lot of rules, it may generate a lot of events and overrun
the buffer. Therefore, verify and update your preset and custom rules. Preset
rules are defined as presetRules
, custom rules are defined as follows:
customRules
customRulesX32
customRulesX64
To verify and update the rules:
In the
Cluster
object of the affected cluster, verify that thepresetRules
string does not start with the!
symbol.Verify all audit rules:
Log in through SSH or directly using the console to the node having the buffer overrun symptoms.
Run the following command:
auditctl -l
In the system response, identify the rules to exclude.
In
/etc/audit/rules.d
, find the files containing the rules to exclude.If the file is named
60-custom.rules
, remove the rules from any of the following parameters located in theCluster
object:customRules
customRulesX32
customRulesX64
If the file is named
50-<NAME>.rules
, and you want to exclude all rules from that file, exclude the preset named<NAME>
from the list of allowed presets defined underpresetRules
in theCluster
object.If the file is named
50-<NAME>.rules
, and you want to exclude only several rules from that file:Copy the rules you want to keep to one of the following parameters located in the
Cluster
object:customRules
customRulesX32
customRulesX64
Exclude the preset named
<NAME>
from the list of allowed presets.
Adjust the size of the backlog buffer¶
By default, the backlog buffer size is set to 8192, which is enough for most use cases. To prevent buffer overrun, you can adjust the default value to fit your needs. But keep in mind that increasing this value leads to higher memory requirements because the buffer uses RAM.
To estimate RAM requirements for the buffer, you can use the following calculation:
A buffer of 8192 audit records uses ~70 MiB of RAM
A buffer of 15000 audit records uses ~128 MiB of RAM
To change the backlog buffer size, adjust the backlogLimit
value in the
Cluster
object of the affected cluster.
You may also want to change the size directly on the system and verify
the result at once. But to change the size permanently, use the Cluster
object.
To adjust the size of the backlog buffer on a node:
Log in to the affected node through SSH or directly through the console.
If
enabledAtBoot
is enabled, adjust theaudit_backlog_limit
value in kernel options:List grub configuration files where
GRUB_CMDLINE_LINUX
is defined:grep -rn 'GRUB_CMDLINE_LINUX' /etc/default/grub /etc/default/grub.d/* \ | cut -d: -f1 | sort -u
In each file obtained in the previous step, edit the
GRUB_CMDLINE_LINUX
string by changing the integer value afteraudit_backlog_limit=
to the desired value.
In
/etc/audit/rules.d/audit.rules
, adjust the buffer size by editing the integer value after-b
.Select from the following options:
If the auditd configuration is not immutable, restart the
auditd
service:systemctl restart auditd.service
If the auditd configuration is immutable, reboot the node. The auditd configuration is immutable if any of the following conditions are met:
In the auditctl -s output, the
enabled
parameter is set to2
The
-e 2
flag is defined explicitly in parameters of any custom ruleThe
immutable
preset is defined explicitlyThe virtual preset
all
is enabled and theimmutable
preset is not excluded explicitly
Caution
Arrange the time to reboot the node according to your maintenance schedule. For the exact reboot procedure, use your maintenance policies.
If the
backlog limit exceeded
message disappears, adjust the size permanently using thebacklogLimit
value in theCluster
object.