Univention Bugzilla – Bug 48288
ryslogd consumes plenty of memory
Last modified: 2020-07-03 20:52:55 CEST
In a customer environment 'rsyslogd' consumes 55% of memory and raising. Info about the environment: UCS 4.2-3 rsyslogd 8.4.2, compiled with: FEATURE_REGEXP: Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: Yes memory allocator: system default Runtime Instrumentation (slow code): No uuid support: Yes Number of Bits in RainerScript integers: 64 See Also: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794848
Additional Infos: root@ucs:~# free -m total used free shared buffers cached Mem: 4329 3879 450 19 281 420 -/+ buffers/cache: 3177 1152 Swap: 2048 2048 0 root@ucs:~# top top - 08:59:53 up 135 days, 3:02, 1 user, load average: 1.86, 1.81, 1.86 Tasks: 151 total, 5 running, 146 sleeping, 0 stopped, 0 zombie %Cpu(s): 18.4 us, 29.3 sy, 0.0 ni, 52.2 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 4433548 total, 3973876 used, 459672 free, 288140 buffers KiB Swap: 2098172 total, 2098172 used, 0 free. 431088 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5197 root 20 0 4650356 2.359g 1792 S 0.0 55.8 134:11.82 rsyslogd 190 root 20 0 316036 149136 146768 S 0.0 3.4 1214:35 systemd-journal 20468 root 20 0 583328 89748 55192 S 0.0 2.0 2:26.88 apache2 === syslog === Nov 29 09:17:20 ucs rsyslogd-2177: imuxsock[pid 5217]: begin to drop messages due to rate-limiting Nov 29 09:17:23 ucs rsyslogd-2177: imuxsock[pid 5217]: 165 messages lost due to rate-limiting Nov 29 09:17:27 ucs rsyslogd-2177: imuxsock[pid 5217]: begin to drop messages due to rate-limiting Nov 29 09:17:29 ucs rsyslogd-2177: imuxsock[pid 5217]: 137 messages lost due to rate-limiting Nov 29 09:17:32 ucs rsyslogd-2177: imuxsock[pid 5217]: begin to drop messages due to rate-limiting Nov 29 09:17:35 ucs rsyslogd-2177: imuxsock[pid 5217]: 173 messages lost due to rate-limiting Nov 29 09:17:38 ucs rsyslogd-2177: imuxsock[pid 5217]: begin to drop messages due to rate-limiting Nov 29 09:17:41 ucs rsyslogd-2177: imuxsock[pid 5217]: 137 messages lost due to rate-limiting Nov 29 09:17:45 ucs rsyslogd-2177: imuxsock[pid 5217]: begin to drop messages due to rate-limiting Nov 29 09:17:47 ucs rsyslogd-2177: imuxsock[pid 5217]: 104 messages lost due to rate-limiting Nov 29 09:17:50 ucs rsyslogd-2177: imuxsock[pid 5217]: begin to drop messages due to rate-limiting Nov 29 09:17:53 ucs rsyslogd-2177: imuxsock[pid 5217]: 119 messages lost due to rate-limiting Nov 29 09:17:57 ucs rsyslogd-2177: imuxsock[pid 5217]: begin to drop messages due to rate-limiting Nov 29 09:17:59 ucs rsyslogd-2177: imuxsock[pid 5217]: 132 messages lost due to rate-limiting
The bug report you referenced states that the issue was that a remote server configured to receive logs was not available, so the messages were cached locally. Is that the case here?
https://www.usenix.org/system/files/login/articles/06_lang.pdf
Since I was reading into rsyslog config for another project, I'll leave some links here that might be useful to understand configuration of rsyslog queues, blocking, Memory- and Disk-Assisted queues: https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html https://www.rsyslog.com/doc/v8-stable/concepts/queues.html https://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html https://www.rsyslog.com/doc/master/whitepapers/queues_analogy.html
(In reply to Erik Damrose from comment #2) > The bug report you referenced states that the issue was that a remote server > configured to receive logs was not available, so the messages were cached > locally. Is that the case here? absolutely, every now and than the remote host is not reachable so I assume the message cache can't be entirely processed. That's probably why the cache consumes more and more memory until nothing is left. A configurable Threshold would be nice.
Our UCR template includes /etc/rsyslog.d/*.conf before defining the actions for remote logging (UCR syslog/remote), so I guess you could set the MainMsgQueueSize and similar parameters there, see: https://www.rsyslog.com/doc/master/configuration/global/options/rsconf1_mainmsgqueuesize.html Other parameters may need adjustment, too, to tell rsyslog what to do with incoming messages once the queue is full.
So what 'more info' is needed?
E.g. if the customer reporting this issue had * special UCR variables (I can only guess that syslog/remote was enabled, but we need to know for sure) * special includes in /etc/rsyslog.d/*.conf
*** Bug 48522 has been marked as a duplicate of this bug. ***
(In reply to Arvid Requate from comment #8) > E.g. if the customer reporting this issue had > > * special UCR variables (I can only guess that syslog/remote was enabled, > but we need to know for sure) syslog/remote='@@192.168.1.2' > > * special includes in /etc/rsyslog.d/*.conf etc/rsyslog.d/postfix.conf: # Create an additional socket in postfix's chroot in order not to break # mail logging when rsyslog is restarted. If the directory is missing, # rsyslog will silently skip creating the socket. $AddUnixListenSocket /var/spool/postfix/dev/log --- Quite apart from that, wouldn't it be good to integrate a more recent version into UCS 4.2 (backport) that doesn't have the problem?
My understanding is that this occures only if a remote syslog server is configured and the remote servers has been down for a while, and it's the intended behavioud or the default rsyslogd configuration found in any other distro as well, and workarounds are documented. So this is a potential enhancement, not a bug.
this is all about a backport of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794848 because the handling has changed in a newer version of rsyslog.
Hmm, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794848#29 says that's a configuration issue, and there is no patch to backport?
This issue has been filed against UCS 4.2. UCS 4.2 is out of maintenance and many UCS components have changed in later releases. Thus, this issue is now being closed. If this issue still occurs in newer UCS versions, please use "Clone this bug" or reopen it and update the UCS version. In this case please provide detailed information on how this issue is affecting you.