Description Christian Völker 2019-09-06 15:12:05 CEST

It happens frequently the filesystem /var getting full

Logging itself is important and can be configured for rsyslogd through UCRV and works together with logrotate. Unfortunately, logging is done in a double way with journald as part of systemd does logging, too.

The configurable limits usually do not really help as journald takes only the widest limitation into account (which usually is to keep around 4GB free and fill the rest with journald-logs).

But frequently 4GB is not enough for a busy slave to get logrotate clean them up on a daily base. So the filesystem gets full, because journald only leaves 4GB of free space.

Citing from man journald:
"If the file system is nearly full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually free. This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up, journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either."


Overall it is quite confusing and customers have several gigabytes of journald aging back two or three years.

I would suggest to configure journald to use only "volatile" logging under /var/run.
There should be no loss in case of server breakdown because rsyslog is still logging persistently to disk /var/log.