Univention Bugzilla – Bug 43685
Convert univention-debug to journald (or rsyslog)
Last modified: 2020-02-10 09:06:25 CET
Currently most UCS services using univention-debug log to files in /var/log/univention/. This does not work well with systemd using journald or with containers, as the information is stored in multiple locations which are hard to aggregate and monitor.
For example when a UCS service is started via runit, the shell script /etc/runit/univention-$foo/run opens /var/log/univention/$foo.log and redirect STDOUT/STDERR to that file. As systemd tries to get rid of shell scripts, the redirection would require an intermediate shell script or the logic of wringing log files to be moved into the daemon itself.
With containers the current best practice is to just log messages to STDOUT/STDERR and to leave it to the invoking environment to handle the output:
- systemd will collect the output it its journal
- docker will collect it somewhere else
Another option would be to convert them to use syslog, which would allow us to forward messages easily to a central logging server (Bug #15728), which is especially useful wen debugging concurrency problems like with UDL, UDN, OpenLDAP. Using an intermediate service like rsyslog also simplifies rotating log files, as only the intermediate service needs to handle the close/reopen cycle - currently most UCS services need to be restarted, which interrupts any currently running service. Some services even miss that:
/etc/runit/univention-dhcp/run: exec /usr/sbin/dhcpd -q -f >>/var/log/daemon.log
Other options would be to use <https://www.elastic.co/de/products/logstash> or equivalent, which would allow searching.
There are also lots of request to allow changing debug levels without restarting the service: Bug #43445, Bug #35707, Bug #43515
- Documentation would need to be updated
- Radical change for old admins where to look for information
- Major change to most packages
Pro/con journald: <https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs>
- Allows (local) searching
- Can forward to syslog
- Would simplify conversion to systemd as it would do the right thing out-of-the-box
- Unclear of how to handle legacy univention-debug categories (MESSAGE_ID?) and levels (PRIORITY) - as journald was designed to better handle structured data, it should be easier as mapping to syslog.
- There even is a Python3 binding: <https://www.freedesktop.org/software/systemd/python-systemd/journal.html>, which can be connected to Pythons logging API.
- no remote support yet.
- syslog only has a limited number of "facilities" (LOCAL0-7) for free use - its unclear is they should be used for different services - univention-debug currently has 22 categories and 5 levels
- python-logging has a syslog adapter: <https://docs.python.org/2/library/logging.handlers.html#sysloghandler>
I had a longer discussion on logs filling up the disk during a technical training today. One of the attendees reported that they increased the debug level of a service over several hours. Because of this, the disk filled up and some databases were damaged (MySQL and Listener-Cache at least).
IMHO the fact that this can happen at all is a waekness of the time-based logrotate mechanism of rsyslog/sysvinit. Therefore I want to highlight one of the pro journald points mentioned in the Google Doc that Philipp linked to:
→ "journald automatically rotates journal files if they grow above certain limits. This is built right into the disk space allocation logic, in order to avoid vulnerability windows due to purely time-based rotation. Rotation not only takes a maximum disk usage limit into account, but also monitors general disk usage levels in order to ensure that a certain amount of space is always kept free on disk."
Customer is in process of getting certifiedfor PCI-DSS (see https://de.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard) and thus seems to need a centralized log server and log notices in proper syslog format.
The separated log files for UCS specific components are a major pain as no central logging is configurable.
PS: journald is also a pain as no service specific log retention can be configured. You only can purge all entries older than X days.
Dearly requested again by a customer.
There seems to be environments where a central logging is much needed.
Requested by a key customer.
adding another customer ID where centralized logging was requested too