Bug 43685 - Convert univention-debug to journald (or rsyslog)
Convert univention-debug to journald (or rsyslog)
Status: NEW
Product: UCS
Classification: Unclassified
Component: General
UCS 4.4
Other Linux
: P5 normal with 6 votes (vote)
: ---
Assigned To: UCS maintainers
UCS maintainers
: systemd
Depends on:
Blocks: 43330 43686 43687 43688 43689 43872
  Show dependency treegraph
 
Reported: 2017-03-01 13:31 CET by Philipp Hahn
Modified: 2022-06-24 13:55 CEST (History)
11 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?: Yes
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2018071821000221, 2019081321000753
Bug group (optional): API change, Debt Technical
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2017-03-01 13:31:16 CET
Currently most UCS services using univention-debug log to files in /var/log/univention/. This does not work well with systemd using journald or with containers, as the information is stored in multiple locations which are hard to aggregate and monitor.

For example when a UCS service is started via runit, the shell script /etc/runit/univention-$foo/run opens /var/log/univention/$foo.log and redirect STDOUT/STDERR to that file. As systemd tries to get rid of shell scripts, the redirection would require an intermediate shell script or the logic of wringing log files to be moved into the daemon itself.

With containers the current best practice is to just log messages to STDOUT/STDERR and to leave it to the invoking environment to handle the output:
- systemd will collect the output it its journal
- docker will collect it somewhere else
- ...

Another option would be to convert them to use syslog, which would allow us to forward messages easily to a central logging server (Bug #15728), which is especially useful wen debugging concurrency problems like with UDL, UDN, OpenLDAP. Using an intermediate service like rsyslog also simplifies rotating log files, as only the intermediate service needs to handle the close/reopen cycle - currently most UCS services need to be restarted, which interrupts any currently running service. Some services even miss that:
 /etc/runit/univention-dhcp/run: exec /usr/sbin/dhcpd -q -f >>/var/log/daemon.log

Other options would be to use <https://www.elastic.co/de/products/logstash> or equivalent, which would allow searching.

There are also lots of request to allow changing debug levels without restarting the service: Bug #43445, Bug #35707, Bug #43515

Impact:
- Documentation would need to be updated
- Radical change for old admins where to look for information
- Major change to most packages

Pro/con journald: <https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs>
- Allows (local) searching
- Can forward to syslog
- Would simplify conversion to systemd as it would do the right thing out-of-the-box
- Unclear of how to handle legacy univention-debug categories (MESSAGE_ID?) and levels (PRIORITY) - as journald was designed to better handle structured data, it should be easier as mapping to syslog.
- There even is a Python3 binding: <https://www.freedesktop.org/software/systemd/python-systemd/journal.html>, which can be connected to Pythons logging API.
- no remote support yet.

Pro/con syslog:
- syslog only has a limited number of "facilities" (LOCAL0-7) for free use - its unclear is they should be used for different services - univention-debug currently has 22 categories and 5 levels
- python-logging has a syslog adapter: <https://docs.python.org/2/library/logging.handlers.html#sysloghandler>
Comment 1 Michael Grandjean univentionstaff 2017-04-25 21:34:41 CEST
I had a longer discussion on logs filling up the disk during a technical training today. One of the attendees reported that they increased the debug level of a service over several hours. Because of this, the disk filled up and some databases were damaged (MySQL and Listener-Cache at least).
IMHO the fact that this can happen at all is a waekness of the time-based logrotate mechanism of rsyslog/sysvinit. Therefore I want to highlight one of the pro journald points mentioned in the Google Doc that Philipp linked to:

→ "journald automatically rotates journal files if they grow above certain limits. This is built right into the disk space allocation logic, in order to avoid vulnerability windows due to purely time-based rotation. Rotation not only takes a maximum disk usage limit into account, but also monitors general disk usage levels in order to ensure that a certain amount of space is always kept free on disk."
Comment 2 Christian Völker univentionstaff 2018-07-24 10:41:55 CEST
Customer is in process of getting certifiedfor PCI-DSS (see https://de.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard) and thus seems to need a centralized log server and log notices in proper syslog format.
Comment 3 Philipp Hahn univentionstaff 2019-03-18 13:17:00 CET
The separated log files for UCS specific components are a major pain as no central logging is configurable.

PS: journald is also a pain as no service specific log retention can be configured. You only can purge all entries older than X days.
Comment 4 Nico Stöckigt univentionstaff 2019-08-14 12:47:36 CEST
Dearly requested again by a customer.

There seems to be environments where a central logging is much needed.
Comment 5 Benjamin Fels univentionstaff 2019-10-01 10:33:45 CEST
Requested by a key customer.
Comment 6 Dirk Ahrnke univentionstaff 2019-12-19 13:53:28 CET
adding another customer ID where centralized logging was requested too