Description ageukes 2023-12-07 21:26:16 CET

Critical Bug Report: Univention Management Console Server Fails to Release File Handles After Log Rotation

Description:
The customer has identified a critical issue with the Univention Management Console (UMC) Server on the Primary DC. This issue has been observed twice in the last few weeks, where the UMC Server retains hundreds of open file handles for already rotated log files, causing the /var/log directory to fill up.


Reproduction Steps:
Execute the following command to identify the number of open file handles for deleted log files:
root@nayru:~# lsof 2>/dev/null | grep '/var/log/univention/management.*deleted' | wc -l
The result shows 862 open file handles.

Check the Multithreading configuration of the UMC Server:
root@nayru:~# ucr get umc/server/processes
The Multithreading configuration is set to 15.

The customer suspects that the issue may be related to the Multithreading configuration of the UMC, which is not active on any other UCS system.
The customer suggests checking if a similar result can be obtained with a Multithreaded UMC-Server after log rotation. After restarting the UMC-Server, the customer observed a release of 27GB of memory.


Issue Details:
The problem seems to be associated with the Multithreading configuration of the UMC Server.
The log file /var/log/univention/management-console-server.log.1 appears to be specifically affected.
Despite log rotation, the UMC Server retains file handles on already rotated (compressed) log files and continues logging into them until the UMC process is terminated.
SIGHUP signals sent to the UMC Server after log rotation do not seem to take effect.


Reproduction on another System:
The issue has been successfully reproduced on another system with the following steps:
1. Set Multithreading configuration:
ucr set umc/server/processes=3 umc/http/processes=3

2. Restart the UMC Service:
service univention-management-console-server restart

3. Force log rotation:
logrotate -f /etc/logrotate.d/univention-management-console

4. Check for open file handles:
lsof 2>/dev/null | grep '/var/log/univention/management.*deleted'

5. The result shows open file handles for the deleted log file /var/log/univention/management-console-server.log.1.


Impact:
The UMC Server continues writing to a deleted and growing file until the next restart.
Logs written during this period are lost permanently.
The customer experienced a loss of all logs in the specified log file since mid-October after a UMC restart on the Primary DC.


Recommendation:
We recommend investigating and addressing the issue with the Multithreading configuration and the failure of SIGHUP signals to release file handles after log rotation. This issue poses a significant risk of data loss and impacts the overall stability and reliability of the Univention Management Console Server.