Bug 37159 - logrotate does not trigger UMC components to reopen their logfiles
logrotate does not trigger UMC components to reopen their logfiles
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 3.2
Other Linux
: P5 normal (vote)
: UCS 3.2-3-errata
Assigned To: Florian Best
Alexander Kramer
:
: 36603 (view as bug list)
Depends on: 36473 38563
Blocks: 37316 37317 38143
  Show dependency treegraph
 
Reported: 2014-12-03 11:50 CET by Alexander Kläser
Modified: 2015-05-19 13:40 CEST (History)
6 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional): Error handling, External feedback, Troubleshooting
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kläser univentionstaff 2014-12-03 11:50:12 CET
In case there is a crash with UMC, this problem leads to empty UMC log files:
* logrotate renames the log files → UMC server process still holds the file handle to the moved files
* after a day, logrotate compresses the log file → output into the old file handles is not written anymore to hard disk

In addition to Bug 36473, UMC server would need to handle the corresponding signal emitted by logrotate. 

+++ This bug was initially created as a clone of Bug #36473 +++

logrotate does not trigger the running applications to reopen their logfiles. Thus they are still logging into already deleted files and the log entries are lost.

root@master:~# lsof | grep .log.1
apache2    1445   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2    1629   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2    5946   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2    5947   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2    6637   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
squid3     8153       root    3u      REG              253,0     18665     399352 /var/log/squid3/cache.log.1 (deleted)
squid3     8156      proxy    3u      REG              253,0     18665     399352 /var/log/squid3/cache.log.1 (deleted)
univentio 11056       root    4u      REG              253,0     37070     400458 /var/log/univention/management-console-web-server.log.1 (deleted)
apache2   11104       root    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2   18150   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2   18157   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2   20148   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2   23107   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
apache2   31323   www-data    3u      REG              253,0      1069     400448 /var/log/univention/management-console-module-lib.log.1 (deleted)
Comment 1 Jan Christoph Ebersbach univentionstaff 2014-12-12 09:15:02 CET
This very problem was also seen at Ticket #2014110221000053 and the corresponding USI.  Somehow the UMC server was not available during exam mode but no further analysis is possible, because UMC's has not been updated for ~4 days.
Comment 2 Florian Best univentionstaff 2014-12-12 17:30:16 CET
Logrotate now sends a SIGHUP (1) to the UMC-web-server and UMC-server process.
UMC-server and web-server triggers univention.debug.reload(). A segfault there has been fixed.

Fix: svn r56747 r56746 r56758 r56761 r56763

2014-12-12-univention-debug.yaml (3.2-3 & 3.2-4)
2014-12-12-univention-management-console-frontend.yaml (only 3.2-3)
2014-12-12-univention-management-console.yaml (3.2-3 & 3.2-4)
Comment 3 Florian Best univentionstaff 2014-12-15 12:10:09 CET
*** Bug 36603 has been marked as a duplicate of this bug. ***
Comment 4 Florian Best univentionstaff 2014-12-15 12:33:57 CET
QA: the following command helps to manually trigger a logrotate: logrotate -f /etc/logrotate.d/univention-management-console
Comment 5 Florian Best univentionstaff 2014-12-15 14:42:58 CET
Some notes for the QA:
It has to be tested what happens when logrotate triggers the rotation of all UMC logfiles. After rotation the logfiles /var/log/univention/management-console-{server,web-server,module-computerroom,module-udm}.log should be existant and contain the content of all log messages which are done after rotation.
The logrotation sends a SIGHUP signal to the umc-webserver umc-server and all running module processes. Please check if all processes still run after a logrotation. Especially the UCS@school computerroom module - as it also uses QT as notifier implementation. And one normal module like UDM.

I removed the cross dependency between the packages. Test what happens if you only upgrade univention-management-console-web-server before the rest of the packages. A invoke-rc.d univention-management-console-web-server restart should still work and a umc login should be possible.
Comment 6 Florian Best univentionstaff 2014-12-15 17:04:21 CET
REOPEN: The logrotate script is not executable and does not exit with 0 (so the last command exitcode is used which is 1 when no UMC module processes exists).
Comment 7 Florian Best univentionstaff 2014-12-15 17:24:01 CET
A test case has been added to ucs-test for UCS 4.0: svn r56851
Comment 8 Florian Best univentionstaff 2014-12-15 17:55:52 CET
(In reply to Florian Best from comment #6)
> REOPEN: The logrotate script is not executable and does not exit with 0 (so
> the last command exitcode is used which is 1 when no UMC module processes
> exists).
This has been fixed in svn r56852 r56853 r56854.
Comment 9 Alexander Kramer univentionstaff 2014-12-16 15:55:15 CET
OK - Adapted yaml 56900
OK - logs after logrotate
OK - process alive
OK - changelog
Comment 10 Moritz Muehlenhoff univentionstaff 2014-12-17 12:54:31 CET
http://errata.univention.de/ucs/3.2/251.html
Comment 11 Moritz Muehlenhoff univentionstaff 2014-12-17 12:54:44 CET
http://errata.univention.de/ucs/3.2/250.html
Comment 12 Moritz Muehlenhoff univentionstaff 2014-12-17 12:55:16 CET
http://errata.univention.de/ucs/3.2/249.html
Comment 13 Janek Walkenhorst univentionstaff 2014-12-17 15:38:37 CET
http://errata.univention.de/ucs/3.2/256.html