Bug 56076 - After a logrotate, the progress of listener module initialization cannot be monitored in the existing logfiles
After a logrotate, the progress of listener module initialization cannot be m...
Status: NEW
Product: UCS
Classification: Unclassified
Component: Listener (univention-directory-listener)
UCS 5.0
Other Linux
: P5 normal (vote)
: ---
Assigned To: UCS maintainers
UCS maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2023-05-17 18:00 CEST by Erik Damrose
Modified: 2023-05-26 09:24 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 4: Minor Usability: Impairs usability in secondary scenarios
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.046
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2023050821000359
Bug group (optional): bitesize
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Erik Damrose univentionstaff 2023-05-17 18:00:20 CEST
Initializing or reinitializing a listener module in a domain with many affected objects can take a long time, because all matching objects are fetched from LDAP and passed to the module. The duration can also be longer if an external service is contacted by the listener module, e.g. in the MS365 connector.

Problem: If a logrotate happens during that time, the listener module progess cannot be followed in the listener.log, there are no new entries in the file. It seems like the listener is hanging, although it is still processing entries. One might suspect something is wrong and restart the listener, which will only restart the module initialization from the start.

The reason is, that signal processing is deactivated while initializing the listener. The listener also currently handles the initialization as one atomic event and will start from scratch after it is restarted, which is what the logrotate config is trying to do.

management/univention-directory-listener/src/main.c has this in main():

/* XXX: we shouldn't block all signals for so long */
signals_block();
[...]
handlers_init();
[...]
signals_unblock();

After the logrotate, you can see the listener process logging into a deleted file:
$ lsof -p <listener-pid>
univentio 2047 listener    1w   REG  253,0    27130 1309466 /var/log/univention/listener.log.1 (deleted)
univentio 2047 listener    2w   REG  253,0    27130 1309466 /var/log/univention/listener.log.1 (deleted)
Comment 1 Philipp Hahn univentionstaff 2023-05-18 06:33:05 CEST
Use `less /proc/2047/fd/1` to continue reading the deleted file.

We should add "delaycompress" to most of our logrotate configuration files to delay compressing for one cycle. That way processes continue to log to the previous moved file until they are restarted.