Bug 54139 - univention-directory-notifier service can reach configured systemd start limit during quick subsequent imports
univention-directory-notifier service can reach configured systemd start limi...
Status: NEW
Product: UCS@school
Classification: Unclassified
Component: Import scripts
UCS@school 5.0
Other Linux
: P5 normal (vote)
: ---
Assigned To: UCS@school maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2021-11-24 17:51 CET by Jürn Brodersen
Modified: 2021-11-25 12:48 CET (History)
0 users

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jürn Brodersen univentionstaff 2021-11-24 17:51:02 CET
univention-directory-notifier service reaches configured systemd start limit during import tests

The univention-directory-notifier service has a restart burst limit of 50 starts within 1000 sec. This limit is sometimes reached during our jenkins tests. If the limit is reached the services will not restart automatically and needs to be started manually after a cool down time.
This results in effectively hanging tests because most tests are waiting for the replication, which isn't happening because the notifier is stopped.
Comment 1 Jürn Brodersen univentionstaff 2021-11-25 12:46:47 CET
If you see the following error while trying to start the univention-directory-notifier you probably have this issue:

Nov 25 12:29:29 school systemd[1]: univention-directory-notifier.service: Start request repeated too quickly.
Nov 25 12:29:29 school systemd[1]: univention-directory-notifier.service: Failed with result 'start-limit-hit'.
Nov 25 12:29:29 school systemd[1]: Failed to start Univention Directory Notifier Daemon.


A quick fix is calling:
systemctl reset-failed univention-directory-notifier.service

A more permanent solution is overwriting the StartLimitBurst value for the service:
mkdir -p /etc/systemd/system/univention-directory-notifier.service.d/
printf "[Unit]\nStartLimitBurst=5000\n" > /etc/systemd/system/univention-directory-notifier.service.d/10-IncreaseStartLimitBurst.conf
systemctl daemon-reload

Please note that the value is already quiet high and restarting the notifier that often could slow down the replication process significantly.

In the long run we might want to look into changing the importer in a way that we don't have to stop the notifier.
Comment 2 Jürn Brodersen univentionstaff 2021-11-25 12:48:01 CET
I added a workaround to our tests:
[5.0 29011a1ab] Bug #54139: reset notifier restart burst limit for each test