Bug 45960 - Listener hangs due to saml restart in UCS 4.3 setup
Listener hangs due to saml restart in UCS 4.3 setup
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: SAML
UCS 4.3
Other Linux
: P5 normal (vote)
: UCS 4.3
Assigned To: Stefan Gohmann
Jürn Brodersen
: interim-1
: 43486 45925 (view as bug list)
Depends on:
Blocks: 46212
  Show dependency treegraph
 
Reported: 2017-12-28 07:25 CET by Stefan Gohmann
Modified: 2018-03-14 14:38 CET (History)
2 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
autotest-091-master-s4.log.bz2 (1.29 MB, application/x-bzip)
2017-12-28 07:29 CET, Stefan Gohmann
Details
journalctl.lst (659.27 KB, text/plain)
2017-12-28 07:29 CET, Stefan Gohmann
Details
syslog (474.98 KB, text/plain)
2017-12-28 07:29 CET, Stefan Gohmann
Details
daemon.log (452.03 KB, text/plain)
2017-12-28 07:30 CET, Stefan Gohmann
Details
system-stats.log (1.38 MB, text/plain)
2017-12-28 07:30 CET, Stefan Gohmann
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2017-12-28 07:25:49 CET
The listener gets stuck during UCS 4.3 Amazon EC2 setup:

root     16487  0.0  0.0   4060   616 ?        Ss   01:08   0:00  \_ runsv univention-directory-listener
root     23083 12.6  1.4 2716512 56796 ?       S    01:21   0:05  |   \_ /usr/sbin/univention-directory-listener -F -d 4 -b dc=AutoTest091,dc=local -m /usr/lib/univention-directory-listener/system -c /var/lib/un
root     23138  0.0  0.0   4288  1596 ?        S    01:21   0:00  |       \_ /bin/sh /usr/sbin/invoke-rc.d univention-saml restart
root     23147  0.0  0.0  37948  2152 ?        S    01:21   0:00  |           \_ systemctl restart univention-saml.service


It looks like the restart doesn't return:
root     23100  0.0  0.0  13028  3392 ?        Ss   01:21   0:00 /bin/bash /etc/init.d/univention-saml stop
root     23121  0.0  0.0   4288  1628 ?        S    01:21   0:00  \_ /bin/sh -e /etc/init.d/stunnel4 stop
root     23127  0.0  0.0  41128  2200 ?        S    01:21   0:00      \_ /bin/systemctl --no-pager stop stunnel4.service


I've changed the listener debug to 4 and set system stats to run every minute.
Comment 1 Stefan Gohmann univentionstaff 2017-12-28 07:29:08 CET
Created attachment 9331 [details]
autotest-091-master-s4.log.bz2
Comment 2 Stefan Gohmann univentionstaff 2017-12-28 07:29:24 CET
Created attachment 9332 [details]
journalctl.lst
Comment 3 Stefan Gohmann univentionstaff 2017-12-28 07:29:43 CET
Created attachment 9333 [details]
syslog
Comment 4 Stefan Gohmann univentionstaff 2017-12-28 07:30:01 CET
Created attachment 9334 [details]
daemon.log
Comment 5 Stefan Gohmann univentionstaff 2017-12-28 07:30:19 CET
Created attachment 9335 [details]
system-stats.log
Comment 6 Stefan Gohmann univentionstaff 2017-12-28 07:47:57 CET
It is reproducible in this way:

root@master093:~# /etc/init.d/memcached start univention_saml
[ ok ] Starting memcached (via systemctl): memcached.service.
root@master093:~# /etc/init.d/stunnel4 start
[ ok ] Starting stunnel4 (via systemctl): stunnel4.service.
root@master093:~# /etc/init.d/memcached stop univention_saml
[ ok ] Stopping memcached (via systemctl): memcached.service.
root@master093:~# /etc/init.d/stunnel4 stop
[ ok ] Stopping stunnel4 (via systemctl): stunnel4.service.
root@master093:~# /etc/init.d/memcached start univention_saml
[ ok ] Starting memcached (via systemctl): memcached.service.
root@master093:~# /etc/init.d/stunnel4 start
[ ok ] Starting stunnel4 (via systemctl): stunnel4.service.
root@master093:~# /etc/init.d/univention-saml stop
[....] Stopping univention-saml (via systemctl): univention-saml.service

→ hangs
Comment 7 Stefan Gohmann univentionstaff 2017-12-28 15:12:20 CET
Looks like Bug #43486 is one problem.
Comment 8 Stefan Gohmann univentionstaff 2017-12-29 15:01:26 CET
*** Bug 43486 has been marked as a duplicate of this bug. ***
Comment 9 Stefan Gohmann univentionstaff 2017-12-29 15:17:44 CET
I've added a systemd script for univention_saml memcache:

https://git.knut.univention.de/univention/ucs/commit/f5b4726b78e725cab2d2ab404a5be2cf390baecf

The stunnel init script now also creates the piddir /var/run/univention-saml. SVN r17918 - r17923.

Changelog:
https://git.knut.univention.de/univention/ucs/commit/fc07c302286df558da7d291e8fac2e906f4becd8
Comment 10 Jürn Brodersen univentionstaff 2018-01-05 12:35:06 CET
Is there a reason you didn't delete the init file?
The service file replaces the init file. If possible please delete it or replace it by a file that just calls:
'''
# Inject systemd
. /lib/lsb/init-functions
'''
to make that more clear :)
Afaik systemd stops processing an init file if a service file under the same name exists and starts the service instead.
Comment 11 Stefan Gohmann univentionstaff 2018-01-14 19:28:19 CET
(In reply to Jürn Brodersen from comment #10)
> Is there a reason you didn't delete the init file?

Yes, I saw code like 'test -x /etc/init.d/unviention-saml && service restart univention-saml'

> The service file replaces the init file. If possible please delete it or
> replace it by a file that just calls:
> '''
> # Inject systemd
> . /lib/lsb/init-functions
> '''
> to make that more clear :)
> Afaik systemd stops processing an init file if a service file under the same
> name exists and starts the service instead.

Yes, I removed the code from the univention saml init file:

https://git.knut.univention.de/univention/ucs/commit/5419472c08fa6703e2d97138f4e3a33782d6204a
Comment 12 Stefan Gohmann univentionstaff 2018-01-14 19:37:20 CET
*** Bug 45925 has been marked as a duplicate of this bug. ***
Comment 13 Jürn Brodersen univentionstaff 2018-01-17 17:17:12 CET
Looks good now :)

What I tested:
Installation -> OK
systemctl status univention-saml -> OK
systemctl status univention-directory-listener -> OK

-> Verified
Comment 14 Stefan Gohmann univentionstaff 2018-03-14 14:38:36 CET
UCS 4.3 has been released:
 https://docs.software-univention.de/release-notes-4.3-0-en.html
 https://docs.software-univention.de/release-notes-4.3-0-de.html

If this error occurs again, please use "Clone This Bug".