Univention Bugzilla – Bug 57089
USS in OpenStack image sleeps 10m - FD pipe leak in udm-cli-server
Last modified: 2024-03-07 13:07:37 CET
I am currently checking feedback on the performance of our OpenStack environment in relation to the installation / configuration of UCS. I have now seen that UCS hangs for 5 minutes at the end of the installation in 5.0-6. The wizard shows the last join script. However, the join has already been completed. I see this in the process list: root 1142 0.4 9.9 669988 202452 ? Sl 15:03 0:04 \_ /usr/bin/python3 /usr/sbin/univention-management-console-module -m setup -s /run/univention-management-console/588-setup-1708697040012-8b1c3931-1391-4994-b587-67.socket -d 2 -l en_US.UTF-8 root 1637 0.0 0.1 7624 3372 ? S 15:05 0:00 \_ /bin/bash /usr/lib/univention-system-setup/scripts/setup-join.sh --run_cleanup_as_atjob root 2492 0.0 0.1 7624 2332 ? S 15:11 0:00 \_ /bin/bash /usr/lib/univention-system-setup/scripts/setup-join.sh --run_cleanup_as_atjob root 2500 0.0 0.0 5960 748 ? S 15:11 0:00 | \_ tee -a /var/log/univention/join.log root 2493 0.0 0.1 7624 2980 ? S 15:11 0:00 \_ /bin/bash /usr/lib/univention-system-setup/scripts/setup-join.sh --run_cleanup_as_atjob The actual join script is missing. +++ This bug was initially created as a clone of Bug #34106 comment 7+++
Do not shoot the messenger `91univention-saml.inst` as the culprit is: `udm-cli`. The problem is a "file descriptor leak": 1. "setup-join.sh" forks a "tee join.log" to capture the output of all "*.inst". 2. One of them (91univention-saml.inst) uses "udm-cli", which forked the background server, which inherits the "writeable pipe to tee". 3. The join script itself terminates, but the "udm-cli-server" remains as the "only writeable process". 4. Thus "tee" does not get a "EOF" on its "read pipe" and continues running. 5. As such the "( … )" sub-shell does not exit until "udm-cli" kills itself after 10m of inactivity. ```console # ps axfu root 4563 0.3 6.1 625792 124860 ? Sl 09:25 0:04 \_ /usr/bin/python3 /usr/sbin/univention-management-console-module -m setup -s /run/univention-management-console/790-setup-1708935926478-6acf32a1-31cc-480f-b3c4-01.socke root 14061 0.0 0.1 7452 3440 ? S 09:31 0:00 \_ /bin/bash /usr/lib/univention-system-setup/scripts/setup-join.sh --run_cleanup_as_atjob root 20808 0.0 0.1 7452 2416 ? S 09:40 0:00 \_ /bin/bash /usr/lib/univention-system-setup/scripts/setup-join.sh --run_cleanup_as_atjob root 20815 0.0 0.1 7452 2524 ? S 09:40 0:00 | \_ /bin/bash /usr/lib/univention-system-setup/scripts/setup-join.sh --run_cleanup_as_atjob root 23350 0.0 0.1 7896 4060 ? S 09:43 0:00 | | \_ /bin/bash /usr/lib/univention-install/30univention-monitoring-client.inst root 23686 9.6 2.6 153220 54116 ? S 09:43 0:00 | | \_ python3 -m univention.lib.ldap_extension ucs_registerLDAPExtension --packagename univention-monitoring-client --packageversion 1.0.2-6 --ucsversion root 20816 0.0 0.0 5592 688 ? S 09:40 0:00 | \_ tee -a /var/log/univention/join.log root 20809 0.1 0.1 7452 2416 ? S 09:40 0:00 \_ /bin/bash /usr/lib/univention-system-setup/scripts/setup-join.sh --run_cleanup_as_atjob # lsof tee 20816 root 0r FIFO 0,12 0t0 143894 pipe <<<<< tee 20816 root 1w FIFO 0,12 0t0 143893 pipe tee 20816 root 2w FIFO 0,12 0t0 143893 pipe ... univentio 27637 root 3u REG 8,1 1224 4083 /var/log/univention/directory-manager-cmd.log univentio 27637 root 4w REG 8,1 160427 4013 /tmp/91univention-saml.inst.debug.AuijGOh (deleted) univentio 27637 root 5w FIFO 0,12 0t0 143894 pipe <<<<< univentio 27637 root 6u unix 0x000000004285aff8 0t0 192797 /tmp/admincli_0/sock type=STREAM ``` A `pkill -f /usr/share/univention-directory-manager-tools/univention-cli-server` unstucks the installation. Reproducer: ```console $ # Kill old CLI server $ pkill -f /usr/share/univention-directory-manager-tools/univention-cli-server $ # Start new CLI server $ udm 3>&1 | tee /dev/null >/dev/null & $ tee=$! $ # Inspect $ lsof -p "$(pgrep -f /usr/share/univention-directory-manager-tools/univention-cli-server)" -a -d 0-255 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME univentio 19899 root 0u CHR 1,3 0t0 7136 /dev/null univentio 19899 root 1u CHR 1,3 0t0 7136 /dev/null univentio 19899 root 2u CHR 1,3 0t0 7136 /dev/null univentio 19899 root 3w FIFO 0,12 0t0 2433365 pipe univentio 19899 root 4u REG 254,0 34 3670003 /var/log/univention/directory-manager-cmd.log univentio 19899 root 5u unix 0x000000003f3b2494 0t0 2433401 /tmp/admincli_0/sock type=STREAM ```
Package: univention-directory-manager-modules Version: 15.0.25-12 Branch: ucs_5.0-0 Scope: errata5.0-6 [5.0-6] b5a8fe138f refactor(udm): command line parsing management/univention-directory-manager-modules/univention-cli-client | 2 +- management/univention-directory-manager-modules/univention-cli-server | 20 ++++++++++++-------- test/ucs-test/tests/59_udm/05_udm_cli | 2 +- 3 files changed, 14 insertions(+), 10 deletions(-) [5.0-6] 42c23c731a refactor(udm): print error via sys.exit() management/univention-directory-manager-modules/univention-cli-client | 12 ++++-------- management/univention-directory-manager-modules/univention-cli-server | 9 +++------ 2 files changed, 7 insertions(+), 14 deletions(-) [5.0-6] a701e33190 refactor(udm): modernize UCR usage management/univention-directory-manager-modules/debian/control | 2 +- management/univention-directory-manager-modules/univention-cli-client | 4 +--- management/univention-directory-manager-modules/univention-cli-server | 8 +++----- 3 files changed, 5 insertions(+), 9 deletions(-) [5.0-6] 8a0f3a0721 style(udm): use OSError management/univention-directory-manager-modules/univention-cli-client | 10 +++++----- management/univention-directory-manager-modules/univention-cli-server | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) [5.0-6] e046af7b01 style(udm): list.extend -> += management/univention-directory-manager-modules/univention-cli-client | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) [5.0-6] dc9957df8b refactor(udm): convert JSON from str to bytes management/univention-directory-manager-modules/univention-cli-client | 4 +--- management/univention-directory-manager-modules/univention-cli-server | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-) [5.0-6] 8dc0a07185 doc(udm): Add PEP-484 type hints management/univention-directory-manager-modules/univention-cli-client | 19 ++++++++++--------- management/univention-directory-manager-modules/univention-cli-server | 15 ++++++++------- 2 files changed, 18 insertions(+), 16 deletions(-) [5.0-6] 5c0e44c650 fix(udm): close 2nd socket management/univention-directory-manager-modules/univention-cli-client | 1 + 1 file changed, 1 insertion(+) [5.0-6] a3e41df933 fix(udm): file descriptor leak in CLI server doc/errata/staging/univention-directory-manager-modules.yaml | 5 ++-- .../univention-directory-manager-modules/debian/changelog | 6 +++++ .../univention-directory-manager-modules/univention-cli-client | 7 ----- .../univention-directory-manager-modules/univention-cli-server | 40 ++++++++++++++++++++++------- test/ucs-test/tests/59_udm/05_udm_cli | 14 ++++++++++ 5 files changed, 54 insertions(+), 18 deletions(-)
OK: inherited file descriptors are closed OK: CLI server daemonization OK: stdout,stderr is not written to the logfile if writable OK: code review OK: cleanups OK: dropping of Python 2 compatiblity (CLI server is Python3 only) OK: advisory
<https://errata.software-univention.de/#/?erratum=5.0x991>