Univention Bugzilla – Bug 53885
Convert UMC from init scripts to systemd service units
Last modified: 2023-09-25 06:59:30 CEST
TODO management/univention-management-console/debian/univention-management-console-server.init TODO management/univention-management-console/debian/univention-management-console-web-server.init +++ This bug was initially created as a clone of Bug #43691 +++ We should convert all remaining SysV init scripts into systemd service units: - improves robustness - we can get rid of all those horrible shell scripts - we get service supervision for free if desired - it reduces complexity as only one init system gets used - sysv is deprecated $ find -name testframework -prune -o \( -path \*/debian/init -o -path \*/debian/\*.init\* -o -path \*/conffiles/etc/init.d/ \) -printf '%P\n' saml/univention-saml/debian/univention-saml.init container/univention-docker-container-mode/debian/univention-docker-container-mode.init management/univention-directory-policy/debian/univention-directory-policy.init management/univention-directory-listener/debian/univention-directory-listener.init management/univention-directory-notifier/debian/univention-directory-notifier.init management/univention-management-console/debian/univention-management-console-web-server.init management/univention-management-console/debian/univention-management-console-server.init base/univention-updater/debian/univention-updater.univention-maintenance.init base/univention-firewall/debian/univention-firewall.init base/univention-bootsplash/debian/univention-welcome-screen.init base/univention-runit/debian/univention-runit.init base/univention-network-manager/debian/univention-network-common.init base/univention-system-setup/debian/univention-system-setup-boot.init base/univention-system-setup/debian/univention-system-setup-boot.univention-system-setup-boot-prepare.init virtualization/univention-novnc/debian/univention-novnc.init virtualization/univention-virtual-machine-manager-daemon/debian/univention-virtual-machine-manager-daemon.init virtualization/univention-virtual-machine-manager-node/debian/univention-virtual-machine-manager-node-kvm.init test/univention-demo-configuration/debian/univention-demo-configuration.init services/univention-net-installer/debian/univention-net-installer-daemon.init.d services/univention-ad-connector/debian/univention-ad-connector.init services/univention-cloud-init/debian/univention-cloud-init.init services/univention-s4-connector/debian/univention-s4-connector.init services/univention-dhcp/debian/univention-dhcp.init
https://git.knut.univention.de/univention/ucs/-/merge_requests/137
The UMC services have been migrated from init-scripts to systemd services. univention-management-console (12.0.14-1) 5ee46be62453 | Bug #53885: remove UNIX socket before startup 613233f5261f | Bug #53885: set NOFILE limit via systemd 11b9f8d51346 | Bug #53885: /var/run/ → /run/ 7db8dcb4002b | Bug #53885: Remove daemonizing of UMC-Server 4e9698c549a4 | Bug #53885: Remove daemonizing of UMC-Web-Server 1f5001b66f86 | Bug #53885: change from init script to systemd service 26e9ac506710 | docs: add changelog Bug #53885
KVM template 5.0-3+e572 was stuck in USS: Firefox still showed USS running "92saml….inst", but `ps axf` no longer showed any UMC module processes as running. Running `univention-check-join-status` showed my Primary to be fully joined. I quit Firefox with `Ctrl-Q`, but afterwards `univention-managfement-console-web.service` was failed because `/usr/sbin/univention-management-console-web-server` was not executable: only had permissions `0644` The missing executable-permissions were probably caused by `base/univention-updater/script/disable-apache2-umc` which no matching `enable-apache2-umc` being called because I quit USS — or its module process was terminated before it had a change to call it. This is most probably caused by this change, but needs further investigation.
(In reply to Philipp Hahn from comment #4) > This is most probably caused by this change, but needs further investigation. ==> ./univention/setup.log <== Create umc/saml/idp-server Module: setup_saml_sp Try to download idp metadata (1/60) ==> ./daemon.log <== Feb 3 12:49:31 unassigned-hostname systemd[1]: Stopped target Start UMC-Web-Server in multiprocessing mode. Feb 3 12:49:31 unassigned-hostname systemd[1]: Stopping Start UMC-Web-Server in multiprocessing mode. Feb 3 12:49:31 unassigned-hostname systemd[1]: Stopping Univention Management Console Web Server... Feb 3 12:49:31 unassigned-hostname univention-management-console-web-server[594]: [03/Feb/2023:12:49:31] ENGINE Caught signal SIGTERM. /etc/univention/templates/modules/setup_saml_sp.py:124 > call(['systemctl', 'reload', 'univention-management-console-web-server']) This now kills the web-server and it cannot be restarted because of the missing executable permissions for /usr/sbin/univention-management-console-web-server. Removing the executable permissions might have worked in the SysV-init-days, but with systemd it is plain broken! It was introduced back in UCS 3 by Bug #22904.
Feb 03 16:24:35 ma28 systemd[1]: Reloading Univention Management Console Web Server. Feb 03 16:24:35 ma28 systemd[1]: Reloaded Univention Management Console Web Server. Feb 03 16:24:35 ma28 univention-management-console-web-server[573]: [03/Feb/2023:16:24:35] ENGINE Caught signal SIGUSR1. So far this looks like /etc/univention/templates/modules/setup_saml_sp.py doing the `reload`. But next follows this: Feb 03 16:24:36 ma28 systemd[1]: Stopping Univention Management Console Web Server... Feb 03 16:24:36 ma28 univention-management-console-web-server[573]: [03/Feb/2023:16:24:36] ENGINE Caught signal SIGTERM. Feb 03 16:24:36 ma28 univention-management-console-web-server[573]: [03/Feb/2023:16:24:36] ENGINE Bus STOPPING Feb 03 16:26:07 ma28 systemd[1]: univention-management-console-web-server.service: State 'stop-sigterm' timed out. Killing. Feb 03 16:26:07 ma28 systemd[1]: univention-management-console-web-server.service: Killing process 573 (univention-mana) with signal SIGKILL. Feb 03 16:26:07 ma28 systemd[1]: univention-management-console-web-server.service: Main process exited, code=killed, status=9/KILL Feb 03 16:26:07 ma28 systemd[1]: univention-management-console-web-server.service: Failed with result 'timeout'. Feb 03 16:26:07 ma28 systemd[1]: Stopped Univention Management Console Web Server. Feb 03 16:26:07 ma28 systemd[1]: Starting Univention Management Console Web Server... Feb 03 16:26:07 ma28 systemd[1]: Started Univention Management Console Web Server. This is from management/univention-management-console/92univention-management-console-web-server.inst:116 > systemctl restart univention-management-console-web-server after the `restart` the browser-session is gone and Firefox will no longer show any progress.
Back in [UCS 4.3-1 Bug #47437](https://forge.univention.org/bugzilla/show_bug.cgi?id=47436) days we introduced those durned scripts `/usr/share/univention-updater/{en,dis}able-apache2-umc` to remove the executable permissions from `apache2` and `umc-[web-]server` to prevent any Debian-Package-Maintainer- or UCS-join-script from *restarting* those services: When such a `restart` is triggered via some UMC action (running joins scripts, performing release updates, …), it would kill the users web-session and they may — in case of a new UCS 5.0-3 definitely will — get stuck in a dead browser session until they do a manual reload and re-login. In UCS-4.3 time all 3 services still used SysV-init scripts and removing the executable bit from the binaries was sufficient. Fast forward to 5.0-3: Now all 3 services are controlled via `systemd` and the removed executable bit now leads to the service getting *killed* and *not* being restarted. Basically the system is screwed as USS is unable to finish and the bits are permanently "lost". (There's a `@reboot` cron-job to restore them, but you cannot trigger it via the dead UMC.) The correct way is to *never* invoke `systemctl` directly, but to use Debians `deb-systemd-invoke` which adds a _policy layer_ `policy-rc.d`, which can be used to prevent certain actions during — for example — package upgrades or initial system installation. A quick `git grep systemctl` finds 436 locations where `systemctl` is directly used. Excluding `tests/` and `**/debian/changelog` it's still 274, which should be changed. It must be done for any `**/debian/*.{pre,post}{inst,rm}` script and also should be done for any `*.[u]inst`, also for any transitive script or UCR trigger or UDL listener should also be changed as otherwise calling them would lead to the broken state. Minimum would be any use of `systemctl $action {apache2,univention-management-console{,-web}-server[.service]`, which has 29 locations as of `git grep 'systemctl.*\(apache2\|univention-management-console\)' -- ':!test/' ':!**/debian/changelog'`. 5.0-3 would be a good time to do this as - we need to rebuild all packages anyway for the tab-2-spaces change. - 5.0-3 installation is broken because of this *BLOCKER* Alternative: divert `/bin/systemctl` to `.bin` and replace by a shell-script-wapper to filter out those forbidden actions while the update is running. This might easily break the system as `systemd` is the central peace of any modern Linux distribution. And there might be many corner cases which need to be handles, as `systemctl` supports many options which the wrapper then must also handle.
(In reply to Philipp Hahn from comment #7) Dirk found a simpler solution and `mask`ing the services is enough, which has been implemented: [phahn/53885-umc-systemd] 865c51dea1 fix(up): Prevent UMC service from being restarted base/univention-lib/python/package_manager.py | 7 ++---- .../usr/lib/univention-system-setup/scripts/setup-join.sh | 2 +- base/univention-updater/script/disable-apache2-umc | 10 ++------- base/univention-updater/script/enable-apache2-umc | 34 ++++++++--------------------- .../univention-appcenter/umc/python/appcenter/__init__.py | 4 ++-- management/univention-join/umc/python/join/__init__.py | 2 +- 6 files changed, 17 insertions(+), 42 deletions(-) OK: With the installation works again. OK: systemctl status univention-management-console-*server.service RFC: https://git.knut.univention.de/univention/ucs/-/merge_requests/653
[5.0-3] 4821e96853 feat(umc): Use sd_notidy() to signal services are ready management/univention-management-console/debian/changelog | 6 ++++++ management/univention-management-console/debian/control | 2 ++ .../univention-management-console/debian/univention-management-console-server.service | 2 +- .../debian/univention-management-console-web-server.service | 2 +- .../src/univention/management/console/protocol/modserver.py | 1 + .../univention-management-console/src/univention/management/console/protocol/server.py | 5 +++++ management/univention-management-console/univention-management-console-web-server | 3 +++ 7 files changed, 19 insertions(+), 2 deletions(-) Package: univention-management-console Version: 12.0.17-3A~5.0.0.202302071823 Branch: ucs_5.0-0 Scope: ucs5.0-3
OK: new install OK: upgrade
UCS 5.0-3 has been released. https://docs.software-univention.de/release-notes/5.0-3/en/ If this error occurs again, please clone this bug.