Bug 55753 - UMC-web-server killed on upgrade
UMC-web-server killed on upgrade
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 5.0
Other Linux
: P5 normal (vote)
: UCS 5.0-3-errata
Assigned To: Philipp Hahn
Florian Best
https://git.knut.univention.de/univen...
:
: 55772 (view as bug list)
Depends on: 53885 55726
Blocks:
  Show dependency treegraph
 
Reported: 2023-02-23 19:36 CET by Philipp Hahn
Modified: 2023-06-27 16:49 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 6: Setup Problem: Issue for the setup process
Who will be affected by this bug?: 5: Will affect all installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.857
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2023-02-23 19:36:14 CET
Bug #53885 converted the UMC init scripts to systemd.service units.
Bug #55726 is the first erratum 5.0-3+583 for UMC.
Installing the erratum *after* having setup the system via UMC kills only the UMC session.
This is even worse when it is installed as an errata update from USS as then the installation aborts and may packages are in an inconsistent state: `dpkg --audit`.
The same happens for a fresh installation from ISO:

> Vorbereitung zum Entpacken von .../40-univention-management-console-web-server_12.0.17-4A~5.0.0.202302201440_all.deb ...
> + [ -x /etc/init.d/univention-management-console-web-server ]
> + dpkg-maintscript-helper rm_conffile /etc/univention/templates/files/etc/apache2/sites-available/univention-proxy.conf 12.0.0~ -- upgrade 12.0.17-4A~5.0.0.202302201440
> + dpkg-maintscript-helper rm_conffile /etc/init.d/univention-management-console-web-server 12.0.12-17~ -- upgrade 12.0.17-4A~5.0.0.202302201440
> + dpkg-maintscript-helper mv_conffile /etc/univention/templates/files/etc/apache2/sites-available/univention-management-console /etc/univention/templates/files/etc/apache2/sites-available/univention.conf 9.0.0~ -- upgrade 12.0.17-4A~5.0.0.202302201440
> + [ -d /run/systemd/system ]
> + deb-systemd-invoke stop univention-management-console-web-server-multiprocessing.target univention-management-console-web-server.service

For 5.0-3 we switch the mechanism to prevent the UMC restart to use `systemctl mask --runtime`, which creates symbolic links `/run/systemd/systemd/univention-manegement-console[-web]-server.service` -> `/dev/null`, which do not survive a `reboot`. At least our internal KVM images start with apache2.service and univention-management-console[-web]-server.serice unmasked, which would protect them from a `systemctl restart` which is required while USS is running.

But even running /usr/share/univention-updater/disable-apache2-umc manually leads to UMC-web-server being killed by /var/lib/dpkg/info/univention-management-console-web-server.prerm:52
> # Automatically added by dh_systemd_start/12.1.1
> if [ -d /run/systemd/system ]; then
>         deb-systemd-invoke stop 'univention-management-console-web-server-multiprocessing.target' 'univention-management-console-web-server.service' >/dev/null || true
> fi
> # End automatically added section

The masking does not prevent the "stop", only a "restart" and "start"!

Quoting from <man:dh_systemd_start(1)>:
> --restart-after-upgrade
>   Do not stop the unit file until after the package upgrade has been completed.  This is the default behavior in compat 10.
>   In earlier compat levels the default was to stop the unit file in the prerm, and start it again in the postinst.
>   This can be useful for daemons that should not have a possibly long downtime during upgrade. …

Sadly management/univention-management-console/debian/compat=9, so the old behavior of stopping the service during the upgrade is used!

As an alternative the following can be added to debian/rules:
> override_dh_systemd_start:
> <Tab>dh_systemd_start --restart-after-upgrade


Sadly the broken code is part of the already installed *old* `u-m-c-web-server.prerm upgrade $v2`, which is invoked *before* the *new* `u-m-c-web-server.preinst upgrade $v1` has a chance to intervene; the *new* `u-m-c-web-server.prerm failed-upgrade v1` would only be called as a fallback if the old scripts would throw an error. See <https://phahn.gitpages.knut.univention.de/talks/img13.html>


The issue can be mitigated by *not* installing errata updates at the end of USS. That way USS is able to finish successfully. Afterwards errata updates can be installed from UMC, but the session will then be killed, which is not nice, but at least the upgrade continues in the background and you can login again after some time.
Comment 1 Philipp Hahn univentionstaff 2023-02-24 07:31:04 CET
What might work is adding a new "Pre-Depends: umc-fixup," to the package "u-m-c-web-server", which either makes the old "u-m-c-web-server.prerm" fail; or patches out the relevant lines from that file so they no longer get executed.

"Pre-Depends"-packages must be fully configured even before the depending packages is unpacked. APT handles this situation in the way, that it will move the installation of the Pre-Depends to a separate "dpkg" invocation, which will happen *before* the depending package (UMC*) is unpacked.

See MR; works for me in my VM.
Comment 2 Philipp Hahn univentionstaff 2023-02-24 16:02:29 CET
[5.0-3] e5724d2aa2 chore(univention-management-console): update advisory
 doc/errata/staging/univention-management-console.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[5.0-3] 5719d731d5 fix(umc): add missing Python 2.7 dependency
 doc/errata/staging/univention-management-console.yaml     | 4 +++-
 management/univention-management-console/debian/changelog | 6 ++++++
 management/univention-management-console/debian/control   | 2 ++
 3 files changed, 11 insertions(+), 1 deletion(-)

[5.0-3] 19f06b3744 fix(umc): restart during upgrade 5.0-3+572..583
 doc/errata/staging/univention-management-console.yaml           | 15 +++++++++++
 management/univention-management-console/debian/changelog       |  6 +++++
 management/univention-management-console/debian/control         |  8 ++++++
 management/univention-management-console/debian/rules           |  3 +++
 .../univention-management-console-web-server-fix.postinst       | 42 +++++++++++++++++++++++++++++
 5 files changed, 74 insertions(+)
Comment 3 Florian Best univentionstaff 2023-02-24 16:04:15 CET
OK: adjusted /usr/lib/univention-system-setup/scripts/90_postjoin/20upgrade to include:
echo 'deb [trusted=yes] http://omar.knut.univention.de/build2/git/phahn-umc-systemd-upgrade/ git main' >> /etc/apt/sources.list                                             
echo 'deb [trusted=yes] http://updates-test.knut.univention.de/ ucs503 main' >> /etc/apt/sources.list 

Running System Setup worked again.
OK: YAML
Comment 5 Florian Best univentionstaff 2023-03-02 12:27:22 CET
*** Bug 55772 has been marked as a duplicate of this bug. ***