Bug 41203 - apache2 service not running after update to UCS 3.3
apache2 service not running after update to UCS 3.3
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Update - Release updates
UCS 3.3
Other Linux
: P5 normal (vote)
: UCS 3.3
Assigned To: Philipp Hahn
Stefan Gohmann
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-05-04 13:39 CEST by Erik Damrose
Modified: 2016-09-29 17:25 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
updalter.log, apache2 not running at the end (220.62 KB, text/x-log)
2016-05-04 13:39 CEST, Erik Damrose
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Erik Damrose univentionstaff 2016-05-04 13:39:14 CEST
Created attachment 7638 [details]
updalter.log, apache2 not running at the end

Seen on 2 systems, master and slave: When updating to UCS 3.3 via UMC Updater module, apache2 is not running at the end of the update. Thus, no UMC interactions are possible, the user gets no notification that the update is finished.

apache2 runs fine after the update is finished if started manually.

updater.log appended
Comment 1 Philipp Hahn univentionstaff 2016-05-17 17:33:31 CEST
# tail -n1 /var/log/apache2/error.log 
apache2: Syntax error on line 244 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/negotiation.load: Cannot load /usr/lib/apache2/modules/mod_negotiation.so into server: /usr/lib/apache2/modules/mod_negotiation.so: undefined symbol: ap_set_accept_ranges
Comment 2 Philipp Hahn univentionstaff 2016-05-19 15:54:02 CEST
# Reproducer:
 ucr set repository/online/server=apt.knut.univention.de version/version=3.3 version/patchlevel=0 version/erratalevel=0
 /usr/share/univention-updater/disable-apache2-umc
 ls -l /usr/sbin/apache2 /usr/lib/apache2/mpm-prefork/apache2
 cd /var/log
 truncate -s 0 univention/updater.log dpkg.log apache2/error.log
 tail -f univention/updater.log dpkg.log apache2/error.log | tee /tmp/41203.log &
 univention-install -qq libapache2-mod-php5 apache2-mpm-prefork

==> dpkg.log <==
2016-05-19 14:03:20 status half-configured apache2-mpm-prefork:amd64 2.2.22-13~ucs3.3.99.201602232007
==> univention/updater.log <==
Starting web server: apache2httpd (pid 1467) already running
==> dpkg.log <==
2016-05-19 14:03:20 status installed apache2-mpm-prefork:amd64 2.2.22-13~ucs3.3.99.201602232007
...
2016-05-19 14:03:20 status half-configured libapache2-mod-php5:amd64 5.4.45-0~ucs3.3.230.201603072027
==> apache2/error.log <==
[Thu May 19 14:03:21 2016] [notice] Graceful restart requested, doing restart
==> apache2/error.log <==
apache2: Syntax error on line 244 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/negotiation.load: Cannot load /usr/lib/apache2/modules/mod_negotiation.so into server: /usr/lib/apache2/modules/mod_negotiation.so: undefined symbol: ap_set_accept_ranges
==> dpkg.log <==
2016-05-19 14:05:21 trigproc libapache2-mod-php5:amd64 5.4.45-0~ucs3.3.230.201603072027 <none>
2016-05-19 14:05:21 status half-configured libapache2-mod-php5:amd64 5.4.45-0~ucs3.3.230.201603072027
==> univention/updater.log <==
Reloading web server config: apache2 not running.


Its "/usr/share/univention-updater/disable-apache2-umc" again:
- UMC removed the execute permissions from "/usr/sbin/apache2" through "script/disable-apache2-umc" thus breaking any restart
- Then "apache2-mpm-prefork" is unpacked and updated by dpkg; as the restart has been disabled, the old binary keeps running
- When "libapache2-mod-php5" in configured, it tries to "force-reload" "/etc/init.d/apache2" as it needs its file reloaded
- That invokes the new "apache2ctl configtest" binary, which finds no errors, as it is a new process and thus consistent with the modules.
- Then Apache's init script calls "apache2ctl graceful" to request a graceful reload of the running process still using the *old* binaries
- The old binary fails to load the new modules and commits suicide to terminates itself
- "apache2ctl status" finds no running Apache and returns success anyway.
- Now Apache is crashed and any restart failed due to the removed executable permissions.


Why is UMC mucking with Apache2 restarts?
Please teach UMC to handle connections problems and to re-connect instead of breaking binaries during package upgrade.
dpkg must be able to stop/[re]start processes during upgrade, especially binaries using dynamic module loading like Apache!


r69406 | Bug #41203 up: Re-enable apache restart in preup.sh
r69405 | Bug #41203 up: Remove broken disabling of apache2 restart
 Remove broken "chmod -x /usr/sbin/apache2"

Package: univention-updater
Version: 9.100.5-3.1476.201605191543
Branch: ucs_3.3-0
Comment 3 Stefan Gohmann univentionstaff 2016-05-23 08:58:54 CEST
My tests were successful. As far as I know, UMC can handle apache restarts, for example Bug #29809.
Comment 4 Alexander Kläser univentionstaff 2016-05-25 14:20:50 CEST
(In reply to Stefan Gohmann from comment #3)
> My tests were successful. As far as I know, UMC can handle apache restarts,
> for example Bug #29809.

Yes, AFAI this can only be somewhat critical during system setup as certificates are regenerated which in turn would require a UMC + Apache restart + user login. During an update, we only would not like to see a UMC restart which would result in a login dialog. Apache should be fine, though.

(In reply to Philipp Hahn from comment #2)
> [...]
> Why is UMC mucking with Apache2 restarts?
> Please teach UMC to handle connections problems and to re-connect instead of
> breaking binaries during package upgrade.
> dpkg must be able to stop/[re]start processes during upgrade, especially
> binaries using dynamic module loading like Apache!

I agree with you. I opened Bug 41342 for this issue.
Comment 5 Stefan Gohmann univentionstaff 2016-06-07 21:35:46 CEST
UCS 3.3 has been released:
 https://docs.software-univention.de/release-notes-3.3-0-en.html
 https://docs.software-univention.de/release-notes-3.3-0-de.html

If this error occurs again, please use "Clone This Bug".