Bug 47029 - Apache fails to restart after UCS-4.3-0 errat41 update
Apache fails to restart after UCS-4.3-0 errat41 update
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 4.3
Other Linux
: P5 normal (vote)
: UCS 4.3-0-errata
Assigned To: Erik Damrose
Jürn Brodersen
:
Depends on: 46772
Blocks:
  Show dependency treegraph
 
Reported: 2018-05-17 15:22 CEST by Philipp Hahn
Modified: 2018-05-18 13:32 CEST (History)
2 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 5: Will affect all installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.429
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2018-05-17 15:22:07 CEST
Today I updated my UCS-4.3-0 VM from errata 35 to 83, which includes <http://errata.software-univention.de/ucs/4.3/41.html>.
The UMC update dialog was stuck telling me for the update to finish.
Nothing happened for ~½h.
Trying to reload the UMC failed.
Logging in to the server showed that apache2.service was no longer running.

From /var/log/univention/updater.log:
> 2018-05-17 01:46:19.463878188+02:00 (in joinscript_init)
> Joinscript 34univention-management-console-server.inst finished with exitcode 1
> univention-management-console-server.service is not active, cannot reload.
> invoke-rc.d: initscript univention-management-console-server, action "reload" failed.
...
> apache2 (2.4.25-3+deb9u4A~4.3.0.201804040703) wird eingerichtet ...
> Job for apache2.service failed because the control process exited with error code.
> See "systemctl status apache2.service" and "journalctl -xe" for details.
> invoke-rc.d: initscript apache2, action "start" failed.
> ● apache2.service - The Apache HTTP Server
>    Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
>    Active: failed (Result: exit-code) since Thu 2018-05-17 01:46:35 CEST; 15ms ago
>   Process: 7404 ExecStart=/usr/sbin/apachectl start (code=exited, status=126)
>  Main PID: 1718 (code=exited, status=0/SUCCESS)
>       CPU: 4ms
> 
> Mai 17 01:46:35 ma43 systemd[1]: Starting The Apache HTTP Server...
> Mai 17 01:46:35 ma43 apachectl[7404]: /usr/sbin/apachectl: 174: /usr/sbin/apachectl: /usr/sbin/apache2: Permission denied
> Mai 17 01:46:35 ma43 apachectl[7404]: Action 'start' failed.
> Mai 17 01:46:35 ma43 apachectl[7404]: The Apache error log may have more in…ion.
> Mai 17 01:46:35 ma43 systemd[1]: apache2.service: Control process exited, code=exited status=126
> Mai 17 01:46:35 ma43 systemd[1]: Failed to start The Apache HTTP Server.
> Mai 17 01:46:35 ma43 systemd[1]: apache2.service: Unit entered failed state.
> Mai 17 01:46:35 ma43 systemd[1]: apache2.service: Failed with result 'exit-code'.

The permissions were already fine again:
# ls -l /usr/sbin/apache2
-rwxr-xr-x 1 root root 666760 Apr  4 17:11 /usr/sbin/apache2

Restarting the service fixed the problem and made UMC resume the update process:
# systemctl restart apache2


Fiddling with the file permission seems to be a bad idea to me as there have been multiple similar issues in the past.
Maybe <https://trello.com/c/prdMpY9Q/3-%C3%BCberarbeitung-updater-maintenance-mode>?

Reverting and doing the same update again shows the error again.
Comment 1 Erik Damrose univentionstaff 2018-05-17 17:00:26 CEST
Please provide the updater.log / umc-updater.log

In a quick test updating from 4.3-0e0 to 4.3-0e81 via UMC worked, but in the meantime there was the fix from bug #46847 in errata 21.
Comment 2 Erik Damrose univentionstaff 2018-05-17 17:01:48 CEST
It is not obvious, but here someone might have had the same problem: https://help.univention.com/t/solved-issue-update-to-4-3-errata-83/8782
Comment 3 Erik Damrose univentionstaff 2018-05-17 17:28:02 CEST
(In reply to Erik Damrose from comment #1)
> Please provide the updater.log / umc-updater.log

Or can i use your VM with a snapshot to reproduce it?
Comment 4 Erik Damrose univentionstaff 2018-05-17 17:40:28 CEST
I made another quick test and updated the umc packages to the state of errata21:

10.0.5-1A~4.3.0.201804181755

After installing all available errata updates via UMC apache2 was still/again running
Comment 5 Erik Damrose univentionstaff 2018-05-18 12:37:20 CEST
Analysis: apache2 is stopped in the previous' version prerm. Since the UMC updater calls 
/usr/share/univention-updater/disable-apache2-umc
the apache2 binary x bit is removed, so the apache2.postinst is unable to start the process. So it is only an issue for apache2 / umc updates.

I fixed it by adapting the corresponding enable-apache2-umc script. The idea is to have a working state by starting all affected services. There is an option --no-restart (which is set by the UMC updater). This is probably there to avoid ending existing sessions and connections. But simply starting an already running service should cause no issue (famous last words..), and we definitely want all three services running at the end of an update.

Reproducer: install u-updater package from errata20, and an apache2 update has to be pending.

apt-get install univention-updater=13.0.1-42A~4.3.0.201804181752 univention-management-console-module-updater=13.0.1-42A~4.3.0.201804181752 apache2=2.4.25-3+deb9u3A~4.3.0.201711272012 apache2-bin=2.4.25-3+deb9u3A~4.3.0.201711272012 apache2-utils=2.4.25-3+deb9u3A~4.3.0.201711272012 apache2-data=2.4.25-3+deb9u3A~4.3.0.201711272012

a508badd Try to start all services that are (re-)enabled in enable--apache2-umc script
univention-updater 13.0.1-44A~4.3.0.201805181218
1a74c408 yaml
Comment 6 Jürn Brodersen univentionstaff 2018-05-18 13:02:29 CEST
[4.3-0 248676b675] Bug #47029: yaml (typo)
Comment 7 Jürn Brodersen univentionstaff 2018-05-18 13:06:55 CEST
What I tested:
Updated using the umc -> apache is running after the update -> OK

YAML -> OK
Comment 8 Quality Assurance univentionstaff 2018-05-18 13:32:32 CEST
<http://errata.software-univention.de/ucs/4.3/84.html>