Bug 41781 - Consecutive apache2 restarts error: Address already in use: make_sock: could not bind to address [::]:443
Consecutive apache2 restarts error: Address already in use: make_sock: could ...
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Apache
UCS 4.1
Other Linux
: P5 normal (vote)
: UCS 4.1-3-errata
Assigned To: Erik Damrose
Felix Botner
:
: 38797 (view as bug list)
Depends on:
Blocks: 41961
  Show dependency treegraph
 
Reported: 2016-07-12 14:58 CEST by Erik Damrose
Modified: 2016-09-29 17:31 CEST (History)
4 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 6: Setup Problem: Issue for the setup process
Who will be affected by this bug?: 5: Will affect all installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.857
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
Fix for initscript, wait for pidfile after apache2 start (1.38 KB, patch)
2016-07-12 14:58 CEST, Erik Damrose
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Erik Damrose univentionstaff 2016-07-12 14:58:36 CEST
Created attachment 7796 [details]
Fix for initscript, wait for pidfile after apache2 start

To reproduce:
service apache2 restart; service apache2 restart;
[ ok ] Restarting web server: apache2 ... waiting .
[....] Restarting web server: apache2(98)Address already in use: make_sock: could not bind to address [::]:443
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
no listening sockets available, shutting down
Unable to open logs
Action 'start' failed.
The Apache error log may have more information.
 failed!

This error is caused by apache2 not immediately writing its PIDFILE after starting the daemon. The restart part of the init script only checks for the existance of the pidfile, and tries to start apache2 immediately if the PIDFILE is not present.

If multiple packages are installed that set up apache2 config options and restart apache2, a race condition may occur, and apache2 is not restarted every time, which may leave some config options present but not picked up by the apache2 daemon.

Attached patch waits for the apache 2 pidfile after apache2 is started via initscript.
Comment 1 Erik Damrose univentionstaff 2016-09-05 16:55:54 CEST
r16697 patch comitted -> apache2 2.2.22-13.100.201609051644
r72260 yaml
Comment 2 Erik Damrose univentionstaff 2016-09-06 16:36:43 CEST
r16723 migrate patches to ucs 4.2. I noticed that there is already a apache_wait_start() function with very similar functionality in the init script for apache 2.4, so i did not port the fix from this bug over.

ucs 4.2: apache2 2.4.10-10+deb8u5A~4.2.0.201609061628
Comment 3 Philipp Hahn univentionstaff 2016-09-08 16:30:38 CEST
# invoke-rc.d apache2 reload
[....] Reloading web server: apache2/usr/sbin/apache2ctl: 101: /usr/sbin/apache2ctl: www-browser: not found
'www-browser -dump http://localhost:80/server-status' failed.
Maybe you need to install a package providing www-browser or you
need to adjust the APACHE_LYNX variable in /etc/apache2/envvars
/etc/init.d/apache2: Zeile 385: pidof_apache: Kommando nicht gefunden.
[....] Restarting web server: apache2
[FAIL] Restarting web server: apache2 failed!
. ok 

1. Fix the use of "pidof_apache" in the patch:

$ grep -n pidof_apache *
10-apache2-reload.patch:40:+    if ! pidof_apache >/dev/null; then

Change that to
  PIDTMP=$(pidofproc -p $PIDFILE $DAEMON)
  if [ -n "${PIDTMP:-}" ] && kill -0 "${PIDTMP:-}" 2> /dev/null; then

Checks if PIDFILE corresponds the the daemons name and is still running.


2. Now what? "failed" or "ok"
This happens because `"$0" restart` is called before log_..._msg.



3. While you are at it: convert "apachectl status" to "wget -O/dev/null -q --timeout=5 http://localhost:80/"

# grep -Fn STATUSURL /usr/sbin/apachectl 
75:STATUSURL="${APACHE_STATUSURL:-http://localhost:80/server-status}"
101:    if ! $LYNX $STATUSURL ; then
102:        echo "'$LYNX $STATUSURL'" failed. >&2



$ svn diff 10-apache2-reload.patch 
Index: 10-apache2-reload.patch
===================================================================
--- 10-apache2-reload.patch     (Revision 16732)
+++ 10-apache2-reload.patch     (Arbeitskopie)
@@ -26,28 +26,30 @@
  
  
  case "$1" in
-@@ -363,6 +375,20 @@
+@@ -363,6 +375,22 @@
        log_daemon_msg "Reloading $DESC" "$NAME"
        do_reload
        RET_STATUS=$?
 +
 +      restart=false
-+      if ! timeout 5 apache2ctl status >/dev/null; then
++      if ! timeout 5 wget -O/dev/null -q --timeout=5 http://localhost:80/; then
 +              $APACHE2CTL stop
 +              pkill -9 -f '^/usr/sbin/apache2 -k'
 +              restart=true
 +      fi
-+      if ! pidof_apache >/dev/null; then
++      PIDTMP=$(pidofproc -p $PIDFILE $DAEMON)
++      if [ -n "${PIDTMP:-}" ] && kill -0 "${PIDTMP:-}" 2> /dev/null; then
 +              restart=true
 +      fi
 +      if $restart; then
-+              "$0" restart
++              do_stop stop
++              do_start
 +      fi
 +
        case "$RET_STATUS" in
                0|1)
                        log_success_msg
-@@ -386,6 +412,7 @@
+@@ -386,6 +414,7 @@
        do_stop stop
        case "$?" in
                0|1)
@@ -55,7 +57,7 @@
                        do_start
                        case "$?" in
                                0)
-@@ -408,6 +435,7 @@
+@@ -408,6 +437,7 @@
        ;;
    start-htcacheclean)
        log_daemon_msg "Starting htcacheclean"
Comment 4 Florian Best univentionstaff 2016-09-09 12:52:15 CEST
Currently a "invoke-rc.d apache2 reload" let's apache crash due to this (UCS 4.2).
syslog shows:
Sep  9 12:50:13 xen3 systemd[1]: Failed to set cpu.cfs_period_us on /system.slice/apache2.service: Permission denied
Sep  9 12:50:13 xen3 systemd[1]: Failed to set cpu.cfs_quota_us on /system.slice/apache2.service: Permission denied
Comment 5 Florian Best univentionstaff 2016-09-13 14:28:27 CEST
*** Bug 38797 has been marked as a duplicate of this bug. ***
Comment 6 Florian Best univentionstaff 2016-09-13 14:29:00 CEST
Please reenable the test cases of Bug #38797 afterwards.
Comment 7 Philipp Hahn univentionstaff 2016-09-14 11:46:16 CEST
The issue from comment 3 blocks the UCS-4.2 DVD - please fix it ASAP!
Comment 8 Erik Damrose univentionstaff 2016-09-14 13:04:34 CEST
I disabled 10-apache2-reload.patch for now. It is unclear if it is required for UCS 4.2 anyway - the patch that is developed at this bug for apache 2.2 is not necessary for apache2.4.

I saw that the apache2 patches for ucs 4.2 will be reviewed again at bug 41929, so i am setting this bug as resolved for ucs 4.1-3
Comment 9 Felix Botner univentionstaff 2016-09-15 10:02:22 CEST
(In reply to Erik Damrose from comment #8)
> I disabled 10-apache2-reload.patch for now. It is unclear if it is required
> for UCS 4.2 anyway - the patch that is developed at this bug for apache 2.2
> is not necessary for apache2.4.
> 
> I saw that the apache2 patches for ucs 4.2 will be reviewed again at bug
> 41929, so i am setting this bug as resolved for ucs 4.1-3

i think this is fixed in debian/jessie

OK - apache reload errata4.1-3
OK - patch
OK - yaml
Comment 10 Janek Walkenhorst univentionstaff 2016-09-21 14:30:19 CEST
<http://errata.software-univention.de/ucs/4.1/274.html>