Bug 37952 - Restarting apache2 stalls
Restarting apache2 stalls
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Apache
UCS 4.0
Other Linux
: P5 normal (vote)
: UCS 4.0-1-errata
Assigned To: Philipp Hahn
Stefan Gohmann
:
Depends on: 37792
Blocks:
  Show dependency treegraph
 
Reported: 2015-03-06 11:14 CET by Philipp Hahn
Modified: 2015-03-11 15:05 CET (History)
3 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
Close FDs in apache2-2.init (1.66 KB, patch)
2015-03-06 13:04 CET, Philipp Hahn
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2015-03-06 11:14:17 CET
With Bug #37792 the init-script was changed to restart the Apache2 process if a graceful restart fails. This breaks the install of some Apps (tine20) which hangs in tine20-webstack.postinst.

This is caused by an unfortunate combination of conditions:
- the postinst uses debconf, which opens a PIPE for the internal communication
  - It does not call 'db_stop' as <http://www.fifi.org/doc/debconf-doc/tutorial.html#AEN198> recommends.
- as the graceful restart fails, apache2 is force-restarted (Bug #37792)
- the new apache2 process inherits the open file descriptor if the pipe.
- Apache2 only closes STDIN, STDOUT, and STDERR, but not all other FDs.
  <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=713967>
- The Apache2 process is now an potential writer, for which the shell scripts waits.

# lsof | grep 12572[45]
frontend    564             root    7w     FIFO                0,8      0t0     125724 pipe
frontend    564             root    8r     FIFO                0,8      0t0     125725 pipe
apache2     678             root    3w     FIFO                0,8      0t0     125725 pipe
apache2     680         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     681         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     682         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     683         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     684         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2    2084         www-data    3w     FIFO                0,8      0t0     125725 pipe

# /tmp/x/usr/bin/strace -p 564
read(8, ^C <unfinished ...>

# ps xf
 2607 ?        Ss     0:00 sshd: root@notty 
31783 ?        Ss     0:00  \_ bash -c . utils.sh; install_apps tine20
31784 ?        Sl     0:27      \_ /usr/bin/python2.7 /usr/sbin/univention-add-app -a --latest tine20
31841 ?        S      0:01          \_ /usr/bin/python2.7 /usr/sbin/univention-add-app -a --latest tine20
32297 ?        S      0:00              \_ /usr/bin/dpkg --force-confold --force-confold --status-fd 7 --configure l
  564 ?        S      0:00                  \_ /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/tine2
  567 ?        Z      0:00                      \_ [tine20-webstack] <defunct>

/etc/init.d/apache2 restart from a clean environment resolved the hang.

I can re-produce the "Apache inherits open FDs problem" with this:
 exec 3> >(exec tee /tmp/foo)
 /etc/init.d/apache2 restart
 lsof | grep $(readlink /proc/self/fd/3 | egrep -o '[0-9]+')
bash        523             root    3w     FIFO                0,8      0t0     329617 pipe
apache2   11132             root    3w     FIFO                0,8      0t0     329617 pipe
apache2   11157         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11159         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11160         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11161         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11162         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   14266         www-data    3w     FIFO                0,8      0t0     329617 pipe
grep      23125             root    3w     FIFO                0,8      0t0     329617 pipe


I filed <https://bz.apache.org/bugzilla/show_bug.cgi?id=57671>

+++ This bug was initially created as a clone of Bug #37792 +++
Comment 1 Philipp Hahn univentionstaff 2015-03-06 13:04:28 CET
Created attachment 6748 [details]
Close FDs in apache2-2.init

Requires BASH, as
  eval exec "$fd<&-"
does not work.

Works as verified by
# exec 3>/dev/null
# lsof -a -p $$ -d 3
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    6277 root    3w   CHR    1,3      0t0 1028 /dev/null
# service apache2 restart
# lsof -a -p $(</var/run/apache2.pid) -d 3
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
apache2 3440 root    3u  sock    0,7      0t0  13663 can't identify protocol
Comment 2 Stefan Gohmann univentionstaff 2015-03-09 06:21:52 CET
(In reply to Philipp Hahn from comment #1)
> Created attachment 6748 [details]
> Close FDs in apache2-2.init

We should increase the timeout only to 3 seconds. The patch changes it to 5 seconds.
Comment 3 Philipp Hahn univentionstaff 2015-03-09 09:22:31 CET
r14435 | Bug #37952 apache2: Close FDs

Package: apache2
Version: 2.2.22-13.87.201503090840
Branch: ucs_4.0-0
Scope: errata4.0-1

r58724 | Bug #37952 Apache: Close FDs YAML
 2015-03-09-apache2.yaml

r58725 | Bug #37952: 01/85apache_close_fds
 tests/01_base/85apache_close_fds
 QA: return early from /etc/init.d/apache2 close_fds() to verify test
Comment 4 Stefan Gohmann univentionstaff 2015-03-10 14:36:10 CET
Tests: OK

YAML: OK

Code review: OK
Comment 5 Moritz Muehlenhoff univentionstaff 2015-03-11 15:05:53 CET
http://errata.univention.de/ucs/4.0/109.html