Bug 37952

Summary:	Restarting apache2 stalls
Product:	UCS	Reporter:	Philipp Hahn <hahn>
Component:	Apache	Assignee:	Philipp Hahn <hahn>
Status:	CLOSED FIXED	QA Contact:	Stefan Gohmann <gohmann>
Severity:	normal
Priority:	P5	CC:	best, gohmann, jmm
Version:	UCS 4.0
Target Milestone:	UCS 4.0-1-errata
Hardware:	Other
OS:	Linux
What kind of report is it?:	---	What type of bug is this?:	---
Who will be affected by this bug?:	---	How will those affected feel about the bug?:	---
User Pain:		Enterprise Customer affected?:
School Customer affected?:		ISV affected?:
Waiting Support:		Flags outvoted (downgraded) after PO Review:
Ticket number:		Bug group (optional):
Max CVSS v3 score:
Bug Depends on:	37792
Bug Blocks:
Attachments:	Close FDs in apache2-2.init

Description Philipp Hahn

2015-03-06 11:14:17 CET

With Bug #37792 the init-script was changed to restart the Apache2 process if a graceful restart fails. This breaks the install of some Apps (tine20) which hangs in tine20-webstack.postinst.

This is caused by an unfortunate combination of conditions:
- the postinst uses debconf, which opens a PIPE for the internal communication
  - It does not call 'db_stop' as <http://www.fifi.org/doc/debconf-doc/tutorial.html#AEN198> recommends.
- as the graceful restart fails, apache2 is force-restarted (Bug #37792)
- the new apache2 process inherits the open file descriptor if the pipe.
- Apache2 only closes STDIN, STDOUT, and STDERR, but not all other FDs.
  <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=713967>
- The Apache2 process is now an potential writer, for which the shell scripts waits.

# lsof | grep 12572[45]
frontend    564             root    7w     FIFO                0,8      0t0     125724 pipe
frontend    564             root    8r     FIFO                0,8      0t0     125725 pipe
apache2     678             root    3w     FIFO                0,8      0t0     125725 pipe
apache2     680         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     681         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     682         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     683         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2     684         www-data    3w     FIFO                0,8      0t0     125725 pipe
apache2    2084         www-data    3w     FIFO                0,8      0t0     125725 pipe

# /tmp/x/usr/bin/strace -p 564
read(8, ^C <unfinished ...>

# ps xf
 2607 ?        Ss     0:00 sshd: root@notty 
31783 ?        Ss     0:00  \_ bash -c . utils.sh; install_apps tine20
31784 ?        Sl     0:27      \_ /usr/bin/python2.7 /usr/sbin/univention-add-app -a --latest tine20
31841 ?        S      0:01          \_ /usr/bin/python2.7 /usr/sbin/univention-add-app -a --latest tine20
32297 ?        S      0:00              \_ /usr/bin/dpkg --force-confold --force-confold --status-fd 7 --configure l
  564 ?        S      0:00                  \_ /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/tine2
  567 ?        Z      0:00                      \_ [tine20-webstack] <defunct>

/etc/init.d/apache2 restart from a clean environment resolved the hang.

I can re-produce the "Apache inherits open FDs problem" with this:
 exec 3> >(exec tee /tmp/foo)
 /etc/init.d/apache2 restart
 lsof | grep $(readlink /proc/self/fd/3 | egrep -o '[0-9]+')
bash        523             root    3w     FIFO                0,8      0t0     329617 pipe
apache2   11132             root    3w     FIFO                0,8      0t0     329617 pipe
apache2   11157         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11159         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11160         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11161         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   11162         www-data    3w     FIFO                0,8      0t0     329617 pipe
apache2   14266         www-data    3w     FIFO                0,8      0t0     329617 pipe
grep      23125             root    3w     FIFO                0,8      0t0     329617 pipe


I filed <https://bz.apache.org/bugzilla/show_bug.cgi?id=57671>

+++ This bug was initially created as a clone of Bug #37792 +++

Comment 1 Philipp Hahn

2015-03-06 13:04:28 CET

Created attachment 6748 [details]
Close FDs in apache2-2.init

Requires BASH, as
  eval exec "$fd<&-"
does not work.

Works as verified by
# exec 3>/dev/null
# lsof -a -p $$ -d 3
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    6277 root    3w   CHR    1,3      0t0 1028 /dev/null
# service apache2 restart
# lsof -a -p $(</var/run/apache2.pid) -d 3
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
apache2 3440 root    3u  sock    0,7      0t0  13663 can't identify protocol

Comment 2 Stefan Gohmann

2015-03-09 06:21:52 CET

(In reply to Philipp Hahn from comment #1)
> Created attachment 6748 [details]
> Close FDs in apache2-2.init

We should increase the timeout only to 3 seconds. The patch changes it to 5 seconds.

Comment 3 Philipp Hahn

2015-03-09 09:22:31 CET

r14435 | Bug #37952 apache2: Close FDs

Package: apache2
Version: 2.2.22-13.87.201503090840
Branch: ucs_4.0-0
Scope: errata4.0-1

r58724 | Bug #37952 Apache: Close FDs YAML
 2015-03-09-apache2.yaml

r58725 | Bug #37952: 01/85apache_close_fds
 tests/01_base/85apache_close_fds
 QA: return early from /etc/init.d/apache2 close_fds() to verify test

Comment 4 Stefan Gohmann

2015-03-10 14:36:10 CET

Tests: OK

YAML: OK

Code review: OK

Comment 5 Moritz Muehlenhoff

2015-03-11 15:05:53 CET

http://errata.univention.de/ucs/4.0/109.html