Bug 34834

Summary:	Installation got stuck in 35univention-management-console-module-appcenter.inst
Product:	UCS	Reporter:	Stefan Gohmann <gohmann>
Component:	UMC - App-Center	Assignee:	Stefan Gohmann <gohmann>
Status:	CLOSED FIXED	QA Contact:	Philipp Hahn <hahn>
Severity:	normal
Priority:	P5	CC:	best, gohmann, hahn, klaeser, uf, walkenhorst
Version:	UCS 3.2
Target Milestone:	UCS 3.2-2
Hardware:	Other
OS:	Linux
URL:	http://tools.ietf.org/html/rfc2616#section-8.1
What kind of report is it?:	---	What type of bug is this?:	---
Who will be affected by this bug?:	---	How will those affected feel about the bug?:	---
User Pain:		Enterprise Customer affected?:
School Customer affected?:		ISV affected?:
Waiting Support:		Flags outvoted (downgraded) after PO Review:
Ticket number:		Bug group (optional):
Max CVSS v3 score:
Bug Depends on:	32935
Bug Blocks:	34804

Description Stefan Gohmann

2014-05-14 15:42:52 CEST

> During the Release-tests for UCS-3.2-2 several (2+) VMs got stuck during the
> /usr/sbin/univention-register-apps; one has 120 open TCP+SSL connections to
> <https://download2.software-univention.de/>. The VM is stuck since >20h. 


+++ This bug was initially created as a clone of Bug #32935 +++

The app_center.py-module currently starts one thread for each file to be downloaded.
This leads to (currently) 209 threads with 209 TCP-connections (trying to) starting 209 Apache processes on the server.
(Each with a TCP and HTTPS handshake)

Every added App increases this problem.

As these files are static a single connection with properly pipelined requests is the fastest and most efficient (for both client and server).

Emulating this process with wget allows fetching all 209 files in just 4.9 seconds using a single connection, whereas using 209 connections requires 6.5 seconds (on a quad-core machine) but using about 100 times more CPU time.

(I think wget does not do pipelining so the runtime gains should be even greater)


Not using threads would also allow the code to do error handling (currently missing files are only logged, not handled)


This may impact the release negatively because of the release many new installations at the same time.

Comment 1 Stefan Gohmann

2014-05-14 15:50:02 CEST

A timeout has been added to univention-register-apps: r50256

Revert of Bug #32935: r50255

Changelog: r50257

Comment 2 Philipp Hahn

2014-05-14 21:20:19 CEST

OK: revert r50255
OK: alarm r50256
OK: changelog r50257

OK: tc qdisc ... tc class ... tc filter ... univention-register-apps

FYI: In once test I received the very long error output like:
...
14.05.14 15:14:18.307 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/univention-demoapp.png: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.307 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/samba-memberserver.ini: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.308 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/samba-memberserver.png: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.308 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/zarafa_20140206.ini: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.308 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/zarafa_20140206.png: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.308 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/univention-repository/3.2/maintained/component/zarafa_20140206/README_UPDATE: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.308 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/univention-repository/3.2/maintained/component/uccbeta_20140501/README_EN: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.308 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/univention-repository/3.2/maintained/component/uccbeta_20140501/README_DE: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.309 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/uccbeta_20140501.png: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.309 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/uccbeta_20140501.ini: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.309 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/oxseforucs_20131122.ini: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.309 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/oxseforucs_20131122.png: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.309 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/univention-repository/3.2/maintained/component/sugarcrm_20140310/LICENSE_AGREEMENT: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.309 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/sugarcrm_20140310.ini: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.310 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/meta-inf/3.2/sugarcrm_20140310.png: <urlopen error [Errno 101] Network is unreachable>
14.05.14 15:14:18.310 MODULE (ERROR): Error downloading https://appcenter.software-univention.de/univention-repository/3.2/maintained/component/sugarcrm_20140310/README_UPDATE: <urlopen error [Errno 101] Network is unreachable>
...
After that the process could not even be terminated by SIGINT. SIGTERM did it.

Comment 3 Stefan Gohmann

2014-05-20 07:53:29 CEST

UCS 3.2-2 has been released:
 http://docs.univention.de/release-notes-3.2-2-en.html
 http://docs.univention.de/release-notes-3.2-2-de.html

If this error occurs again, please use "Clone This Bug".