Univention Bugzilla – Bug 34804
AppCenter starts N threads
Last modified: 2016-10-11 00:08:08 CEST
More robust fix could be: Put all ini files, icons, READMEs in one all.tar.gz and download this with a single request. Size is currently less than 1 MB Whether download is necessary could be determined by Last-Modified/ETag. Downside: UMC App Center has to download everything even if only one App was added or, "worse", only one ini file changed. We should evaluate this process and if it works fine, update the script that generates the index.json to generate all.tar.gz (name is subject to changes) +++ This bug was initially created as a clone of Bug #32935 +++ The app_center.py-module currently starts one thread for each file to be downloaded. This leads to (currently) 209 threads with 209 TCP-connections (trying to) starting 209 Apache processes on the server. (Each with a TCP and HTTPS handshake) Every added App increases this problem. As these files are static a single connection with properly pipelined requests is the fastest and most efficient (for both client and server). Emulating this process with wget allows fetching all 209 files in just 4.9 seconds using a single connection, whereas using 209 connections requires 6.5 seconds (on a quad-core machine) but using about 100 times more CPU time. (I think wget does not do pipelining so the runtime gains should be even greater) Not using threads would also allow the code to do error handling (currently missing files are only logged, not handled) This may impact the release negatively because of the release many new installations at the same time.
We could also employ HEAD requests (=GET requests without the body) in order to determine whether the file has changed or not. Also see Bug 32935, comment 4 in order to speed up processing multiple requests.
(In reply to Alexander Kläser from comment #1) > We could also employ HEAD requests (=GET requests without the body) in order > to determine whether the file has changed or not. Look at the code on why this wouldn't help: The app center already caches all files locally and skips those already present: file dates are NOT compared, so the app-center never fetches a file twice and never updates a file already downloaded (expect index.json.gz). Even better would be "If-Modified-Since", as this combines the benefit of HEAD with GET: If the file is not modified, a simple "304 NOT MODIFIED" is returned. Otherwise a normal "200 OK" is performed, which saves one round-trip which would be needed in your modified case. Other options: 1. Embed all images base64 encoded in the index.json.gz? 2. Use a version control system like SVN: then you can ask for updates since revision-last-time. 3. Fetch images only on demand and only then cache them locally.
Different approach: storing App Center meta data in a separate package at release time, then only a few file updates/new files will be downloaded.
Was implemented at some point.