Bug 34804 - AppCenter starts N threads
AppCenter starts N threads
Product: UCS
Classification: Unclassified
Component: App Center
UCS 3.2
Other Linux
: P5 normal (vote)
: ---
Assigned To: App Center maintainers
Depends on: 32935 34834
  Show dependency treegraph
Reported: 2014-05-12 01:31 CEST by Dirk Wiesenthal
Modified: 2016-10-11 00:08 CEST (History)
5 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 2: Improvement: Would be a product improvement
Who will be affected by this bug?: 4: Will affect most installed domains
How will those affected feel about the bug?: 1: Nuisance – not a big deal but noticeable
User Pain: 0.046
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Wiesenthal univentionstaff 2014-05-12 01:31:29 CEST
More robust fix could be:
  Put all ini files, icons, READMEs in one all.tar.gz and download this with a single request. Size is currently less than 1 MB

Whether download is necessary could be determined by Last-Modified/ETag. 

Downside: UMC App Center has to download everything even if only one App was added or, "worse", only one ini file changed.

We should evaluate this process and if it works fine, update the script that generates the index.json to generate all.tar.gz (name is subject to changes)

+++ This bug was initially created as a clone of Bug #32935 +++

The app_center.py-module currently starts one thread for each file to be downloaded.
This leads to (currently) 209 threads with 209 TCP-connections (trying to) starting 209 Apache processes on the server.
(Each with a TCP and HTTPS handshake)

Every added App increases this problem.

As these files are static a single connection with properly pipelined requests is the fastest and most efficient (for both client and server).

Emulating this process with wget allows fetching all 209 files in just 4.9 seconds using a single connection, whereas using 209 connections requires 6.5 seconds (on a quad-core machine) but using about 100 times more CPU time.

(I think wget does not do pipelining so the runtime gains should be even greater)

Not using threads would also allow the code to do error handling (currently missing files are only logged, not handled)

This may impact the release negatively because of the release many new installations at the same time.
Comment 1 Alexander Kläser univentionstaff 2014-05-12 13:28:04 CEST
We could also employ HEAD requests (=GET requests without the body) in order to determine whether the file has changed or not.

Also see Bug 32935, comment 4 in order to speed up processing multiple requests.
Comment 2 Philipp Hahn univentionstaff 2014-05-12 14:31:00 CEST
(In reply to Alexander Kläser from comment #1)
> We could also employ HEAD requests (=GET requests without the body) in order
> to determine whether the file has changed or not.

Look at the code on why this wouldn't help: The app center already caches all files locally and skips those already present: file dates are NOT compared, so the app-center never fetches a file twice and never updates a file already downloaded (expect index.json.gz).

Even better would be "If-Modified-Since", as this combines the benefit of HEAD with GET: If the file is not modified, a simple "304 NOT MODIFIED" is returned. Otherwise a normal "200 OK" is performed, which saves one round-trip which would be needed in your modified case.

Other options:
1. Embed all images base64 encoded in the index.json.gz?
2. Use a version control system like SVN: then you can ask for updates since revision-last-time.
3. Fetch images only on demand and only then cache them locally.
Comment 3 Alexander Kläser univentionstaff 2014-05-12 15:58:22 CEST
Different approach: storing App Center meta data in a separate package at release time, then only a few file updates/new files will be downloaded.
Comment 4 Dirk Wiesenthal univentionstaff 2016-10-11 00:08:08 CEST
Was implemented at some point.