Bug 55162 - High CPU load by computerroom UMC module when connecting to Veyon proxy, traceback: Max retries exceeded with url: /api/v1/authentication/$IP Failed to establish a new connection: [Errno 111]
Summary: High CPU load by computerroom UMC module when connecting to Veyon proxy, trac...
Status: CLOSED FIXED
Alias: None
Product: UCS@school
Classification: Unclassified
Component: Veyon
Version: UCS@school 5.0
Hardware: Other Linux
: P5 normal
Target Milestone: UCS@school 5.0 v3-errata
Assignee: UCS@school maintainers
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-09-02 10:35 CEST by Christina Scheinig
Modified: 2023-04-21 10:24 CEST (History)
5 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 7: Crash: Bug causes crash or data loss
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.400
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2022083021000358
Bug group (optional): Workaround is available
Customer ID: 00009
Max CVSS v3 score:

Attachments
veyon traceback (4.13 KB, text/plain)
2022-09-02 10:35 CEST, Christina Scheinig 		Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Christina Scheinig univentionstaff 2022-09-02 10:35:58 CEST 
Created attachment 10983 [details]
veyon traceback

UCS@School 5.0 V3 Computerraum-Modul

Veyon still generates tracebacks because of too many connections.
in the management-console-module-computerroom.log there are still several tracebacks because of too many connections. This happens even if not a single computer in the computer room is switched on.

This seems to cause an extrem server load:

As long as a room is open in the computer room module in a browser, this generates a massive load on the school server, sometimes with values of more than 30 over several minutes. The load only goes down again when the module has been closed and all processes of the computer room module have "died".

The two processes with the highest load were
/usr/bin/python3 /usr/sbin/univention-management-console-module -m computerroom -s /var/run/univention-management-console/1401-1661933944231.socket -d 4 -l de_DE.UTF-8
/usr/bin/veyon-cli webapi runserver

No significant IO load on the data volumes

Number of Computers in room:
First school:  118 Rechner, UMC-Probleme, Veyon-Tracebacks
Second school: 78 Rechner, Keine UMC-Probleme, keine Veyon-Tracebacks
Third school: 107 Rechner, UMC-Probleme, Veyon-Tracebacks,but a short time later both problems suddenly stopped



There seems to be a connection between the occurrence of Veyon tracebacks in management-console-module-computerroom.log and the operating problems in the UMC. Because without tracebacks, the flickering (Bug 55160) also stops and, in particular, all computers can actually be checked via the checkbox! (Bug 55161)
Comment 1 Daniel Tröder univentionstaff 2022-09-05 08:25:58 CEST 
The traceback does not say anything about "too many connections". It is about not being able to connect to the Veyon proxy ("Failed to establish a new connection").

The Veyon proxy may have simply hung/crashed.

It seems the UMC module is in a busy loop trying to connect to the proxy. That may be the reason it uses so much CPU. It should wait for a sec after a NewConnectionError.

Also: if the exception is handled, it should only log a simple warning, not a complete traceback.
Comment 3 Markus Dählmann 2022-10-05 09:17:36 CEST 
Yes, Veyon crashes, with segfaults, *all the time*, on *all our servers*. At least when opening a computer room with a larger number of clients. Please see tickets
Ticket#2022083021000358 and
Ticket#2022092921000546

Computer room functionality is unusable since UCS 5. Right at this moment a school cannot even exit the exam mode, the computer room module won't let them, because it complains about the non-running (because constantly crashing) Veyon Proxy. Manually (re)starting Veyon doesn't do anything, because it will instantly crash again, as long as the computer room module is open in a browser.
Comment 4 Christina Scheinig univentionstaff 2022-10-06 14:18:48 CEST 
We updated the existing container with the new provided version veyon/webapi-proxy:4.7.4.14
-------------
apt install univention-appcenter-dev
univention-app remove ucsschool-veyon-proxy
docker ps # Prüfen: Is the container really stopped?
univention-app dev-set ucsschool-veyon-proxy "DockerImage=veyon/webapi-proxy:4.7.4.14"
univention-app install ucsschool-veyon-proxy
-----------------

With this new container the container crashes are solved!
Comment 5 Jan-Luca Kiok univentionstaff 2023-03-24 11:29:09 CET 
Asked for clarification if the issue still persists or if Veyon Proxy version 4.7.4.14 solved it.
Comment 6 Jan-Luca Kiok univentionstaff 2023-03-31 16:06:33 CEST 
This bug was reported as resolved with Veyon Proxy version 4.7.4.14.

If this error occurs again, please clone this bug.