Bug 51729 - UMC-Web-Server is limited to 1024 file descriptors
UMC-Web-Server is limited to 1024 file descriptors
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-5-errata
Assigned To: Julia Bremer
Felix Botner
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-07-24 14:21 CEST by Sönke Schwardt-Krummrich
Modified: 2020-08-05 15:15 CEST (History)
10 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.286
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support: Yes
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:
best: Patch_Available+


Attachments
patch (git:fbest/51729-webserver-open-file-limits) (1.07 KB, patch)
2020-08-03 11:45 CEST, Florian Best
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2020-07-24 14:21:42 CEST
The UMC web server is started with a ulimit of 1024 file descriptors. In large environments with many open sessions through UMC/Univention Portal, this means that no more sessions can be opened.

In the test the limit for the process was raised to 65535 and it was no problem to open 2000 parallel SAML sessions at a rate of 1-2 SAML logins per second.

We should raise the limit and make it configurable via a UCR variable.
Comment 1 Sönke Schwardt-Krummrich univentionstaff 2020-07-24 14:23:29 CEST
1024 is the soft limit, 4096 the hard limit.
With 2000 sessions there were around 4400 open file descriptors. So maybe we should also raise the hard limit.
Comment 2 Jürn Brodersen univentionstaff 2020-07-24 14:37:31 CEST
Workraround: increase allowed number of open files for all


mkdir /etc/systemd/system.conf.d/
In: "/etc/systemd/system.conf.d/limits.conf"
```
[Manager]
DefaultLimitNOFILE=65535
```
Dienste neustarten:
```
systemctl daemon-reexec
systemctl restart univention-management-console-web-server.service
systemctl restart univention-management-console-server.service
```
Comment 4 Florian Best univentionstaff 2020-07-27 13:10:02 CEST
We already documented how to set the ulimt for open files per process in our perforemance documentation?!:
https://docs.software-univention.de/performance-guide-4.4.html#kernel
Comment 5 Erik Damrose univentionstaff 2020-07-27 15:05:17 CEST
Type in original bug title and bug description: The file descriptor limit for the UMC server, not the web server, should be raised
Comment 6 Jürn Brodersen univentionstaff 2020-08-03 11:17:17 CEST
(In reply to Erik Damrose from comment #5)
> Type in original bug title and bug description: The file descriptor limit
> for the UMC server, not the web server, should be raised

My bad, web server was right. The umc server already has the limit increased.

@Assignee 
Please see "management/univention-management-console/scripts/univention-management-console-server":208
`resource.setrlimit(resource.RLIMIT_NOFILE, (64512, 64512))`
and implement something similar in "management/univention-management-console/univention-management-console-web-server"
Comment 7 Florian Best univentionstaff 2020-08-03 11:45:32 CEST
Created attachment 10446 [details]
patch (git:fbest/51729-webserver-open-file-limits)
Comment 8 Julia Bremer univentionstaff 2020-08-04 10:10:40 CEST
6b6584877c Bug #51729: Restart umc-web-server after test
182bde1a5a Bug #51729: Adjust test description
466e7c1fa2 Bug #51729: Add UCR Variable to set the limit of file descriptors in UMC-Web-Server
b1d4bb0a0f Bug #51279: yaml

------------------------------------------
Successful build
Package: univention-management-console
Version: 11.0.4-99A~4.4.0.202008032042
Branch: ucs_4.4-0
Scope: errata4.4-5
------------------------------------------

I raised the default file_descriptor limit in the umc-web-server to 65535 
and added a test case. 

TODO: Documentation
Comment 9 Julia Bremer univentionstaff 2020-08-04 12:38:15 CEST
I renamed the ucr variable, since Florian had a better suggestion

48b5d4c815 Bug #51729: rename UCR variable per florians request

Successful build
Package: univention-management-console
Version: 11.0.4-101A~4.4.0.202008041231
Branch: ucs_4.4-0
Scope: errata4.4-5
Comment 10 Julia Bremer univentionstaff 2020-08-04 13:09:25 CEST
902c2c40e8 Bug #51729: Add documentation
Comment 11 Felix Botner univentionstaff 2020-08-04 15:59:05 CEST
FAILED - yaml (version and umc/max/file_descriptors -> umc/http/max-open-file-descriptors)

FAILED - ucs-test http://jenkins.knut.univention.de:8080/job/UCS-4.4/job/UCS-4.4-5/job/AutotestUpgrade/SambaVersion=s4,Systemrolle=master-part-II/lastCompletedBuild/testReport/86_selenium/13_umc_max_fd/

OK - UCRV desc
OK - max open files after update

-> cat /proc/$(pidof -x /usr/sbin/univention-management-console-web-server)/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             11867                11867                processes 
Max open files            65535                65535                files

OK - changing UCRV

-> ucr set umc/http/max-open-file-descriptors='1024' 

-> service univention-management-console-web-server restart
-> cat /proc/$(pidof -x /usr/sbin/univention-management-console-web-server)/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             11867                11867                processes 
Max open files            1024                 1024                 files

OK - documentation
Comment 12 Julia Bremer univentionstaff 2020-08-05 08:49:30 CEST
7d8b21c618 Bug #51729: Print exception, stabilize
17c4b781a1 Bug #51729: yaml update

The test did not fail again.
Comment 13 Felix Botner univentionstaff 2020-08-05 10:09:19 CEST
OK - yaml
OK - test