Bug 37453 - [Errno 24] Too many open files
[Errno 24] Too many open files
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 4.0
Other Linux
: P5 normal (vote)
: UCS 4.0-0-errata
Assigned To: Florian Best
Johannes Keiser
:
Depends on:
Blocks: 43615
  Show dependency treegraph
 
Reported: 2015-01-06 09:24 CET by Alexander Kläser
Modified: 2017-02-22 17:34 CET (History)
2 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Error handling
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kläser univentionstaff 2015-01-06 09:24:10 CET
The following error occurred on our demo system, a login was not possible anymore. After restarting the UMC server component, the login was fine.

root@master40:~# tail -n30 /var/log/univention/management-console-server.log
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | reboot/*             | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | services/*           | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | setup/*              | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | sysinfo/*            | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | top/*                | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | ucr/*                | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | udm/*                | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | udm/request_new_license | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :  False | master40
       | *               | updater/*            | {}
05.01.15 19:03:49.157  ACL         ( PROCESS ) :
05.01.15 19:03:49.294  MAIN        ( ERROR   ) : Traceback (most recent
call last):
  File "/usr/sbin/univention-management-console-server", line 209, in
<module>
  File "/usr/lib/pymodules/python2.7/daemon/runner.py", line 186, in
do_action
  File "/usr/sbin/univention-management-console-server", line 142, in
_restart
  File "/usr/lib/pymodules/python2.7/daemon/runner.py", line 131, in _start
  File "/usr/sbin/univention-management-console-server", line 192, in run
  File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 284,
in loop
  File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 271,
in step
  File
"/usr/lib/pymodules/python2.7/univention/management/console/protocol/server.py",
line 168, in _receive
  File
"/usr/lib/pymodules/python2.7/univention/management/console/protocol/server.py",
line 286, in _handle
  File
"/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py",
line 287, in request
  File
"/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py",
line 452, in handle_request_get
  File
"/usr/lib/pymodules/python2.7/univention/management/console/locales.py",
line 180, in _
  File
"/usr/lib/pymodules/python2.7/univention/management/console/locales.py",
line 78, in __init__
  File
"/usr/lib/pymodules/python2.7/univention/management/console/locales.py",
line 108, in load
  File "/usr/lib/python2.7/dist-packages/polib.py", line 141, in mofile
  File "/usr/lib/python2.7/dist-packages/polib.py", line 64, in
_pofile_or_mofile
  File "/usr/lib/python2.7/dist-packages/polib.py", line 187, in
detect_encoding
IOError: [Errno 24] Zu viele offene Dateien:
'/usr/share/univention-management-console/i18n/de/mrtg.mo'
Comment 1 Alexander Kläser univentionstaff 2015-01-06 09:47:03 CET
The reason for this is that sockets are not closed properly. DIFFing the output of two lsof outputs of the UMC server process reveals the following:

> --- lsof.01060930.txt   2015-01-06 09:30:40.858160000 +0100
> +++ lsof.01060942.txt   2015-01-06 09:42:17.926160000 +0100
> [...]
> @@ -386,24 +387,24 @@
>  univentio 10667 root  224u  sock    0,7      0t0  358638 can't identify protocol
>  univentio 10667 root  225u  sock    0,7      0t0  360111 can't identify protocol
>  univentio 10667 root  226u  sock    0,7      0t0  369637 can't identify protocol
> -univentio 10667 root  227u  IPv4 377569      0t0     TCP master40.demo.univention.de:47041->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  228u  IPv4 430599      0t0     TCP master40.demo.univention.de:51338->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  229u  IPv4 431419      0t0     TCP master40.demo.univention.de:51414->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  230u  IPv4 377733      0t0     TCP master40.demo.univention.de:47063->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  231u  IPv4 377824      0t0     TCP master40.demo.univention.de:47066->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  232u  IPv4 383773      0t0     TCP master40.demo.univention.de:47554->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  233u  IPv4 387087      0t0     TCP master40.demo.univention.de:47819->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  234u  IPv4 390613      0t0     TCP master40.demo.univention.de:48089->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  235u  IPv4 406370      0t0     TCP master40.demo.univention.de:49360->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  236u  IPv4 407996      0t0     TCP master40.demo.univention.de:49506->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  237u  IPv4 409695      0t0     TCP master40.demo.univention.de:49648->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  238u  IPv4 419913      0t0     TCP master40.demo.univention.de:50454->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  239u  IPv4 427937      0t0     TCP master40.demo.univention.de:51119->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  240u  IPv6 963600      0t0     TCP localhost:6670->localhost:50312 (ESTABLISHED)
> +univentio 10667 root  227u  sock                0,7      0t0  377569 can't identify protocol
> +univentio 10667 root  228u  sock                0,7      0t0  430599 can't identify protocol
> +univentio 10667 root  229u  sock                0,7      0t0  431419 can't identify protocol
> +univentio 10667 root  230u  sock                0,7      0t0  377733 can't identify protocol
> +univentio 10667 root  231u  sock                0,7      0t0  377824 can't identify protocol
> +univentio 10667 root  232u  sock                0,7      0t0  383773 can't identify protocol
> +univentio 10667 root  233u  sock                0,7      0t0  387087 can't identify protocol
> +univentio 10667 root  234u  sock                0,7      0t0  390613 can't identify protocol
> +univentio 10667 root  235u  sock                0,7      0t0  406370 can't identify protocol
> +univentio 10667 root  236u  sock                0,7      0t0  407996 can't identify protocol
> +univentio 10667 root  237u  sock                0,7      0t0  409695 can't identify protocol
> +univentio 10667 root  238u  sock                0,7      0t0  419913 can't identify protocol
> +univentio 10667 root  239u  sock                0,7      0t0  427937 can't identify protocol
> +univentio 10667 root  240u  IPv6            1015404      0t0     TCP localhost:6670->localhost:54525 (ESTABLISHED)
>  univentio 10667 root  241u  IPv4 480338      0t0     TCP master40.demo.univention.de:55404->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  242u  IPv4 428159      0t0     TCP master40.demo.univention.de:51140->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  243u  IPv4 428177      0t0     TCP master40.demo.univention.de:51143->master40.demo.univention.de:7389 (CLOSE_WAIT)
> -univentio 10667 root  244u  IPv4 433869      0t0     TCP master40.demo.univention.de:51616->master40.demo.univention.de:7389 (CLOSE_WAIT)
> +univentio 10667 root  242u  sock                0,7      0t0  428159 can't identify protocol
> +univentio 10667 root  243u  sock                0,7      0t0  428177 can't identify protocol
> +univentio 10667 root  244u  sock                0,7      0t0  433869 can't identify protocol
>  univentio 10667 root  245u  IPv4 437184      0t0     TCP master40.demo.univention.de:51886->master40.demo.univention.de:7389 (CLOSE_WAIT)
>  univentio 10667 root  246u  IPv4 440544      0t0     TCP master40.demo.univention.de:52166->master40.demo.univention.de:7389 (CLOSE_WAIT)
>  univentio 10667 root  247u  IPv4 456082      0t0     TCP master40.demo.univention.de:53442->master40.demo.univention.de:7389 (CLOSE_WAIT)
> [...]
Comment 2 Alexander Kläser univentionstaff 2015-01-06 10:12:12 CET
master40.demo.univention.de:7389
→ these connections seem to be connections to the LDAP server
Comment 3 Florian Best univentionstaff 2015-01-06 15:02:54 CET
Never trust __del__() to be called, It never has been! Probably because notifier.Callback has a reference to a State() instance (but even more, and I didn't want to call gc.collect()).
Fix: svn r57100
YAML: 2014-12-04-univention-management-console.yaml

Reproduce:
invoke-rc.d univention-management-console-server restart
lsof -p $(pgrep -f /usr/sbin/univention-management-console-server)
umc-command -U Administrator -P univention ucr/get > /dev/null
umc-command -U Administrator -P univention ucr/get > /dev/null
lsof -p $(pgrep -f /usr/sbin/univention-management-console-server)
lsof -p $(pgrep -f /usr/sbin/univention-management-console-server) | grep -e TCP -e 'identify protocol'
Comment 4 Johannes Keiser univentionstaff 2015-01-12 15:25:28 CET
Fix: OK
YAML: OK
Comment 5 Janek Walkenhorst univentionstaff 2015-01-22 11:54:59 CET
<http://errata.univention.de/ucs/4.0/44.html>