Bug 32327 - max inotify childs to small for midsize/large environments
max inotify childs to small for midsize/large environments
Product: UCS
Classification: Unclassified
Component: General
UCS 3.1
Other Linux
: P3 critical (vote)
: UCS 3.1-1-errata
Assigned To: Erik Damrose
Philipp Hahn
Depends on: 24165
  Show dependency treegraph
Reported: 2013-08-23 09:43 CEST by Ingo Steuwer
Modified: 2013-08-28 13:42 CEST (History)
7 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:

inotify test script (2.50 KB, text/plain)
2013-08-27 17:03 CEST, Philipp Hahn

Note You need to log in before you can comment on or make changes to this bug.
Description Ingo Steuwer univentionstaff 2013-08-23 09:43:33 CEST
seen on 2013081921003867, 2013052221001635, 2013031421001851, 2013012221000698 and maybe others

The behaviour described below is not a reached "max open files" but a "max inotify users/watchlist". Defaults in 3.1 are:

$ cat /proc/sys/fs/inotify/max_user_instances /proc/sys/fs/inotify/max_user_watches /proc/sys/fs/inotify/max_queued_events 

This needs to be increased and should be made configurable by UCR.

Know issues related to this are:
- samba-failures like in the cloned bug
- failing UMC module processes (fails to open the socket files for communication)
- error-messages from "tail" (tail: inotify cannot be used, reverting to polling: Too many open files)

+++ This bug was initially created as a clone of Bug #24165 +++

Ticket#: 2011053110000545 
Ticket#: 2011102421003249

Aus der log.smbd:
2011/05/31 10:50:38.822505,  0] smbd/notify_inotify.c:294(inotify_setup)
   Failed to init inotify - Too many open files

[2011/10/24 10:48:45.016578,  0] smbd/notify_inotify.c:294(inotify_setup)
    Failed to init inotify - Zu viele offene Dateien

Samba setzt die Limits beim Start selbst anhand der Konfigurationsoption "max open files" (minimal/default 16404).
# pgrep -f smbd | while read pid; do grep "open files" /proc/$pid/limits; done

Um das Limit zu verdoppeln kann die "/etc/samba/local.conf" erweitert werden:
   max open files = 32808
Comment 1 Florian Best univentionstaff 2013-08-23 09:55:18 CEST
- failing UMC module processes (fails to open the socket files for communication)
→ Bug #31752
Comment 2 Erik Damrose univentionstaff 2013-08-26 17:06:37 CEST
default values of some distros:
/proc/sys/fs/inotify/max_user_instances /proc/sys/fs/inotify/max_user_watches /proc/sys/fs/inotify/max_queued_events 

ubuntu 12.04.1 server:  128, 8192, 16384

kubuntu 12.04.2:  128, 524288, 16384 (https://bugs.launchpad.net/ubuntu/+source/procps/+bug/879902)

fedora 19:  128, 8192 ,16384
Comment 3 Erik Damrose univentionstaff 2013-08-27 11:40:27 CEST
Our support seems to get good results with the following values, which are now default values for the following new UCR Variables:
* kernel/fs/inotify/max_user_instances (default:   512)
* kernel/fs/inotify/max_user_watches   (default: 32768)
* kernel/fs/inotify/max_queued_events  (default: 16384)

The mentioned support tickets describe problems relating to too low values for kernel/fs/inotify/max_user_watches. For the new default values a conversative approach was used, as each new user-watch takes about 1KB of kernel memory.

new packages have been built for errata3.1-1:
univention-base-files 2.0.15-3.142.201308271124

for UCS 3.2:
univention-base-files 3.0.3-1.143.201308271132
Changelog entry added
Comment 4 Erik Damrose univentionstaff 2013-08-27 14:23:17 CEST
Yaml: 2013-08-23-univention-base-files.yaml
Comment 5 Philipp Hahn univentionstaff 2013-08-27 17:03:03 CEST
OK: 3.2-0 svn43464
OK: 3.2-0 ChangeLog svn43465
OK: univention-base-files_2.0.15-3.142.201308271124

OK: 3.1-1 svn43466
OK: 3.1-1 svn43468,43477
  needs to be merges into 2013-08-23-univention-base-files.yaml
OK: univention-base-files_3.0.3-1.143.201308271132

OK: 3.1-1 → erratum → 3.2-0
OK: 3.1-1 → 3.2-0

OK: ucr set kernel/fs/inotify/max_user_instances=511
OK: ucr set kernel/fs/inotify/max_user_watches=32767
OK: ucr set  kernel/fs/inotify/max_queued_events=16383

OK: changing the settings fixed the problems described in the tickets
Comment 6 Philipp Hahn univentionstaff 2013-08-27 17:03:38 CEST
Created attachment 5398 [details]
inotify test script
Comment 7 Moritz Muehlenhoff univentionstaff 2013-08-28 13:42:57 CEST