Bug 35866 - Too many open files messages in samba logfiles
Too many open files messages in samba logfiles
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC - System diagnostic
UCS 3.2
Other Linux
: P5 enhancement (vote)
: UCS 4.0
Assigned To: Florian Best
Felix Botner
: interim-3
: 36345 (view as bug list)
Depends on: 35868
Blocks:
  Show dependency treegraph
 
Reported: 2014-09-10 08:54 CEST by Janis Meybohm
Modified: 2017-06-06 14:09 CEST (History)
4 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Janis Meybohm univentionstaff 2014-09-10 08:54:59 CEST
Add a plugin that checks the last samba logfiles for "Too many open files" messages.

If found, show some information on how the determine the cause and increase the limits: 
http://forum.univention.de/viewtopic.php?f=48&t=2100
https://forge.univention.org/bugzilla/show_bug.cgi?id=32415

Maybe we could/should add an SDB article to which the plugin could redirect.
Comment 1 Stefan Gohmann univentionstaff 2014-09-10 09:08:08 CEST
(In reply to Janis Meybohm from comment #0)
> Add a plugin that checks the last samba logfiles for "Too many open files"
> messages.
> 
> If found, show some information on how the determine the cause and increase
> the limits: 
> http://forum.univention.de/viewtopic.php?f=48&t=2100
> https://forge.univention.org/bugzilla/show_bug.cgi?id=32415
> 
> Maybe we could/should add an SDB article to which the plugin could redirect.

Yes: Bug #35868.
Comment 2 Florian Best univentionstaff 2014-09-18 14:56:25 CEST
Which logfiles can contain the error?
Only /var/log/samba/log.smbd? Or every logfile in /var/log/samba? Other logfiles?

How often can the string occur so that the message is irrelevant?
Comment 3 Janis Meybohm univentionstaff 2014-09-18 15:41:25 CEST
It's most likely in /var/log/samba/log.smbd but every log may contain the message (meaning different things and requires different steps to fix of cause).

See http://forum.univention.de/viewtopic.php?f=48&t=2464#p8650 for an example

"Too many open files, unable to open more! smbd's max open files = 16404"
Comment 4 Florian Best univentionstaff 2014-10-02 17:47:54 CEST
The plugin has been added as security_limits.py. It has the following output:

Critical: Security limits exceeded
The security limits (e.g. for max_open_files) are currently not configured properly.
This can cause several different serious problems (e.g. the login at samba servers may be impossible, file operations (copy, move) on shares can fail, etc.)
It is suggested to increase the security limits either manually by using "Univention Configuration Registry" module or to use the automatically suggested limits:
security/limits/user/*/soft/nofile=32828
security/limits/user/*/hard/nofile=32828
[Adjust to suggested limits]

The plugin will be marked as failed if the string "Too many open files" occurs at least once in /var/log/samba/log.smbd and the limits are smaller than 32828.
Comment 5 Janis Meybohm univentionstaff 2014-10-06 07:55:25 CEST
Depending on samba version there might also be a samba specific max_open_files option which needs to be adjusted:
"samba/max_open_files"
Comment 6 Florian Best univentionstaff 2014-10-06 11:17:58 CEST
(In reply to Janis Meybohm from comment #5)
> Depending on samba version there might also be a samba specific
> max_open_files option which needs to be adjusted:
> "samba/max_open_files"
Also with the value 32828?
Comment 7 Arvid Requate univentionstaff 2014-10-15 13:31:41 CEST
Some additional points:

* limits.conf only applys to pam logins, not to things started from init
* Our default UCR limits.conf settings for "user" don't apply to root
* Both soft and hard limits can be set with "-", see bug 27709
* Starting with errata Bug 34514 we use ulimit in the samba-ad-dc init script
  samba/max_open_files is used for this. Thus, all the security/limits/* stuff
  doesn't apply any longer, I guess. Maybe I would restrict the recommendation
  to samba/max_open_files.


* The value of "32828" seems like a pretty arbitrary number,
  in the recommendation message I would suggest 32768.
* For our peculiar numerical default for samba/max_open_files
  see Bug 34514 Comment 6
* Recently the samba default has been raised to 16384 (affects UCS 4.0).
* For the actual maximum value samba takes the minimum of system supplied limits and it's own "max open files" limit.

* See bug 32327 for inotify limits relevant for samba. Don't know if this can
  be covered here as well. If not, a separate bug should be created.
Comment 8 Stefan Gohmann univentionstaff 2014-10-19 19:52:47 CEST
Move all unfinished MS1 and MS2 bugs to RC.
Comment 9 Stefan Gohmann univentionstaff 2014-10-29 06:18:16 CET
(In reply to Florian Best from comment #6)
> (In reply to Janis Meybohm from comment #5)
> > Depending on samba version there might also be a samba specific
> > max_open_files option which needs to be adjusted:
> > "samba/max_open_files"
> Also with the value 32828?

Florian, I think you can split it into a new bug.
Comment 10 Florian Best univentionstaff 2014-10-29 22:34:34 CET
Bug #36245 was created for the other things. For now closing it with the current state.
Comment 11 Felix Botner univentionstaff 2014-10-30 13:10:08 CET
* maybe better to read the log.smbd line pre line (log.smbd can become very big)

  with open("log.txt") as infile:
      for line in infile:
          do_something_with(line)

* The test for security/limits/user/*/soft|hard/nofile makes no sence for samba
  since 3.2-3 samba3/samba4 uses samba/max_open_files 

  please replace 'security/limits/user/*/soft/nofile' resp.
  'security/limits/user/*/hard/nofile' with samba/max_open_files
Comment 12 Florian Best univentionstaff 2014-11-03 11:47:39 CET
(In reply to Felix Botner from comment #11)
> * maybe better to read the log.smbd line pre line (log.smbd can become very
> big)
yes, it is now read per line.

> * The test for security/limits/user/*/soft|hard/nofile makes no sence for
> samba
samba/max_open_files with default of 32768 is now used instead.
Comment 13 Felix Botner univentionstaff 2014-11-03 12:07:18 CET
(In reply to Florian Best from comment #12)
> (In reply to Felix Botner from comment #11)
> > * maybe better to read the log.smbd line pre line (log.smbd can become very
> > big)
> yes, it is now read per line.
> 
> > * The test for security/limits/user/*/soft|hard/nofile makes no sence for
> > samba
> samba/max_open_files with default of 32768 is now used instead.

our default is samba/max_open_files?32808, so better check this value (or should we increase our default in univention-samba(4)?)
Comment 14 Florian Best univentionstaff 2014-11-03 16:44:42 CET
(In reply to Felix Botner from comment #13)
The default has been adapted.
Comment 15 Felix Botner univentionstaff 2014-11-03 16:57:00 CET
Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/__init__.py", line 245, in execute result = execute(**kwargs) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/security_limits.py", line 59, in adjust 'samba/max_open_files=%d' % (suggested_limit_hard,) NameError: global name 'suggested_limit_hard' is not defined
Comment 16 Florian Best univentionstaff 2014-11-03 17:57:18 CET
fixed typo
Comment 17 Felix Botner univentionstaff 2014-11-05 13:28:03 CET
OK
Comment 18 Stefan Gohmann univentionstaff 2014-11-26 06:54:15 CET
UCS 4.0-0 has been released:
 http://docs.univention.de/release-notes-4.0-0-en.html
 http://docs.univention.de/release-notes-4.0-0-de.html

If this error occurs again, please use "Clone This Bug".
Comment 19 Lukas Oyen univentionstaff 2017-06-06 14:09:32 CEST
*** Bug 36345 has been marked as a duplicate of this bug. ***