Bug 41054 - samba problems with very high number of different xattr operations
samba problems with very high number of different xattr operations
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Kernel
UCS 4.1
Other Linux
: P5 normal (vote)
: UCS 4.1-1-errata
Assigned To: Philipp Hahn
Janek Walkenhorst
:
: 42614 42927 (view as bug list)
Depends on:
Blocks: 42983 42984 43032 43049
  Show dependency treegraph
 
Reported: 2016-04-14 14:38 CEST by Jens Thorp-Hansen
Modified: 2016-11-30 06:29 CET (History)
4 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 7: Crash: Bug causes crash or data loss
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.600
Enterprise Customer affected?: Yes
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Ticket number: 2016091621002544
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jens Thorp-Hansen univentionstaff 2016-04-14 14:38:14 CEST
UCS Version 4.1-1

Ticket#2015091521000246 #24: "NMI watchdog: BUG: soft lockup – CPU#0 stuck for 22s! [smbd:8822]"

https://bugzilla.kernel.org/show_bug.cgi?id=107301
Comment 1 Jens Thorp-Hansen univentionstaff 2016-04-14 14:58:53 CEST
[Citation: https://bugzilla.kernel.org/show_bug.cgi?id=107301#c24]

[281537.032878] Call Trace:
[281537.032882]  [<ffffffffa0117c7c>] ? mb_cache_entry_get+0x1ac/0x1f0 [mbcache]
[281537.032883]  [<ffffffff81208f1f>] ? __find_get_block+0xef/0x120
[281537.032889]  [<ffffffffa033de70>] ? ext4_xattr_block_set+0x80/0xa50 [ext4]
[281537.032890]  [<ffffffff81208f1f>] ? __find_get_block+0xef/0x120
[281537.032896]  [<ffffffffa033d1da>] ? ext4_xattr_set_entrg+0x2a/0x350 [ext4]
[281537.032901]  [<ffffffffa033f3e6>] ? ext4_xattr_set_handle+0x376/0x4d0 [ext4]
[281537.032907]  [<ffffffffa033f614>] ? ext4_xattr_set+0xd4/0x130 [ext4]
[281537.032909]  [<ffffffff811f917e>] ? generic_setxattr+0x6e/0xa0
[281537.032910]  [<ffffffff811f9d91>] ? __ufs_setxattr_noperm+0x71/0x1d0
[281537.032912]  [<ffffffff811f9fb4>] ? ufs_setxattr+0xc4/0xd0
[281537.032914]  [<ffffffff811fa0f4>] ? setxattr+0x134/0x1f0
[281537.032916]  [<ffffffff811e1301>] ? filenane_lookup+0x31/0xd0
[281537.032917]  [<ffffffff811e50bc>] ? user_path_at_emptg+0x60/0xc0
[281537.032918]  [<ffffffff811d7473>] ? __sb_start_urite+0x53/0x100
[281537.032919]  [<ffffffff811fa240>] ? path_setxattr+0x90/0xc0
[281537.032921]  [<ffffffff811fa314>] ? SgS_setxattr+0x14/0x20
[281537.032922]  [<ffffffff8159de72>] ? system_call_fast_compare_end+0x0/0x6b
[281537.032933] Code: f0 0f c1 07 89 02 c1 ea 10 66 39 02 75 01 03 0f b7 f2 b8 00 80 00 00 0f b7 0f 41 89 08 41 31 d0 41 81 e0 f
e ff 00 00 74 10 f3 90 <83> e8 01 75 e7 0f 1f 80 00 00 00 00 eb d9 0f b7 f1 e8 08 75 ff
Comment 2 Stefan Gohmann univentionstaff 2016-04-14 15:07:56 CEST
As discussed, please build a test kernel with a deactivated mbcache and announce the scope as an test kernel scope. If it works in the customer environment, we should provide the patch as erratum.
Comment 3 Philipp Hahn univentionstaff 2016-04-15 12:45:42 CEST
r16409 | Bug #41054: linux ext4-mbcache

Package: linux
Version: 4.1.6-1.190.201604142226
Branch: ucs_4.1-0
Scope: glibc


The "Filesystem Meta Information Block Cache (mbcache)" is an in-memory cache used to minimize the number by identical EAs used for different files. It can be disabled by mounting the (ext4) file system with the additional option "no_mbcache".

One-time:
  mount -o remount,no_mbcache /

Permanent:
  cp /etc/fstab /etc/fstab.orig && awk '$3=="ext4"{$4=$4.",no_mbcache"}{print}' /etc/fstab.orig >/etc/fstab

You can check is the option is enabled for a file system (FS):
 # grep no_mbcache /proc/fs/ext4/*/options 
 /proc/fs/ext4/dm-0/options:no_mbcache

Small "Extended Attributes (EA)" are stored within the inode, if they fit. Check the "Inode size":

 # tune2fs -l /dev/dm-0 | grep '^Inode size'
 Inode size:               256


Otherwise additional blocks are allocated:

 # PAGER=cat debugfs /dev/dm-0 -R 'stat /etc/nagios-plugins/config/ping.cfg'
...
 File ACL: 8405586    Directory ACL: 0
           ^^^^^^^ this is the block used for EAs. ACLs are also one form of EA.
...
 Extended attributes stored in inode body: 
   date = "/etc/nagios-plugins/config" (26)
   ^^^^ This is an in-inode-EA

 # setfattr -n user.test -v "$(seq 100)" /etc/nagios-plugins/config/ping.cfg
 # debugfs /dev/dm-0 -R bd\ 8405586
 0000  0000 02ea 0100 0000 0100 0000 043e 199d  .............>..
 0020  0000 0000 0000 0000 0000 0000 0000 0000  ................
 0040  0401 dc0e 0000 0000 2301 0000 043e 199d  ........#....>..
 0060  7465 7374 0000 0000 0000 0000 0000 0000  test............
 0100  0000 0000 0000 0000 0000 0000 0000 0000  ................
 *
 7320  0000 0000 0000 0000 0000 0000 310a 320a  ............1.2.
...
 7760  3936 0a39 370a 3938 0a39 390a 3130 3000  96.97.98.99.100.

e2fsprogs in UCS-4 is too old to support more detailed EA debug output. With <https://packages.debian.org/jessie/e2fsprogs> all external EAs are shown as well:
 # PAGER=cat debugfs /dev/dm-0 -R 'ea_list /etc/nagios-plugins/config/ping.cfg'
...

There is an upper limit for EAs: <http://man7.org/linux/man-pages/man7/xattr.7.html>
> In the current ext2, ext3, and ext4 filesystem implementations, the
> total bytes used by the names and values of all of a files extended
> attributes must fit in a single filesystem block (1024, 2048 or 4096
> bytes, depending on the block size specified when the filesystem was
> created).
 # tune2fs -l /dev/dm-0  | grep '^Block size:'
 Block size:               4096

<https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes>


$ announce_ucs_scope.py -r 4.1-0 -s glibc -n --skip-tag -u -k /etc/archive-keys/ucs4.0.txt -K 6B8BFD3C
<http://univention-repository.knut.univention.de/4.1/unmaintained/component/glibc/>

OK: amd64 @ kvm
TODO: QA
TODO: ssh omar sudo update_mirror.sh -v ftp/4.1/unmaintained/component/glibc
Comment 4 Stefan Gohmann univentionstaff 2016-04-15 13:15:20 CEST
(In reply to Philipp Hahn from comment #3)
> TODO: ssh omar sudo update_mirror.sh -v ftp/4.1/unmaintained/component/glibc

Done.

@Jens, please report if the new kernel solves the problem.
Comment 5 Jens Thorp-Hansen univentionstaff 2016-04-27 14:43:43 CEST
Customer reports: the new kernel seems to work (no_mbcache)
Comment 6 Philipp Hahn univentionstaff 2016-04-27 19:07:19 CEST
Package: linux
Version: 4.1.6-1.190.201604142226
Branch: ucs_4.1-0-glibc
Scope: errata4.1-1

r68975 | Bug #41054 kernel: Update to ucs190
r68977 | Bug #41054 kernel: Update to ucs190

Package: univention-kernel-image-signed
Version: 2.0.0-7.18.201604271902
Branch: ucs_4.1-0
Scope: errata4.1-1

r68976 | Bug #41054 kernel: Add option to disable mbcache

Package: univention-kernel-image
Version: 9.0.0-10.101.201604271859
Branch: ucs_4.1-0
Scope: errata4.1-1

r68978 | Bug #41054 kernel: Add option to disable mbcache YAML
 linux.yaml
 univention-kernel-image-signed.yaml
 univention-kernel-image.yaml
Comment 7 Nico Stöckigt univentionstaff 2016-04-28 09:23:33 CEST
Also happend here with UCS 4.0-5 (Ticket#2016042521000332)
Comment 8 Janek Walkenhorst univentionstaff 2016-04-29 17:53:17 CEST
Tests: Hardware, UEFI, Secureboot, KVM (i386, amd64): OK
Code review: OK
Comment 10 Philipp Hahn univentionstaff 2016-11-28 09:09:35 CET
*** Bug 42614 has been marked as a duplicate of this bug. ***
Comment 11 Stefan Gohmann univentionstaff 2016-11-30 06:29:18 CET
*** Bug 42927 has been marked as a duplicate of this bug. ***