Univention Bugzilla – Bug 41054
samba problems with very high number of different xattr operations
Last modified: 2016-11-30 06:29:18 CET
UCS Version 4.1-1 Ticket#2015091521000246 #24: "NMI watchdog: BUG: soft lockup – CPU#0 stuck for 22s! [smbd:8822]" https://bugzilla.kernel.org/show_bug.cgi?id=107301
[Citation: https://bugzilla.kernel.org/show_bug.cgi?id=107301#c24] [281537.032878] Call Trace: [281537.032882] [<ffffffffa0117c7c>] ? mb_cache_entry_get+0x1ac/0x1f0 [mbcache] [281537.032883] [<ffffffff81208f1f>] ? __find_get_block+0xef/0x120 [281537.032889] [<ffffffffa033de70>] ? ext4_xattr_block_set+0x80/0xa50 [ext4] [281537.032890] [<ffffffff81208f1f>] ? __find_get_block+0xef/0x120 [281537.032896] [<ffffffffa033d1da>] ? ext4_xattr_set_entrg+0x2a/0x350 [ext4] [281537.032901] [<ffffffffa033f3e6>] ? ext4_xattr_set_handle+0x376/0x4d0 [ext4] [281537.032907] [<ffffffffa033f614>] ? ext4_xattr_set+0xd4/0x130 [ext4] [281537.032909] [<ffffffff811f917e>] ? generic_setxattr+0x6e/0xa0 [281537.032910] [<ffffffff811f9d91>] ? __ufs_setxattr_noperm+0x71/0x1d0 [281537.032912] [<ffffffff811f9fb4>] ? ufs_setxattr+0xc4/0xd0 [281537.032914] [<ffffffff811fa0f4>] ? setxattr+0x134/0x1f0 [281537.032916] [<ffffffff811e1301>] ? filenane_lookup+0x31/0xd0 [281537.032917] [<ffffffff811e50bc>] ? user_path_at_emptg+0x60/0xc0 [281537.032918] [<ffffffff811d7473>] ? __sb_start_urite+0x53/0x100 [281537.032919] [<ffffffff811fa240>] ? path_setxattr+0x90/0xc0 [281537.032921] [<ffffffff811fa314>] ? SgS_setxattr+0x14/0x20 [281537.032922] [<ffffffff8159de72>] ? system_call_fast_compare_end+0x0/0x6b [281537.032933] Code: f0 0f c1 07 89 02 c1 ea 10 66 39 02 75 01 03 0f b7 f2 b8 00 80 00 00 0f b7 0f 41 89 08 41 31 d0 41 81 e0 f e ff 00 00 74 10 f3 90 <83> e8 01 75 e7 0f 1f 80 00 00 00 00 eb d9 0f b7 f1 e8 08 75 ff
As discussed, please build a test kernel with a deactivated mbcache and announce the scope as an test kernel scope. If it works in the customer environment, we should provide the patch as erratum.
r16409 | Bug #41054: linux ext4-mbcache Package: linux Version: 4.1.6-1.190.201604142226 Branch: ucs_4.1-0 Scope: glibc The "Filesystem Meta Information Block Cache (mbcache)" is an in-memory cache used to minimize the number by identical EAs used for different files. It can be disabled by mounting the (ext4) file system with the additional option "no_mbcache". One-time: mount -o remount,no_mbcache / Permanent: cp /etc/fstab /etc/fstab.orig && awk '$3=="ext4"{$4=$4.",no_mbcache"}{print}' /etc/fstab.orig >/etc/fstab You can check is the option is enabled for a file system (FS): # grep no_mbcache /proc/fs/ext4/*/options /proc/fs/ext4/dm-0/options:no_mbcache Small "Extended Attributes (EA)" are stored within the inode, if they fit. Check the "Inode size": # tune2fs -l /dev/dm-0 | grep '^Inode size' Inode size: 256 Otherwise additional blocks are allocated: # PAGER=cat debugfs /dev/dm-0 -R 'stat /etc/nagios-plugins/config/ping.cfg' ... File ACL: 8405586 Directory ACL: 0 ^^^^^^^ this is the block used for EAs. ACLs are also one form of EA. ... Extended attributes stored in inode body: date = "/etc/nagios-plugins/config" (26) ^^^^ This is an in-inode-EA # setfattr -n user.test -v "$(seq 100)" /etc/nagios-plugins/config/ping.cfg # debugfs /dev/dm-0 -R bd\ 8405586 0000 0000 02ea 0100 0000 0100 0000 043e 199d .............>.. 0020 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0040 0401 dc0e 0000 0000 2301 0000 043e 199d ........#....>.. 0060 7465 7374 0000 0000 0000 0000 0000 0000 test............ 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 7320 0000 0000 0000 0000 0000 0000 310a 320a ............1.2. ... 7760 3936 0a39 370a 3938 0a39 390a 3130 3000 96.97.98.99.100. e2fsprogs in UCS-4 is too old to support more detailed EA debug output. With <https://packages.debian.org/jessie/e2fsprogs> all external EAs are shown as well: # PAGER=cat debugfs /dev/dm-0 -R 'ea_list /etc/nagios-plugins/config/ping.cfg' ... There is an upper limit for EAs: <http://man7.org/linux/man-pages/man7/xattr.7.html> > In the current ext2, ext3, and ext4 filesystem implementations, the > total bytes used by the names and values of all of a files extended > attributes must fit in a single filesystem block (1024, 2048 or 4096 > bytes, depending on the block size specified when the filesystem was > created). # tune2fs -l /dev/dm-0 | grep '^Block size:' Block size: 4096 <https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes> $ announce_ucs_scope.py -r 4.1-0 -s glibc -n --skip-tag -u -k /etc/archive-keys/ucs4.0.txt -K 6B8BFD3C <http://univention-repository.knut.univention.de/4.1/unmaintained/component/glibc/> OK: amd64 @ kvm TODO: QA TODO: ssh omar sudo update_mirror.sh -v ftp/4.1/unmaintained/component/glibc
(In reply to Philipp Hahn from comment #3) > TODO: ssh omar sudo update_mirror.sh -v ftp/4.1/unmaintained/component/glibc Done. @Jens, please report if the new kernel solves the problem.
Customer reports: the new kernel seems to work (no_mbcache)
Package: linux Version: 4.1.6-1.190.201604142226 Branch: ucs_4.1-0-glibc Scope: errata4.1-1 r68975 | Bug #41054 kernel: Update to ucs190 r68977 | Bug #41054 kernel: Update to ucs190 Package: univention-kernel-image-signed Version: 2.0.0-7.18.201604271902 Branch: ucs_4.1-0 Scope: errata4.1-1 r68976 | Bug #41054 kernel: Add option to disable mbcache Package: univention-kernel-image Version: 9.0.0-10.101.201604271859 Branch: ucs_4.1-0 Scope: errata4.1-1 r68978 | Bug #41054 kernel: Add option to disable mbcache YAML linux.yaml univention-kernel-image-signed.yaml univention-kernel-image.yaml
Also happend here with UCS 4.0-5 (Ticket#2016042521000332)
Tests: Hardware, UEFI, Secureboot, KVM (i386, amd64): OK Code review: OK
<http://errata.software-univention.de/ucs/4.1/166.html> <http://errata.software-univention.de/ucs/4.1/169.html> <http://errata.software-univention.de/ucs/4.1/162.html>
*** Bug 42614 has been marked as a duplicate of this bug. ***
*** Bug 42927 has been marked as a duplicate of this bug. ***