A customer has updated his system and installed kernel 5.10. Since then he has had problems with the Sysvol sync and missing ACLs after the files have been successfully synced. Due to the rsync, the files lose their ACLs and can therefore not be used. The following error message can be seen in the sysvol.log: 2024-08-18 00:20:48 ERROR [dc02] some files from dc02 don't have ACLs set. Will not sync to hot target! The interesting thing about the customer environment is that the DC master has formatted the file system in xfs. With filesystems installed as ext4, these rsync errors do not occur with kernel 5.10. In our tests, we therefore booted an earlier kernel 4.19 and performed another rsync from xfs to ext4 and from xfs to xfs. No errors occurred and all files had the ACL's set correctly.
Informations from our testings and my test environment. It seems that the kernel is to blame for the ACL's being lost when rsyncing to the xfs filesystem. On the primary I ran rsync -aAX /var/lib/samba/sysvol /mnt/ and the files no longer had ACL's set. root@ucs5primary:/mnt# xfs_info /mnt/ meta-data=/dev/sdb1 isize=512 agcount=4, agsize=131072 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 data = bsize=4096 blocks=524288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =Internes Protokoll bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =keine extsz=4096 blocks=0, rtextents=0 root@ucs5primary:/mnt# df -T Dateisystem Typ 1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf udev devtmpfs 4063772 0 4063772 0% /dev tmpfs tmpfs 817036 16056 800980 2% /run /dev/mapper/vg_ucs-root ext4 19746772 6279096 12589636 34% / tmpfs tmpfs 4085176 0 4085176 0% /dev/shm tmpfs tmpfs 5120 0 5120 0% /run/lock tmpfs tmpfs 4085176 0 4085176 0% /sys/fs/cgroup /dev/sda1 ext4 472408 201974 241404 46% /boot /dev/sdb1 xfs 2086912 35252 2051660 2% /mnt Loaded Kernel 5.10.0-0.deb10.30-amd64. root@ucs5primary:~# rsync -aAX /mnt/sysvol/univention.intranet/Policies/ root@ucs5backup:/mnt/ root@ucs5primary:~# cd /mnt/ root@ucs5primary:/mnt# ls -lah insgesamt 8,0K drwxr-xr-x 3 root root 20 Aug 16 15:22 . drwxr-xr-x 18 root root 4,0K Aug 16 15:30 .. drwxrwx--- 3 Administrator Administrators 33 Aug 16 14:48 sysvol root@ucs5primary:/mnt# rm -rf sysvol/ root@ucs5primary:/mnt# ls -lah insgesamt 4,0K drwxr-xr-x 2 root root 6 Aug 16 15:46 . drwxr-xr-x 18 root root 4,0K Aug 16 15:30 .. After reboot with and older kernel version: root@ucs5primary:/mnt# uname -a Linux ucs5primary 4.19.0-27-amd64 #1 SMP Debian 4.19.316-1 (2024-06-25) x86_64 GNU/Linux root@ucs5primary:/mnt# ls -lah /var/lib/samba/sysvol/univention.intranet/ insgesamt 32K drwxrwx---+ 4 Administrator Administrators 4,0K Aug 16 14:48 . drwxrwx---+ 3 Administrator Administrators 4,0K Aug 16 14:48 .. drwxrwx---+ 4 Administrator Administrators 4,0K Aug 16 14:48 Policies drwxrwx---+ 2 Administrator Administrators 4,0K Aug 16 14:48 scripts root@ucs5primary:/mnt# rsync -aAX /var/lib/samba/sysvol/univention.intranet/ /mnt/ root@ucs5primary:/mnt# ls -lah insgesamt 16K drwxrwx---+ 4 Administrator Administrators 37 Aug 16 14:48 . drwxr-xr-x 18 root root 4,0K Aug 16 15:30 .. drwxrwx---+ 4 Administrator Administrators 98 Aug 16 14:48 Policies drwxrwx---+ 2 Administrator Administrators 6 Aug 16 14:48 scripts root@ucs5primary:/mnt# ls -lah Policies/ insgesamt 16K drwxrwx---+ 4 Administrator Administrators 98 Aug 16 14:48 . drwxrwx---+ 4 Administrator Administrators 37 Aug 16 14:48 .. drwxrwx---+ 4 Administrator Domain Admins 48 Aug 16 14:48 {31B2F340-016D-11D2-945F-00C04FB984F9} drwxrwx---+ 4 Administrator Domain Admins 48 Aug 16 14:48 {6AC1786C-016F-11D2-945F-00C04FB984F9} root@ucs5primary:/mnt# getfacl Policies/\{31B2F340-016D-11D2-945F-00C04FB984F9\}/ # file: Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/ # owner: Administrator # group: Domain\040Admins user::rwx user:Administrator:rwx user:5010:r-x user:5035:rwx group::rwx group:Authenticated\040Users:r-x group:System:rwx group:Administrators:rwx group:Server\040Operators:r-x mask::rwx other::--- default:user::rwx default:user:Administrator:rwx default:user:5010:r-x default:user:5035:rwx default:group::--- default:group:Authenticated\040Users:r-x default:group:System:rwx default:group:Administrators:rwx default:group:Server\040Operators:r-x default:mask::rwx default:other::---
Workaround is to reboot an older kernel version on both systems where rsync will be used. In my case this was the kernel: root@ucs5primary:/mnt# uname -a Linux ucs5primary 4.19.0-27-amd64 #1 SMP Debian 4.19.316-1 (2024-06-25) x86_64 GNU/Linux
Since updating from Kernel 4.19 to Kernel 5.10 the behavior of xfs seems to have changed with respect to the handling of xattrs. As a symptom, rsync -AX seems to remove trusted.SGI_ACL_FILE when synchronizing from the sysvol from a system with an ext4 partition. Looks similar to https://github.com/RsyncProject/rsync/issues/301 and the workaround listed there almost works, adjusted to our use case to "--filter='-x! security.NTACL'". but it caused problems in combination with --delete-excluded, which then removed all "excluded" xattrs (i.e. everything except security.NTACL) on the recipient side. According to the man page, the use of that --delete-excluded option modifies the behavior of --filter to only apply server side, but not recipient side any longer. The documentation of the --xattrs option mentions that under certain circumstances, the --filter rules need to be explicitly extended, so now we also add the xattr exclude filter rule explicitly for the recipient side, like "--filter='-xr! security.NTACL'". That seems to work, customer reported positive feedback.
For explanation: The getfacl calls above only show a symptom. The fACLs are stored in extended attributes (aka xattrs) that seem to be file system specific. ext4 apparently stores them in system.posix_acl_access and system.posix_acl_default while xfs stores them in trusted.SGI_ACL_FILE and trusted.SGI_ACL_DEFAULT (see https://lore.kernel.org/all/20151027113133.GA26600@bfoster.bfoster/T/ OTOH Samba also uses the xattr security.NTACL to store the NTACLs (UCS configures smb.conf to do it that way) and sysvol-sync.sh uses rsync with the option -X/--xattrs to sync the extended attributes. It's still unclear to me how this can have worked without problems between ext4 and xfs under Linux Kernel 4.19, but apparently something changed in the interaction of xfs under Linux Kernel 5.10 and rsync and that causes rsync to delete the xfs specific xattrs trusted.SGI_ACL_{FILE,DEFAULT} which destroys the basis for the lookup of fACLs on the target xfs system. In case other people face issues like this, it's helpful to look at the xattrs: EXAMPLE_FILE="/var/lib/samba/sysvol/$(hostname -d)/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/MACHINE" getfattr -m - "$EXAMPLE_FILE" # -d ## optionally show values We now have an adjusted version of sysvol-sync.sh making use of the advanced rsync filter options, which will probably be released as an errata update.
Created attachment 11230 [details] bug57529-xfs-rsync-xattr.patch Patch is in the attachment. For migration the patch, download it and use the following command: patch -p0 -d / < bug57529-xfs-rsync-xattr.patch
09f6e9551a | filter xattrs during sysvolsync Package: univention-samba4 Version: 9.0.18-2 Branch: 5.0-0 Scope: errata5.0-9
OK: Sysvol sync uses filter option to filter out every xattr that is not security.NTACL and thus inhibits the other xattr to be overwritten/deleted during sync. OK: Sync between xfs and ext4 on primary backup OK: Manual run of ucs-test on an xfs /ext4 mixed system OK: Jenkins OK: YAML Verified
<https://errata.software-univention.de/#/?erratum=5.0x1132>