Comment 1 Arvid Requate 2013-12-16 14:06:59 CET

This looks like a general problem of rsync based, filestamp controlled replication:

Changing the ACL of a file does not alter the modification time of the file, only the "change time". Out current "rsync -autAX" does not overwrite files that have a newer modification time, but it does not consider the change time ("status change"), because there is no way for rsync to preserve it: The ctime is set automatically by the system. To understand the implications assume that rsync would have an option e.g.

rsync --overwrite-only-if-ctime-is-newer-than-ctime-of-target

and we would use that. Then the following case could happen:
==========================================================================
1. status/ACL of file1 changed manually on DC1
2. sysvol-sync runs on DC1 and copies only files "with newer status" (ctime) from DC2.
   Good: file1 is not overwritten if the most recent change was on DC1. 
3. For all the files written to DC1 Linux automatically sets the ctime to the current local system time.
4. sysvol-sync runs on DC2 and copies only files "with newer status" from DC1. file1 is copied from to DC2 as it matches this criterion. Linux updates the ctime to the local system time.
5. As resulting artefact the ctime of file1 on DC2 is now newer than that of DC1 and will get copied back to DC1 in the next iteration.
==========================================================================

Thus, the ctime is not a usful criterion for rsync.



Currently I see two/three and a half possible options (thanks to Janis for the discussion), none of which are strictly appealing:

A) Modify the sysvol-sync script to check all sysvol files for cases where the ctime is newer than the mtime (i.e. ACL changed). For all these files update the mtime as well. We face a little chicken and egg problem here, as Linux automatically updates the ctime when we try to set the mtime to the current ctime: After that the ctime wil be newer than the mtime (which was set to the ctime of the file before the change). The next best thing we can do at this point, is to set the mtime to the courent time.

find /var/lib/samba/sysvol -print0 | xargs -0 -I{.} \
    find {.} -maxdepth 0 -cnewer {.} -execdir touch -m {.} \;

This is a bit ugly, as it masks changes made on DC2 in the time interval between time(ACL of file1 was changed on DC1) and time(sysvol-sync was run on DC1). In this case DC1 wins (even if DC2 is the first to run rsync).

B) Maybe it would be possible to invent a Samba4 LDB module that intercepts changes to the ntSecurityDescriptor of a GPO container in the Samba4 directory and touches the files in the sysvol share accordingly. This is only speculation right now and only works IF the GPMC actually changes the ntSecurityDescriptor (untested) after it changes the files in the share (untested).

C) Experiment with FRS/DFSR service

D) Construcct some inotify base solution. AFAIK this would require individual inotify monitors to be set up for all subdirectories of the sysvol (?).

Comment 2 Arvid Requate 2013-12-16 14:11:21 CET

Ticket#: 2013121121001491

Comment 3 Arvid Requate 2013-12-16 14:12:21 CET

*** Bug 33753 has been marked as a duplicate of this bug. ***

Comment 4 Felix Botner 2013-12-16 16:37:28 CET

changing the security filter via GPMC also changes the ntSecurityDescriptor of the grouppolicycontainer object

and samba recognizes if the filesystem acl's and the gpo acl's differ:

-> samba-tool ntacl sysvolcheck
samba-tool ntacl sysvolcheck
WARNING: No path in service IPC$ - making it unavailable!
NOTE: Service IPC$ is flagged unavailable.
ERROR(<class 'samba.provision.ProvisioningError'>): uncaught exception - ProvisioningError: DB ACL on GPO directory /var/lib/samba/sysvol/fff.ggg/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9} O:LAG:DAD:P(A;OICI;0x001f01ff;;;DA)(A;OICI;0x001f01ff;;;EA)(A;OICIIO;0x001f01ff;;;CO)(A;OICI;0x001f01ff;;;DA)(A;OICI;0x001f01ff;;;SY)(A;OICI;0x001200a9;;;AU)(A;OICI;0x001200a9;;;ED) does not match expected value O:DAG:DAD:P(A;OICI;0x001f01ff;;;DA)(A;OICI;0x001f01ff;;;EA)(A;OICIIO;0x001f01ff;;;CO)(A;OICI;0x001f01ff;;;DA)(A;OICI;0x001f01ff;;;SY)(A;OICI;0x001200a9;;;AU)(A;OICI;0x001200a9;;;ED) from GPO object
  File "/usr/lib/python2.6/dist-packages/samba/netcmd/__init__.py", line 175, in _run
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/samba/netcmd/ntacl.py", line 245, in run
    lp)
  File "/usr/lib/python2.6/dist-packages/samba/provision/__init__.py", line 1686, in checksysvolacl
    direct_db_access)
  File "/usr/lib/python2.6/dist-packages/samba/provision/__init__.py", line 1637, in check_gpos_acl
    domainsid, direct_db_access)
  File "/usr/lib/python2.6/dist-packages/samba/provision/__init__.py", line 1587, in check_dir_acl
    raise ProvisioningError('%s ACL on GPO directory %s %s does not match expected value %s from GPO object' % (acl_type(direct_db_access), path, fsacl_sddl, acl))

sysvolreset fixes this according to the gpo acl's

Why not just use "samba-tool ntacl sysvolreset" to keep the acl's in sync?


sysvolreset  - Reset sysvol ACLs to defaults (including correct ACLs on GPOs).

Comment 5 Arvid Requate 2013-12-16 18:46:54 CET

Yes, calling sysvolreset at the en of the sysvol-sync script might be the most efficient way to get the ACLs consistent in this case. It's not the most performant of all tools though, maybe we need to strip it down to the essentials. I'll attach a reduced python script below.


About the traceback you observed:

> and samba recognizes if the filesystem acl's and the gpo acl's differ:
> 
> -> samba-tool ntacl sysvolcheck
> [some traceback]

This is a known problem in cases where a group name appears in the owner field of a NTACL. In the traceback above you see that the SDDL-string-representation of the ntSecurityDescriptor of the group policy container starts with "O:DA", meaning "Owner: Domain Admins". When sysvolreset (or samba) tries to generate Posix-ACLs from this, it asks winbind to look up the Posix-ID and winbind tries to find an idmap entry mapping a SID to a UID. Since our samba4-idmap listener marks the idmap entry for "Domain Admins" as a GID-type mapping, it does not find anything. As a fallback it writes the uid of "Administrator" into the Posix owner field. Probably this does not do any harm in this situation, since the NTACLs/fACLs also explicitely grant rights to "Domain Admins". Anyway, this is not something we need or can easily fix at this point.

Comment 6 Arvid Requate 2013-12-16 18:50:33 CET

Created attachment 5701 [details]
sysvol_dsacl_to_ntacl.py

This reduced version of samba-tool ntacl sysvoltreset skipps the time consuming step to write default NTACLs into the file system before looking up the actual DSACLs in the samba/DSDB. It's still pretty slow though.

Comment 7 Arvid Requate 2014-01-08 13:34:32 CET

Ok, basically the problem is caused by a specific behaviour of the "--update" flag of rsync, which skips "files that are newer on the receiver" instead of skipping files that have a "newer or equal" modification time (and size) in the target directory.

The most simple workaround is to make Samba additionally "touch" the files whenever an NTACL gets modified in the sysvol share.

So I patched Samba to provide a new share option "acl xattr update mtime".
If this option is activated on a share the vfs_acl_xattr module updates the file modification time whenever it writes an NTACL.

univention-samba4 was adjusted as well: The UCR template for base.conf now evaluates a new UCR variable samba/share/sysvol/update_mtime. It sets the new Samba share option on the sysvol share if the UCR variable is unset or true.

Advisories:   2014-01-08-univention-samba4.yaml   2014-01-08-samba.yaml




The QA should setup an UCS 3.2 domain with two Samba4 DCs (e.g. master and slave) and do the following steps:

* Disable the sysvol sync on both DCs:
      ucr set samba4/sysvol/sync/cron='## */5 * * * *'

* Then a GPO should be created from a joined Windows client.

* Check on which DC the GPO was created by comparing the output of
      ls /var/lib/samba/sysvol/*/Policies
  probably the additional GPO was created on the master.

* To replicate the new GPO to both DCs run:
 /usr/share/univention-samba4/scripts/sysvol-sync.sh

* Modify the security filter (aka delegation) setting of the GPO via the Windows Group Policy Management Console, add a new user/group to the access list. After that, the GPMC should be closed and
  /usr/share/univention-samba4/scripts/sysvol-sync.sh
should be called on the machine originally holding the new GPO (e.g. the master)

* Start the GPMC again: Now warning message should pop up. Check the security filter settings of the GPO.

Comment 8 Felix Botner 2014-01-10 14:56:27 CET

That works for files, but not for directories.

One the master:

-> getfacl \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/| grep test1
-> getfacl \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/GPT.INI| grep test1
-> getfacl /var/cache/univention-samba4/sysvol-sync/pbackup/perf.test/Policies/\{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/GPT.INI | grep test1
-> getfacl /var/cache/univention-samba4/sysvol-sync/pbackup/perf.test/Policies/\{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/ | grep test1

No ACL'S for test1 in GPO on master and the backup cache

Now i add test1 via GPMC on the master

ACL's are set

-> getfacl \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/GPT.INI | grep test1
user:test1:r-x
-> getfacl \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/ | grep test1
user:test1:r-x
default:user:test1:r-x

modify time updated

-> stat \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/GPT.INI | grep Modify
Modify: 2014-01-10 14:44:51.416000000 +0100
-> stat \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/ | grep Modify
Modify: 2014-01-10 14:44:51.460000000 +0100

modify time of the backup cache is older

-> stat /var/cache/univention-samba4/sysvol-sync/pbackup/perf.test/Policies/\{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/ | grep Modify:
Modify: 2014-01-10 14:40:44.000000000 +0100
-> stat /var/cache/univention-samba4/sysvol-sync/pbackup/perf.test/Policies/\{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/GPT.INI | grep Modify:
Modify: 2014-01-10 14:40:44.000000000 +0100

trigger the update from the backup cache on the master

-> touch /var/cache/univention-samba4/sysvol-sync/.trigger/pbackup
-> /usr/share/univention-samba4/scripts/sysvol-sync.sh

FAIL, ACL's are broken for directories

-> getfacl \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/| grep test1
-> getfacl \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/GPT.INI| grep test1
user:test1:r-x

FAIL, modify time of "\{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}" is now again
      14:40:44 (even it was set to 14:44:51 before)

-> stat \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\}/GPT.INI | grep Modify
Modify: 2014-01-10 14:44:51.416000000 +0100
-> stat \{4ACDBC7B-82F9-448D-8638-C4AF5B90A1DA\} | grep Modify
Modify: 2014-01-10 14:40:44.000000000 +0100

Comment 9 Arvid Requate 2014-01-13 17:38:06 CET

Created attachment 5731 [details]
update_only_for_directories.patch

The directory mtime changes are pretty hard to fix, this is an rsync feature:

1) rsync doesn't care about the --update switch for directories
2) rsync touches all parent directories of a change

As yet I have no real clue why rsync behaves this way. The attached patch "fixes" this, but I'm not confident that this is without side effects. To be discussed.

Comment 10 Arvid Requate 2014-01-22 19:31:55 CET

Created attachment 5743 [details]
Add new option --dirs-update

The attached patch adds a new option "--dirs-update" to make rsync respect directory modification times in a way similar to the behaviour of "--update" for files. A git-patch against the master branch has been posted to the rsync mailinglist as well.

If we can agree upon applying this patch, this bug needs to be cloned for rsync and univention-samba4-sysvol-sync needs to be patched to use the new option.

Comment 11 Arvid Requate 2014-03-31 20:45:10 CEST

The patch from comment 10 has been split off as Bug #34430.

This bug should be used to
 * make use of the new rsync option in the sysvol-sync script
 * Configure the new samba parameter implemented via Bug 34431

Comment 12 Arvid Requate 2014-05-28 20:28:54 CEST

Created attachment 5930 [details]
test_gpo_acls.sh

The test script still needs to be converted into an ucs-test case but it works. It needs to be run on a DC Backup or DC Slave.

Advisory: 2014-05-28-univention-samba4.yaml

Comment 13 Stefan Gohmann 2014-05-30 07:20:20 CEST

The test case failed:

*** 51_samba4/52replication_sysvol *** Testing the sysvol replication ***
## create the GPO 'kfoTLK1g' on remote DC master095.autotest095.local
params.c:pm_process() - Processing configuration file "/etc/samba/base.conf"
Processing section "[netlogon]"
Processing section "[sysvol]"
Processing section "[IPC$]"
Processing section "[homes]"
Processing section "[printers]"
Processing section "[print$]"
params.c:pm_process() - Processing configuration file "/etc/samba/shares.conf"
params.c:pm_process() - Processing configuration file "/etc/samba/printers.conf"
pm_process() returned Yes
GENSEC backend 'gssapi_spnego' registered
GENSEC backend 'gssapi_krb5' registered
GENSEC backend 'gssapi_krb5_sasl' registered
GENSEC backend 'schannel' registered
GENSEC backend 'spnego' registered
GENSEC backend 'ntlmssp' registered
GENSEC backend 'krb5' registered
GENSEC backend 'fake_gssapi_krb5' registered
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
finddcs: response 0 at '10.210.76.115'
finddcs: performing CLDAP query on 10.210.76.115
finddcs: Found matching DC 10.210.76.115 with server_type=0x000003fd
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
Received smb_krb5 packet of length 329
Received smb_krb5 packet of length 183
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
added interface eth0 ip=10.210.137.138 bcast=10.210.255.255 netmask=255.255.0.0
Received smb_krb5 packet of length 329
Received smb_krb5 packet of length 183
## GPO was created on master095.autotest095.local with ID {FEBDF40A-455D-4095-9C56-36F388B577AF}
## check whether the directory has been created on remote DC master095.autotest095.local
## check whether samba-tool lists the GPO on remote DC master095.autotest095.local
## check whether the directory for the GPO has been replicated to the local system (waiting about 300 seconds)
## Note: to speed things up for interactive tests, you may run /usr/share/univention-samba4/scripts/sysvol-sync.sh manually now.

error 2014-05-29 19:55:37	 Directory for GPO has not been created
error 2014-05-29 19:55:37	 **************** Test failed above this line (1) ****************
params.c:pm_process() - Processing configuration file "/etc/samba/base.conf"

http://jenkins.knut.univention.de:8080/view/Autotest/job/UCS%203.2-2%20Autotest%20MultiEnv/SambaVersion=s4,Systemrolle=slave/28/testReport/51_samba4/52replication_sysvol/test/

Comment 14 Arvid Requate 2014-06-02 18:23:42 CEST

> The test case failed:
>
> *** 51_samba4/52replication_sysvol

That's the other test which existed before. It's a bit flaky every now and then, no clue yet why. Looked good today in jenkins and when run manually. The new test 52replication_sysvol_acls seems to run fine:

http://jenkins.knut.univention.de:8080/job/UCS%203.2-2%20Autotest%20MultiEnv/SambaVersion=s4,Systemrolle=backup/lastCompletedBuild/testReport/51_samba4/52replication_sysvol_acls/history/

Comment 15 Stefan Gohmann 2014-06-04 06:57:26 CEST

YAML: OK

Sysvol sync tests: OK

Test cases: OK

Comment 16 Moritz Muehlenhoff 2014-06-05 15:33:09 CEST

http://errata.univention.de/ucs/3.2/119.html