Univention Bugzilla – Bug 46590
memberOf differences between master and backup in slapcat output
Last modified: 2024-02-29 15:56:05 CET
After comparing slapcat on the master and the backup (of a setup with two DC slaves) with diff-ldif I got the following diff: https://pastebin.knut.univention.de/WATiXDm9 UCS@school 4.3
Content of the pastebin: dn: uid=hbl2,cn=lehrer und mitarbeiter,cn=users,ou=HB,dc=realm1,dc=intranet +memberOf: cn=mitarbeiter-hb,cn=groups,ou=HB,dc=realm1,dc=intranet -memberOf: cn=mitarbeiter-HB,cn=groups,ou=HB,dc=realm1,dc=intranet +memberOf: cn=lehrer-hb,cn=groups,ou=HB,dc=realm1,dc=intranet -memberOf: cn=lehrer-HB,cn=groups,ou=HB,dc=realm1,dc=intranet dn: uid=hbl1,cn=lehrer,cn=users,ou=HB,dc=realm1,dc=intranet +memberOf: cn=lehrer-hb,cn=groups,ou=HB,dc=realm1,dc=intranet -memberOf: cn=lehrer-HB,cn=groups,ou=HB,dc=realm1,dc=intranet dn: uid=aes6,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=aes5,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=aes4,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=aes3,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=aes2,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=aes1,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=ael2,cn=lehrer und mitarbeiter,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=mitarbeiter-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=mitarbeiter-AE,cn=groups,ou=AE,dc=realm1,dc=intranet +memberOf: cn=lehrer-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=lehrer-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=ael1,cn=lehrer,cn=users,ou=AE,dc=realm1,dc=intranet +memberOf: cn=lehrer-ae,cn=groups,ou=AE,dc=realm1,dc=intranet -memberOf: cn=lehrer-AE,cn=groups,ou=AE,dc=realm1,dc=intranet dn: uid=join-backup,cn=users,dc=realm1,dc=intranet -memberOf: cn=Slave Join,cn=groups,dc=realm1,dc=intranet dn: cn=DC Backup Hosts,cn=groups,dc=realm1,dc=intranet +memberOf: cn=Windows Hosts,cn=groups,dc=realm1,dc=intranet dn: cn=master,cn=dc,cn=computers,dc=realm1,dc=intranet -memberOf: cn=DC Backup Hosts,cn=groups,dc=realm1,dc=intranet dn: cn=backup,cn=dc,cn=computers,dc=realm1,dc=intranet -memberOf: cn=DC Backup Hosts,cn=groups,dc=realm1,dc=intranet
I'm able to see the same issue on a normal master and backup installation: root@master501:~# univention-ldapsearch uid=join-backup memberOf -LLL dn: uid=join-backup,cn=users,dc=deadlock50,dc=intranet memberOf: cn=Slave Join,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet root@master501:~# root@backup502:~# univention-ldapsearch uid=join-backup memberOf -LLL dn: uid=join-backup,cn=users,dc=deadlock50,dc=intranet memberOf: cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet root@backup502:~# A workaround might be to xecute /usr/share/univention-ldap-overlay-memberof/univention-update-memberof root@backup502:~# /usr/share/univention-ldap-overlay-memberof/univention-update-memberof modify cn=Computers,cn=groups,dc=deadlock50,dc=intranet modify cn=Slave Join,cn=groups,dc=deadlock50,dc=intranet modify cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet modify cn=Domain Users,cn=groups,dc=deadlock50,dc=intranet modify cn=Domain Admins,cn=groups,dc=deadlock50,dc=intranet modify cn=Windows Hosts,cn=groups,dc=deadlock50,dc=intranet modify cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet modify cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet root@backup502:~# univention-ldapsearch uid=join-backup memberOf -LLL dn: uid=join-backup,cn=users,dc=deadlock50,dc=intranet memberOf: cn=Slave Join,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet memberOf: cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet root@backup502:~# I'll tag it as erratum since I think we should understand what happens here.
This happens during the initial join but can happen again later on when UDL is delayed for some time: Consider the following sequence of modification on the Master: 0. group exists (or is created - not relevant) 1. user1 is added to group 2. user2 is added to group this translates to the following transaction: 1.1 add user1 1.2 add user1 to group 2.1 add user2 2.2 add user2 to group on a replicating system where UDL is delayed transaction 2 and 4 are merged, leading to the following sequence there: 1. add user1 2. add user1 and user2 to group 3. add user2 4. empty update to group Now at step 2 the overlay module "memberof" kicks in and iterates over all users "user1" and "user2" to update their is-"memberOf"-Attribute. As "user2" does not exists yet, its entry cannot be updated in step 2. In step 3 user2 is finally created, but the "memberOf"-attribute is not replicated from the Master and thus is not set. As step 4 is empty (the complete change has already been applied in step 2), the "memberof" overlay module is not triggered again and the "memberOf" attribute is never set.
I can reproduce the steps assumed in comment #3 in UCS 4.4-4 - environment with two DC Slaves dcs1 and dcs2 - stop univention-directory-listener on dcs2 - create group "repltest-01", users "repltestu-01" and "repltestu-02" - start univention-directory-listener on dcs2 Results are: ---- root@dcm:~# univention-ldapsearch -LLL uid=replt* memberof dn: uid=repltestu-01,cn=users,dc=autorem,dc=intranet memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet dn: uid=repltestu-02,cn=users,dc=autorem,dc=intranet memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet ---- root@dcs0:~# univention-ldapsearch -LLL uid=replt* memberof dn: uid=repltestu-01,cn=users,dc=autorem,dc=intranet memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet dn: uid=repltestu-02,cn=users,dc=autorem,dc=intranet memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet ---- root@dcs1:~# univention-ldapsearch -LLL uid=replt* memberof dn: uid=repltestu-01,cn=users,dc=autorem,dc=intranet memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet dn: uid=repltestu-02,cn=users,dc=autorem,dc=intranet ---- This is even more alerting because "memberOf" even doesn't work for "Domain Users" properly.
A mitigation would be to run "/usr/share/univention-ldap-overlay-memberof/univention-update-memberof" more often, in worst case in listener "postrun" after any user or group has been replicated - how "expensive" is this script in case nothing has to be changed?
That script replaces uniqueMember on all groups with the current value to trigger recalculation. With Bug #48545 we tried to avoid that in replication.py. Maybe we could refine the change of Bug #48545 to only add DNs to uniqueMember that actually already exist. That way the memberOf calculation would not get neglected for users that don't exist yet at the time of group membership replication.
Created attachment 10357 [details] Bug #46590 replication.py: Replicate memberOf
Created attachment 10358 [details] v2 Bug #46590 replication.py: Replicate memberOf Inverted logic
In comments 3 and 4 is well described, what causes the problem and how to reproduce it. So far the memberOf attribute was basically filtered out by the replication module, so the memberOf overlay module on backup/slave was responsible for filling in the correct values on these systems. This was done because the LDAP schema for the memberOf attribute is brought along by the overlay module, and it is not ensured that the overlay module is activated on all systems (otherwise there is a failed.ldif). Arvid, Philipp and I have now worked out the following solution: We now remove the filtering if the overlay module is activated via UCR variable. This leads to the fact that the attribute values are taken over 1:1 by the master and are updated by the local overlay module if necessary. This already worked in the first test. However, this variant has two disadvantages: 1) In environments with selective replication, such as UCS@school, not all groups are synchronized to e.g. a school slave. It is then possible that there are memberOf attribute values for groups at the user object, which will never exist in the local LDAP. 2) There is a small time window between activation of the memberOf overlay module via UCR variable and restart of the LDAP server, where a restart of the listener would be unfavorable, because then it comes to a failed.ldif due to missing LDAP schema (The failed.ldif should be imported automatically with the next restart of the LDAP server). Regarding case 1 there is already an analogous case for uniqueMember entries to groups: in UCS@school groups are replicated on school servers, but not all group members are available on the local LDAP server. Therefore these group objects appear inconsistent, because uniqueMember entries for missing user objects appear there. We still think this is the most sensible solution. Other solutions, which we have discarded again: a) automatic adjustment of the replication order: first the users, then the groups → this makes replication much more complex and also more error-prone, since the replication behavior changes for existing listener modules b) when replicating groups, check if the group members already exist locally and if not, remove the corresponding uniqueMember entries. → this makes replication a) more expensive because considerably more local lookups are needed and b) the listener has to be changed extensively so that listener modules only get what is written to the local LDAP. Otherwise there will be inconsistencies between the data status of the listener modules and the local LDAP server. [4.4-4] 8ca278f040 Bug #46590: Merge branch 'sschwardt/46590/4.4-4' into 4.4-4 [4.4-4] 639142ee8c Bug #46590: add changelog entry for ucs-test [4.4-4] d2771438c5 Bug #46590: add 10_ldap/42replication_memberof [4.4-4] adad2c5ac6 Bug #46590: add advisory [4.4-4] feef7a38ce Bug #46590: add changelog entry [4.4-4] 4e92127da7 Bug #46590 replication.py: fix memberOf handling Package: univention-directory-replication Version: 12.0.0-10A~4.4.0.202005122038 Branch: ucs_4.4-0 Scope: errata4.4-4 ackage: ucs-test Version: 9.0.3-202A~4.4.0.202005122041 Branch: ucs_4.4-0 Scope: errata4.4-4
[4.4-4] 3d9fd419ca Bug #46590: wait for samba replication in 10_ldap/42replication_memberof [4.4-4] 9fdfa7069a Bug #46590: increase timeouts of 10_ldap/42replication_memberof ackage: ucs-test Version: 9.0.3-204A~4.4.0.202005131538 Branch: ucs_4.4-0 Scope: errata4.4-4
http://jenkins.knut.univention.de:8080/job/UCS-4.4/job/UCS-4.4-4/job/AutotestJoin/lastCompletedBuild/SambaVersion=s4,Systemrolle=slave/testReport/10_ldap/42replication_memberof/slave095/ the new test case fails from time to time
Already fixed: [4.4-4] 647ce01ca2 Bug #46590: 10_ldap/42replication_memberof: fixed wait_for on UCS@school slave [4.4-4] 4fe79afdfc Bug #46590: fixed bug in udm.py: use of uninitialized variable [4.4-4] eb77c2df32 Bug #46590: also test 10_ldap/42replication_memberof in UCS@school environments [4.4-4] c21525e38e Bug #46590: 10_ldap/42replication_memberof: skip test in certain scenarios
There was no error thrown by the test script in UCS + UCS@school tests last night.
Verified: * Code review * Jenkins-Tests: 10_ldap/42replication_memberof ** Series: Errata,join,Autotest *** s4,master-part-II *** SKIPPED: s4,backup *** s4,slave *** no-samba,master *** no-samba,backup *** no-samba,slave ** Series: join,upgrade=4.4,Autotest *** s4,master-part-II *** SKIPPED: s4,backup *** s4,slave *** no-samba,master *** no-samba,backup *** no-samba,slave ** Series: UCS@school 4.4, Multiserver *** s4,base1,testing (this test an School Slave PDC) * Advisory
<http://errata.software-univention.de/ucs/4.4/606.html>