Bug 46590 - memberOf differences between master and backup in slapcat output
memberOf differences between master and backup in slapcat output
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: LDAP
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-4-errata
Assigned To: Sönke Schwardt-Krummrich
Arvid Requate
:
Depends on:
Blocks: 51341
  Show dependency treegraph
 
Reported: 2018-03-12 10:57 CET by Ole Schwiegert
Modified: 2024-02-29 15:56 CET (History)
7 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 3: Simply Wrong: The implementation doesn't match the docu
Who will be affected by this bug?: 5: Will affect all installed domains
How will those affected feel about the bug?: 4: A User would return the product
User Pain: 0.343
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2020050821000499
Bug group (optional):
Max CVSS v3 score:


Attachments
Bug #46590 replication.py: Replicate memberOf (2.76 KB, patch)
2020-05-12 11:52 CEST, Philipp Hahn
Details | Diff
v2 Bug #46590 replication.py: Replicate memberOf (2.76 KB, patch)
2020-05-12 11:55 CEST, Philipp Hahn
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ole Schwiegert univentionstaff 2018-03-12 10:57:58 CET
After comparing slapcat on the master and the backup (of a setup with two DC slaves) with diff-ldif I got the following diff:

https://pastebin.knut.univention.de/WATiXDm9

UCS@school 4.3
Comment 1 Sönke Schwardt-Krummrich univentionstaff 2018-03-12 21:38:37 CET
Content of the pastebin:

 dn: uid=hbl2,cn=lehrer und mitarbeiter,cn=users,ou=HB,dc=realm1,dc=intranet
+memberOf: cn=mitarbeiter-hb,cn=groups,ou=HB,dc=realm1,dc=intranet
-memberOf: cn=mitarbeiter-HB,cn=groups,ou=HB,dc=realm1,dc=intranet
+memberOf: cn=lehrer-hb,cn=groups,ou=HB,dc=realm1,dc=intranet
-memberOf: cn=lehrer-HB,cn=groups,ou=HB,dc=realm1,dc=intranet

dn: uid=hbl1,cn=lehrer,cn=users,ou=HB,dc=realm1,dc=intranet
+memberOf: cn=lehrer-hb,cn=groups,ou=HB,dc=realm1,dc=intranet
-memberOf: cn=lehrer-HB,cn=groups,ou=HB,dc=realm1,dc=intranet

dn: uid=aes6,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=aes5,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=aes4,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=aes3,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=aes2,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=aes1,cn=schueler,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=schueler-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=schueler-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=ael2,cn=lehrer und mitarbeiter,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=mitarbeiter-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=mitarbeiter-AE,cn=groups,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=lehrer-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=lehrer-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=ael1,cn=lehrer,cn=users,ou=AE,dc=realm1,dc=intranet
+memberOf: cn=lehrer-ae,cn=groups,ou=AE,dc=realm1,dc=intranet
-memberOf: cn=lehrer-AE,cn=groups,ou=AE,dc=realm1,dc=intranet

dn: uid=join-backup,cn=users,dc=realm1,dc=intranet
-memberOf: cn=Slave Join,cn=groups,dc=realm1,dc=intranet

dn: cn=DC Backup Hosts,cn=groups,dc=realm1,dc=intranet
+memberOf: cn=Windows Hosts,cn=groups,dc=realm1,dc=intranet

dn: cn=master,cn=dc,cn=computers,dc=realm1,dc=intranet
-memberOf: cn=DC Backup Hosts,cn=groups,dc=realm1,dc=intranet

dn: cn=backup,cn=dc,cn=computers,dc=realm1,dc=intranet
-memberOf: cn=DC Backup Hosts,cn=groups,dc=realm1,dc=intranet
Comment 2 Stefan Gohmann univentionstaff 2018-03-12 22:09:38 CET
I'm able to see the same issue on a normal master and backup installation:

root@master501:~# univention-ldapsearch uid=join-backup memberOf -LLL
dn: uid=join-backup,cn=users,dc=deadlock50,dc=intranet
memberOf: cn=Slave Join,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet

root@master501:~# 


root@backup502:~# univention-ldapsearch uid=join-backup memberOf -LLL
dn: uid=join-backup,cn=users,dc=deadlock50,dc=intranet
memberOf: cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet

root@backup502:~#

A workaround might be to xecute /usr/share/univention-ldap-overlay-memberof/univention-update-memberof


root@backup502:~# /usr/share/univention-ldap-overlay-memberof/univention-update-memberof 
modify cn=Computers,cn=groups,dc=deadlock50,dc=intranet
modify cn=Slave Join,cn=groups,dc=deadlock50,dc=intranet
modify cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet
modify cn=Domain Users,cn=groups,dc=deadlock50,dc=intranet
modify cn=Domain Admins,cn=groups,dc=deadlock50,dc=intranet
modify cn=Windows Hosts,cn=groups,dc=deadlock50,dc=intranet
modify cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet
modify cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet
root@backup502:~# univention-ldapsearch uid=join-backup memberOf -LLL
dn: uid=join-backup,cn=users,dc=deadlock50,dc=intranet
memberOf: cn=Slave Join,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=Backup Join,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=DC Slave Hosts,cn=groups,dc=deadlock50,dc=intranet
memberOf: cn=DC Backup Hosts,cn=groups,dc=deadlock50,dc=intranet

root@backup502:~# 

I'll tag it as erratum since I think we should understand what happens here.
Comment 3 Philipp Hahn univentionstaff 2020-04-16 14:19:57 CEST
This happens during the initial join but can happen again later on when UDL is delayed for some time:

Consider the following sequence of modification on the Master:

0. group exists (or is created - not relevant)
1. user1 is added to group
2. user2 is added to group

this translates to the following transaction:

1.1 add user1
1.2 add user1 to group
2.1 add user2
2.2 add user2 to group

on a replicating system where UDL is delayed transaction 2 and 4 are merged, leading to the following sequence there:

1. add user1
2. add user1 and user2 to group
3. add user2
4. empty update to group

Now at step 2 the overlay module "memberof" kicks in and iterates over all users "user1" and "user2" to update their is-"memberOf"-Attribute. As "user2" does not exists yet, its entry cannot be updated in step 2.

In step 3 user2 is finally created, but the "memberOf"-attribute is not replicated from the Master and thus is not set.

As step 4 is empty (the complete change has already been applied in step 2), the "memberof" overlay module is not triggered again and the "memberOf" attribute is never set.
Comment 4 Ingo Steuwer univentionstaff 2020-05-11 18:00:43 CEST
I can reproduce the steps assumed in comment #3 in UCS 4.4-4

- environment with two DC Slaves dcs1 and dcs2
- stop univention-directory-listener on dcs2
- create group "repltest-01", users "repltestu-01" and "repltestu-02"
- start univention-directory-listener on dcs2

Results are:

----

root@dcm:~# univention-ldapsearch -LLL uid=replt* memberof
dn: uid=repltestu-01,cn=users,dc=autorem,dc=intranet
memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet
memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet

dn: uid=repltestu-02,cn=users,dc=autorem,dc=intranet
memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet
memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet

----

root@dcs0:~# univention-ldapsearch -LLL uid=replt* memberof
dn: uid=repltestu-01,cn=users,dc=autorem,dc=intranet
memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet
memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet

dn: uid=repltestu-02,cn=users,dc=autorem,dc=intranet
memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet
memberOf: cn=repltest-01,cn=groups,dc=autorem,dc=intranet

----

root@dcs1:~# univention-ldapsearch -LLL uid=replt* memberof
dn: uid=repltestu-01,cn=users,dc=autorem,dc=intranet
memberOf: cn=Domain Users,cn=groups,dc=autorem,dc=intranet

dn: uid=repltestu-02,cn=users,dc=autorem,dc=intranet

----

This is even more alerting because "memberOf" even doesn't work for "Domain Users" properly.
Comment 5 Ingo Steuwer univentionstaff 2020-05-11 18:03:30 CEST
A mitigation would be to run "/usr/share/univention-ldap-overlay-memberof/univention-update-memberof" more often, in worst case in listener "postrun" after any user or group has been replicated - how "expensive" is this script in case nothing has to be changed?
Comment 6 Arvid Requate univentionstaff 2020-05-11 18:14:45 CEST
That script replaces uniqueMember on all groups with the current value to trigger recalculation. With Bug #48545 we tried to avoid that in replication.py.

Maybe we could refine the change of Bug #48545 to only add DNs to uniqueMember that actually already exist. That way the memberOf calculation would not get neglected for users that don't exist yet at the time of group membership replication.
Comment 7 Philipp Hahn univentionstaff 2020-05-12 11:52:47 CEST
Created attachment 10357 [details]
Bug #46590 replication.py: Replicate memberOf
Comment 8 Philipp Hahn univentionstaff 2020-05-12 11:55:12 CEST
Created attachment 10358 [details]
v2  Bug #46590 replication.py: Replicate memberOf

Inverted logic
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2020-05-12 20:41:53 CEST
In comments 3 and 4 is well described, what causes the problem and how to reproduce it.

So far the memberOf attribute was basically filtered out by the replication module, so the memberOf overlay module on backup/slave was responsible for filling in the correct values on these systems.
This was done because the LDAP schema for the memberOf attribute is brought along by the overlay module, and it is not ensured that the overlay module is activated on all systems (otherwise there is a failed.ldif).

Arvid, Philipp and I have now worked out the following solution:
We now remove the filtering if the overlay module is activated via UCR variable. This leads to the fact that the attribute values are taken over 1:1 by the master and are updated by the local overlay module if necessary.
This already worked in the first test. However, this variant has two disadvantages:
1) In environments with selective replication, such as UCS@school, not all groups are synchronized to e.g. a school slave. It is then possible that there are memberOf attribute values for groups at the user object, which will never exist in the local LDAP.
2) There is a small time window between activation of the memberOf overlay module via UCR variable and restart of the LDAP server, where a restart of the listener would be unfavorable, because then it comes to a failed.ldif due to missing LDAP schema (The failed.ldif should be imported automatically with the next restart of the LDAP server).

Regarding case 1 there is already an analogous case for uniqueMember entries to groups: in UCS@school groups are replicated on school servers, but not all group members are available on the local LDAP server. Therefore these group objects appear inconsistent, because uniqueMember entries for missing user objects appear there.

We still think this is the most sensible solution.

Other solutions, which we have discarded again:
a) automatic adjustment of the replication order: first the users, then the groups
→ this makes replication much more complex and also more error-prone, since the replication behavior changes for existing listener modules
b) when replicating groups, check if the group members already exist locally and if not, remove the corresponding uniqueMember entries.
→ this makes replication a) more expensive because considerably more local lookups are needed and b) the listener has to be changed extensively so that listener modules only get what is written to the local LDAP. Otherwise there will be inconsistencies between the data status of the listener modules and the local LDAP server.

[4.4-4] 8ca278f040 Bug #46590: Merge branch 'sschwardt/46590/4.4-4' into 4.4-4
[4.4-4] 639142ee8c Bug #46590: add changelog entry for ucs-test
[4.4-4] d2771438c5 Bug #46590: add 10_ldap/42replication_memberof
[4.4-4] adad2c5ac6 Bug #46590: add advisory
[4.4-4] feef7a38ce Bug #46590: add changelog entry
[4.4-4] 4e92127da7 Bug #46590 replication.py: fix memberOf handling

Package: univention-directory-replication
Version: 12.0.0-10A~4.4.0.202005122038
Branch: ucs_4.4-0
Scope: errata4.4-4

ackage: ucs-test
Version: 9.0.3-202A~4.4.0.202005122041
Branch: ucs_4.4-0
Scope: errata4.4-4
Comment 10 Sönke Schwardt-Krummrich univentionstaff 2020-05-13 15:38:55 CEST
[4.4-4] 3d9fd419ca Bug #46590: wait for samba replication in 10_ldap/42replication_memberof
[4.4-4] 9fdfa7069a Bug #46590: increase timeouts of 10_ldap/42replication_memberof

ackage: ucs-test
Version: 9.0.3-204A~4.4.0.202005131538
Branch: ucs_4.4-0
Scope: errata4.4-4
Comment 12 Sönke Schwardt-Krummrich univentionstaff 2020-05-19 09:06:27 CEST
Already fixed:

[4.4-4] 647ce01ca2 Bug #46590: 10_ldap/42replication_memberof: fixed wait_for on UCS@school slave
[4.4-4] 4fe79afdfc Bug #46590: fixed bug in udm.py: use of uninitialized variable
[4.4-4] eb77c2df32 Bug #46590: also test 10_ldap/42replication_memberof in UCS@school environments
[4.4-4] c21525e38e Bug #46590: 10_ldap/42replication_memberof: skip test in certain scenarios
Comment 13 Sönke Schwardt-Krummrich univentionstaff 2020-05-19 09:09:14 CEST
There was no error thrown by the test script in UCS + UCS@school tests last night.
Comment 14 Arvid Requate univentionstaff 2020-05-19 10:40:44 CEST
Verified:
* Code review

* Jenkins-Tests: 10_ldap/42replication_memberof
** Series: Errata,join,Autotest
*** s4,master-part-II
*** SKIPPED: s4,backup
*** s4,slave
*** no-samba,master
*** no-samba,backup
*** no-samba,slave

** Series: join,upgrade=4.4,Autotest
*** s4,master-part-II
*** SKIPPED: s4,backup
*** s4,slave
*** no-samba,master
*** no-samba,backup
*** no-samba,slave


** Series: UCS@school 4.4, Multiserver
*** s4,base1,testing  (this test an School Slave PDC)

* Advisory
Comment 15 Arvid Requate univentionstaff 2020-05-20 12:29:55 CEST
<http://errata.software-univention.de/ucs/4.4/606.html>