Bug 34835 - Modify and move objects in a short amount of time
Modify and move objects in a short amount of time
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Listener (univention-directory-listener)
UCS 3.2
Other Linux
: P1 normal (vote)
: UCS 3.2-1-errata
Assigned To: Philipp Hahn
Stefan Gohmann
:
: 34785 34925 (view as bug list)
Depends on: 34355
Blocks:
  Show dependency treegraph
 
Reported: 2014-05-14 19:57 CEST by Stefan Gohmann
Modified: 2014-05-27 11:21 CEST (History)
4 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2014-05-14 19:57:55 CEST
In a customer environment the modification of an object is not replicated. The log shows:

14.05.14 18:03:15.737  LISTENER    ( ALL     ) : Delaying update for 'uid=USER,ou=users,ou=z,ou=test,dc=base' until moved to 'uid=USER,ou=users,ou=Z,ou=test,dc=base'

The difference is ou=z vs ou=Z. 

This should be announced as erratum for 3.2-1 and 3.2-2.

+++ This bug was initially created as a clone of Bug #34355 +++

Ticket #2014012921002092

If you modify and move objects in a really short time it is possible that the listener does not find the original object for building new + old for the listener.

For example:
udm computers/windows modify --dn "$dn" --set description=foo1
udm computers/windows move --dn "$dn" --position ou=computers,ou=Z,$ldap_base

This results in the following transaction:
1062029 cn=client1,cn=computers,$ldap_base m
1062030 cn=client1,cn=computers,$ldap_base r
1062031 cn=client1,ou=computers,ou=Z,$ldap_base a

The listener reads 1062029 and searches for the object. If the object is already moved, the listener would find the object and thinks it was deleted. That's a problem for some modules for example the s4 connector.

In the test environment the following workaround seems to "fix" the problem:

 ucr set listener/uniquemember/skip=no listener/memberuid/skip=no
 mv /var/lib/univention-directory-listener/handlers/s4-connector /var/lib/univention-directory-listener/handlers/aas4-connector
 mv /usr/lib/univention-directory-listener/system/s4-connector.py /usr/lib/univention-directory-listener/system/aas4-connector.py
 sed -i 's|s4-connector|aas4-connector|' /usr/lib/univention-directory-listener/system/aas4-connector.py
Comment 1 Stefan Gohmann univentionstaff 2014-05-14 20:21:58 CEST
First patch has been added: r50268
Comment 2 Philipp Hahn univentionstaff 2014-05-21 23:26:49 CEST
Notice that the Notifier passes through the DN as given on the command-line:

# /etc/init.d/univention-directory-notifier stop
# tail -f /var/lib/univention-ldap/listener/listener &
# udm users/user modify --dn uid=Hahn,dc=phahn,dc=dev --set description="$(date)"
                                 ^
55580 uid=Hahn,dc=phahn,dc=dev m
          ^
# udm users/user modify --dn uid=hahn,dc=phahn,dc=dev --set description="$(date)"
                                 ^
55581 uid=hahn,dc=phahn,dc=dev m
          ^
The current implementation wrongly assumes that the Listener sees the DN "normalized", that is as "initially created" and returned by 'entryDN':

# univention-ldapsearch -LLLb uid=hahn,dc=phahn,dc=dev dn entryDN
dn: uid=Hahn,dc=phahn,dc=dev
entryDN: uid=Hahn,dc=phahn,dc=dev

I think this is caused by openldap/3.2-0-0-ucs/2.4.35-1/10_translog_overlay.patch using "op->o_req_dn.bv_val" and "op->orr_newrdn.bv_val" and "op->orr_newSup->bv_val".
Comment 3 Stefan Gohmann univentionstaff 2014-05-22 10:03:30 CEST
(In reply to Stefan Gohmann from comment #0)
> 14.05.14 18:03:15.737  LISTENER    ( ALL     ) : Delaying update for
> 'uid=USER,ou=users,ou=z,ou=test,dc=base' until moved to
> 'uid=USER,ou=users,ou=Z,ou=test,dc=base'

Please set this debug message to PROCESS level. That makes it easier to track it down.
Comment 4 Philipp Hahn univentionstaff 2014-05-22 14:43:02 CEST
r13111: openldap_2.4.35-1.160.201405221112
An OpenLDAP overlay module gets the DN in two variants:
1. the one as currently specified by the client,
2. the normalized one as stored initially in the backend.
    
struct Operation {
        struct berval   o_req_dn;       /* DN of target of request */
        struct berval   o_req_ndn;
    
typedef struct req_modrdn_s {
        struct berval rs_newrdn;    // orr_newrdn
        struct berval rs_nnewrdn;   // orr_nnewrdn
        struct berval *rs_newSup;   // orr_newSup
        struct berval *rs_nnewSup;  // orr_nnewSup
FYI: the MODRDN fields are accessed through pre-processor macros "orr_..."
    
The notifier was using the first, which lead to a mismatch between the notifier DN and the LDAP DN.    
Switch the translog overlay module to use the normalized DNs.

r50556 | Bug #34835 Listener: Update version number
r50552 | Bug #34835 Listener: use same_dn()
r50551 | Bug #34833 Listener: Fix error message
r50550 | Bug #34835 Listener: Check real same_rdn()
r50549 | Bug #34835 translog: Use normalized DN
r50557 | Bug #34835 Listener: Update version number
r50558 | Bug #34835 Listener: Ignore case in DN


(In reply to Stefan Gohmann from comment #3)
> (In reply to Stefan Gohmann from comment #0)
> Please set this debug message to PROCESS level.
r50551 | Bug #34833 Listener: Fix error message


FYI: UCS-3.2-2 needs to be back-merged to UCS-3.2-1
Comment 5 Philipp Hahn univentionstaff 2014-05-22 14:43:09 CEST
*** Bug 34785 has been marked as a duplicate of this bug. ***
Comment 6 Stefan Gohmann univentionstaff 2014-05-22 19:52:34 CEST
(In reply to Philipp Hahn from comment #4)
> r50549 | Bug #34835 translog: Use normalized DN

As discussed, now it is always used the lower case DN.

slapd version: 2.4.35-1.160.201405221112

root@master211:~# udm computers/windows create --set name=TEST1
Object created: cn=TEST1,dc=deadlock21,dc=local
root@master211:~# univention-ldapsearch cn=TEST1 dn -LLL
dn: cn=TEST1,dc=deadlock21,dc=local

root@master211:~# grep -i test1 /var/lib/univention-ldap/notify/transaction
1137 cn=test1$,cn=uid,cn=temporary,cn=univention,dc=deadlock21,dc=local a
1138 cn=test1,dc=deadlock21,dc=local a
1139 cn=test1$,cn=uid,cn=temporary,cn=univention,dc=deadlock21,dc=local d
1140 cn=test1,dc=deadlock21,dc=local m
1141 cn=test1,dc=deadlock21,dc=local m
1144 cn=test1,dc=deadlock21,dc=local m
root@master211:~#
Comment 7 Stefan Gohmann univentionstaff 2014-05-23 09:04:52 CEST
*** Bug 34925 has been marked as a duplicate of this bug. ***
Comment 8 Philipp Hahn univentionstaff 2014-05-23 09:44:37 CEST
(In reply to Stefan Gohmann from comment #6)
> (In reply to Philipp Hahn from comment #4)
> > r50549 | Bug #34835 translog: Use normalized DN
> 
> As discussed, now it is always used the lower case DN.

Reading the source of OpenLDAP "dnNormalize()" is called, which calls the normalization and/or pretty-printing function as defined for each syntax class to create a unique DN, which is e.g. used for looking up the entry in the BDB backend. Most convert the string to lower-case, so this is not the expected solution.

r13112: The change to OpenLDAP has been reverted, the packages been removed from scope.
Comment 9 Philipp Hahn univentionstaff 2014-05-26 08:12:23 CEST
The change was reverted.
One more case to use same_dn()

univention-directory-listener_8.0.1-4.228.201405260820
Comment 10 Philipp Hahn univentionstaff 2014-05-26 15:45:25 CEST
r50648 | Bug #34835 Listener: More same_dn() tests
r50650 | Bug #34835 Listener: Ignore case when comparing DNs for moves
r50686 | Bug #34835 Listener: Change collision message to level INFO

univention-directory-listener_8.0.1-5.229.201405261549
Comment 11 Stefan Gohmann univentionstaff 2014-05-27 07:48:09 CEST
(In reply to Philipp Hahn from comment #4)
> FYI: UCS-3.2-2 needs to be back-merged to UCS-3.2-1

Yes, please merge it.

YAML: OK

Code: OK

Tests: OK (I've added a test case: r50696: 10_ldap/42replication_uppercase)
Comment 12 Philipp Hahn univentionstaff 2014-05-27 08:39:11 CEST
(In reply to Stefan Gohmann from comment #11)
> (In reply to Philipp Hahn from comment #4)
> > FYI: UCS-3.2-2 needs to be back-merged to UCS-3.2-1
> 
> Yes, please merge it.

r50697
(note that the version for UCS-3.2-1 is 8.0.0-12, while for UCS-3.2-2 it is 8.0.1-5. The errata is scheduled for both 3.2-1 and 3.2-2, so the next errata for 3.2-1 only will have to be careful with the next version. Could be solved by building UDL for 3.2-1 only and doing two separate announcements.)
Comment 13 Janek Walkenhorst univentionstaff 2014-05-27 11:21:45 CEST
http://errata.univention.de/ucs/3.2/116.html