Bug 47232 - Make a ad-connector resync configurable for single OUs
Make a ad-connector resync configurable for single OUs
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: AD Connector
UCS 4.2
Other Linux
: P5 normal (vote)
: UCS 4.3-1-errata
Assigned To: Arvid Requate
Felix Botner
https://trello.com/c/mR5RIOeh/88-bug-...
:
: 46254 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-06-22 10:43 CEST by Christina Scheinig
Modified: 2018-08-15 13:14 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 2: Improvement: Would be a product improvement
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.046
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support: Yes
Ticket number: 2018061221000411, 2018081321000201
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christina Scheinig univentionstaff 2018-06-22 10:43:38 CEST
A customer needs to resync the ad-connector, but only for some OUs.
Because of the large environment a resync takes a long time. It would be an improvement to make 
univention-directory-listener-ctrl resync ad-connector
configurable for single OUs.
Comment 1 Stefan Gohmann univentionstaff 2018-06-22 11:29:18 CEST
I think we should merge the following S4 Connector scripts to the AD Connector package:

- remove_s4_rejected.py
- remove_ucs_rejected.py
- resync_object_from_s4.py
- resync_object_from_ucs.py
Comment 2 Arvid Requate univentionstaff 2018-07-25 19:01:35 CEST
c0f93f0773 | Adjust __search_ad for resync_object_from_ad.py
b84d57512a | Add resync_object_from_* and remove_*_rejected
f244f338af | Merge branch 'arequate/bug47232' into 4.3-1
7c3cbd9972 | Advisory
Comment 3 Arvid Requate univentionstaff 2018-07-26 10:25:56 CEST
30b57c73ee | Add short options to resync_object_* and remove_*_rejected
47e3e3f179 | Advisory
Comment 4 Felix Botner univentionstaff 2018-08-01 12:51:34 CEST
OK - remove_s4_rejected.py
OK - remove_ucs_rejected.py
OK - resync_object_from_s4.py
OK - resync_object_from_ucs.py

OK - YAML
Comment 5 Arvid Requate univentionstaff 2018-08-02 12:04:57 CEST
*** Bug 46254 has been marked as a duplicate of this bug. ***
Comment 6 Felix Botner univentionstaff 2018-08-02 16:00:36 CEST
(1)
-> resync_object_from_ad.py -b "DC=w2k12,DC=test"

Traceback (most recent call last):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 250, in <module>
    treated_dns = resync.resync(ad_dns, options.ldapfilter, options.ldapbase)
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 100, in resync
    for ad_dn, guid, usn in self.search_ad(ad_dns, ldapfilter, ldapbase):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 150, in search_ad
    guid_blob = msg[1]["objectGUID"][0]
TypeError: list indices must be integers, not str

msg is (None, [u'ldap://ForestDnsZones.w2k12.test/DC=ForestDnsZones,DC=w2k12,DC=test']) at this point

patch?
                for msg in res:
-                       guid_blob = msg[1]["objectGUID"][0]
-                       guid = ndr_unpack(misc.GUID, guid_blob)
-                       usn = msg[1]["uSNChanged"][0]
-                       search_result.append((str(msg[0]), guid, usn))
+                       if msg[0] is not None:
+                               guid_blob = msg[1]["objectGUID"][0]
+                               guid = ndr_unpack(misc.GUID, guid_blob)
+                               usn = msg[1]["uSNChanged"][0]
+                               search_result.append((str(msg[0]), guid, usn))


(2) 

# wrong base
-> resync_object_from_ucs.py -b "DC=w2k12,DC=testdoesnotexist"

Traceback (most recent call last):
  File "/usr/share/univention-ad-connector/resync_object_from_ucs.py", line 151, in <module>
    print 'ERROR: The LDAP object not found : %s' % ex.args[1]
IndexError: tuple index out of range

patch?

        except ldap.NO_SUCH_OBJECT as ex:
-               print 'ERROR: The LDAP object not found : %s' % ex.args[1]
+               print 'ERROR: The LDAP object not found : %s' % str(ex)


(3) 
with patch from (1) i can trigger the resync of all objects from ad, which the connector can't handle currently

-> resync_object_from_ad.py -b "DC=w2k12,DC=test"

after that i have 67 rejects

connector.log

02.08.2018 16:00:05,985 LDAP        (PROCESS): sync to ucs: Resync rejected dn: CN=BCKUPKEY_PREFERRED Secret,CN=System,DC=w2k12,DC=test
02.08.2018 16:00:05,987 LDAP        (ERROR  ): unexpected Error during ad.resync_rejected
02.08.2018 16:00:05,987 LDAP        (ERROR  ): Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/connector/ad/__init__.py", line 2185, in resync_rejected
    mapped_object = self._object_mapping(property_key, object)
  File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 1704, in _object_mapping
    for attr_key in self.property[key].attributes.keys():
KeyError: None
Comment 7 Arvid Requate univentionstaff 2018-08-07 19:30:49 CEST
66c704c3c2 | Fixes, see below
701ebef93a | Advisory

* Improve robustness of object resync by skipping unidentified AD objects
* Skip referrals in resync_object_from_ad.py
* Fix exception handling in resync_object_from_ucs.py
Comment 8 Felix Botner univentionstaff 2018-08-08 13:53:32 CEST
 
> * Improve robustness of object resync by skipping unidentified AD objects

OK

> * Skip referrals in resync_object_from_ad.py

FAIL - also around line 147

> * Fix exception handling in resync_object_from_ucs.py

OK

and another thing, resync_object_from_ad.py can't handle wrong DN's

resync_object_from_ad.py -b "c=four,dc=threeeee"
resync_object_from_ad.py "c=four,dc=threeeee"


Traceback (most recent call last):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 250, in <module>
    treated_dns = resync.resync(ad_dns, options.ldapfilter, options.ldapbase)
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 100, in resync
    for ad_dn, guid, usn in self.search_ad(ad_dns, ldapfilter, ldapbase):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 122, in search_ad
    res = self.__search_ad(base=targetdn, scope=ldap.SCOPE_BASE, filter=ldapfilter, attrlist=["objectGUID", "uSNChanged"])
  File "/usr/lib/pymodules/python2.7/univention/connector/ad/__init__.py", line 1187, in __search_ad
    rtype, rdata, rmsgid, serverctrls = self.lo_ad.lo.result3(msgid)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3
    resp_ctrl_classes=resp_ctrl_classes
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
    result = func(*args,**kwargs)
ldap.REFERRAL: {'info': 'Referral:\nldap://threeeee/c=four,dc=threeeee', 'desc': 'Referral'}

better give a proper error message like resync_object_from_ucs.py:

-> resync_object_from_ucs.py  "cn=fouer,dc=threeeeee"
ERROR: The LDAP object not found : (1, "No object: ['cn=fouer,dc=threeeeee']", [])
-> resync_object_from_ucs.py  -b "cn=fouer,dc=threeeeee"
ERROR: The LDAP object not found : {'desc': 'No such object'}
Comment 9 Arvid Requate univentionstaff 2018-08-13 19:25:58 CEST
474f60821a | Improve robustness of object resync
dd1c0a289f | Advisory
Comment 10 Felix Botner univentionstaff 2018-08-14 10:19:20 CEST
still fails with "-b"

resync_object_from_ad.py -b cn=a,dc=b

Traceback (most recent call last):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 254, in <module>
    treated_dns = resync.resync(ad_dns, options.ldapfilter, options.ldapbase)
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 100, in resync
    for ad_dn, guid, usn in self.search_ad(ad_dns, ldapfilter, ldapbase):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 149, in search_ad
    res = self.__search_ad(base=ldapbase, scope=ldap.SCOPE_SUBTREE, filter=ldapfilter, attrlist=["objectGUID", "uSNChanged"])
  File "/usr/lib/pymodules/python2.7/univention/connector/ad/__init__.py", line 1187, in __search_ad
    rtype, rdata, rmsgid, serverctrls = self.lo_ad.lo.result3(msgid)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3
    resp_ctrl_classes=resp_ctrl_classes
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
    result = func(*args,**kwargs)
ldap.REFERRAL: {'info': 'Referral:\nldap://b/cn=a,dc=b', 'desc': 'Referral'}
Comment 11 Arvid Requate univentionstaff 2018-08-14 10:36:20 CEST
c5703c4ba7 | handle exception
9d3bdc1466 | Advisory
Comment 12 Felix Botner univentionstaff 2018-08-14 15:10:28 CEST
/usr/share/univention-ad-connector/resync_object_from_ad.py -b cn=a,dc=b

Traceback (most recent call last):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 257, in <module>
    treated_dns = resync.resync(ad_dns, options.ldapfilter, options.ldapbase)
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 100, in resync
    for ad_dn, guid, usn in self.search_ad(ad_dns, ldapfilter, ldapbase):
  File "/usr/share/univention-ad-connector/resync_object_from_ad.py", line 160, in search_ad
    error_dns.append((targetdn, str(ex)))
UnboundLocalError: local variable 'error_dns' referenced before assignment
Comment 13 Arvid Requate univentionstaff 2018-08-14 21:00:04 CEST
Damn, I owe you a drink!

e1a6b7037e | Fix broken code
1168d0d56a | Advisory
Comment 14 Felix Botner univentionstaff 2018-08-15 12:28:30 CEST
OK - univention-ad-connector
OK - yaml
Comment 15 Arvid Requate univentionstaff 2018-08-15 13:14:31 CEST
<http://errata.software-univention.de/ucs/4.3/168.html>