Univention Bugzilla – Bug 44276
w2k3 de connector reject for Domänen-Benutzer
Last modified: 2017-04-21 11:03:04 CEST
Ad takeover with german w2k3 works, but i get numerous tracebacks in the connector log. 04.04.2017 09:52:35,141 LDAP (PROCESS): sync from ucs: [ group] [ modify] cn=Domänen-Benutzer,cn=groups,DC=w2k3,DC=test 04.04.2017 09:52:35,149 LDAP (WARNING): sqlite: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. 04.04.2017 09:52:35,150 LDAP (WARNING): sqlite: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. 04.04.2017 09:52:35,150 LDAP (WARNING): sync failed, saved as rejected /var/lib/univention-connector/s4/1491236336.643083 04.04.2017 09:52:35,150 LDAP (WARNING): Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 843, in __sync_file_from_ucs if ((old_dn and not self.sync_from_ucs(key, object, premapped_ucs_dn, unicode(old_dn, 'utf8'), old, new)) or (not old_dn and not self.sync_from_ucs(key, object, premapped_ucs_dn, old_dn, old, new))): File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 2510, in sync_from_ucs objectSid = decode_sid(objectSid_attr_value) File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 522, in decode_sid sid += "%d" % ord(value[0]) TypeError: 'NoneType' object has no attribute '__getitem__' As the connector constantly tries to sync this objects the connector log gets very big very fast: -> du -h /var/log/univention/connector-s4.log 2,3G /var/log/univention/connector-s4.log
This applies of course to all groups (and users?) with umlauts 04.04.2017 09:59:32,631 LDAP (PROCESS): sync from ucs: [ group] [ add] cn=Domänencomputer,cn=groups,DC=w2k3,DC=test 04.04.2017 09:59:32,640 LDAP (WARNING): sqlite: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. 04.04.2017 09:59:32,640 LDAP (WARNING): sqlite: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. 04.04.2017 09:59:32,640 LDAP (WARNING): sync failed, saved as rejected /var/lib/univention-connector/s4/1491234894.123114 04.04.2017 09:59:32,641 LDAP (WARNING): Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 843, in __sync_file_from_ucs if ((old_dn and not self.sync_from_ucs(key, object, premapped_ucs_dn, unicode(old_dn, 'utf8'), old, new)) or (not old_dn and not self.sync_from_ucs(key, object, premapped_ucs_dn, old_dn, old, new))): File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 2510, in sync_from_ucs objectSid = decode_sid(objectSid_attr_value) File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 522, in decode_sid sid += "%d" % ord(value[0]) TypeError: 'NoneType' object has no attribute '__getitem__'
We should test again with W2K3 R2.
(In reply to Arvid Requate from comment #2) > We should test again with W2K3 R2. also happens with w2k12
At least two things are happening here: 1) Known Bug: fetching the SID in sync_rom_ucs ALREADY_EXISTS handling fails (code introduced via Bug 41864 / Bug 42120), patch is attached to Bug 43368. 2) The connector tries to write 'cn=Domänen-Benutzer,cn=groups,dc=w2k12,dc=test' as reject into sqlite3 which doesn#t accept this utf-8 string. When I convert to unicode I end up with this string in sqlite3 and see it in univention-s4connector-list-rejected, which probably isn't what we want. I guess something must have changed encoding-wise in the S4-Connector. When fixing the second part, we should also change the "(WARNING): sqlite: " into an ERROR.
Ok, the sqlite problem for groups (and users) already happens in UCS 4.1-4 errata407. I triggered it by writing "invalidstuff" into /usr/share/pyshared/univention/s4connector/s4/__init__.py +2410 (just before the ldap.ALREADY_EXISTS): ========================================================================== 03.05.2016 01:33:13,727 LDAP (PROCESS): sync from ucs: [ group] [ add] cn=Domänen Gruppe,cn=groups,DC=ar41i2,DC=qa 03.05.2016 01:33:13,754 LDAP (ERROR ): sync_from_ucs: traceback during add object: cn=Domänen Gruppe,cn=groups,DC=ar41i2,DC=qa 03.05.2016 01:33:13,754 LDAP (ERROR ): sync_from_ucs: traceback due to addlist: [('objectClass', ['top', 'group']), ('groupType', [u'-2147483646']), ('sAMAccountName', [u'Dom\xe4nen Gruppe'])] 03.05.2016 01:33:13,754 LDAP (WARNING): sqlite: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. 03.05.2016 01:33:13,755 LDAP (WARNING): sqlite: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. 03.05.2016 01:33:13,756 LDAP (WARNING): sync failed, saved as rejected /var/lib/univention-connector/s4/1462231951.138939 03.05.2016 01:33:13,756 LDAP (WARNING): Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 843, in __sync_file_from_ucs if ((old_dn and not self.sync_from_ucs(key, object, premapped_ucs_dn, unicode(old_dn, 'utf8'), old, new)) or (not old_dn and not self.sync_from_ucs(key, object, premapped_ucs_dn, old_dn, old, new))): File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 2410, in sync_from_ucs invalidstuff NameError: global name 'invalidstuff' is not defined ==========================================================================
Ok, I can trigger the primary issue that causes the reject (the sqlite issue is a secondary effect), and it only happens for groups with umlauts that already exist in Samba4 when the connector starts. It doesn't happen for users. And it only happens in UCS 4.2, it doesn't happen in UCS 4.1-4 errata407. ========================================================================= root@master100:~# /etc/init.d/univention-s4-connector stop [ ok ] Stopping univention-s4-connector (via systemctl): univention-s4-connector.service. root@master100:~# samba-tool group add Domänen-Gruppe Added group Domänen-Gruppe root@master100:~# ucr set directory/manager/web/modules/groups/group/properties/name/syntax=string Create directory/manager/web/modules/groups/group/properties/name/syntax root@master100:~# pkill -f cli root@master100:~# udm groups/group create --position="cn=groups,$(ucr get ldap/base)" --set name=Domänen-Gruppe Object created: cn=Domänen-Gruppe,cn=groups,dc=ar41pt1,dc=qa root@master100:~# /etc/init.d/univention-s4-connector start [ ok ] Starting univention-s4-connector (via systemctl): univention-s4-connector.service. =========================================================================
Created attachment 8754 [details] proposed_s4_connector_utf8_fixes.patch r78659 provides a minimal fix. The attached patch is a proposal for additional utf-8 fixes, but I'm not sure about those, please review. Advisory: univention-s4-connector.yaml
I've split of the sqlite issue as Bug 44291
*** Bug 44277 has been marked as a duplicate of this bug. ***
Created attachment 8756 [details] Alternative UTF-8 fixes I would propose this slightly modified version. `convert_field()` called `str()` to enable something like `format_escaped("{0!e}", 1)` where the value was not a string. This modified version accepts basestrings (str or unicode) and only calls `str()` otherwise. I also removed the `format_escaped(u"...", ...)` modifications. My tests show no necessity. Only the type of `value` gets promoted to the result type. >>> format_escaped("{0!e}", "ää") '\xc3\xa4\xc3\xa4' >>> format_escaped(u"{0!e}", "ää") '\xc3\xa4\xc3\xa4' >>> format_escaped("{0!e}", u"ää") u'\xe4\xe4' >>> format_escaped(u"{0!e}", u"ää") u'\xe4\xe4' This does not contradict the fix as commited in r78659, as `"{}".format()` behaves differently. >>> "{}".format("ää") '\xc3\xa4\xc3\xa4' >>> u"{}".format("ää") Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128) >>> "{}".format(u"ää") Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128) >>> u"{}".format(u"ää") u'\xe4\xe4' >>> >>> u"{}".format("aa") u'aa'
Ok, I've rebuilt the package with Lukas' version of the patch. Advisory updated.
several tracebacks like 11.04.2017 13:03:51,976 LDAP (PROCESS): sync from ucs: [ user] [ modify] cn=administrator,cn=users,DC=w2k12,DC=test 11.04.2017 13:06:12,772 LDAP (WARNING): Exception during poll/sync_to_ucs 11.04.2017 13:06:12,814 LDAP (WARNING): Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 2311, in poll mapped_object = self._object_mapping(property_key, object) File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 1768, in _object_mapping object = function(self, object, dn_mapping_stored, isUCSobject=(object_type == 'ucs')) File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 413, in user_dn_mapping return samaccountname_dn_mapping(s4connector, given_object, dn_mapping_stored, isUCSobject, 'user', u'samAccountName', u'posixAccount', 'uid', u'user') File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/__init__.py", line 395, in samaccountname_dn_mapping newdn = unicode(ldap.dn.dn2str([newdn_rdn] + exploded_dn[1:]), 'utf8') # guess the old dn TypeError: decoding Unicode is not supported now
unicode(foo, 'utf8') doesn't work for unicode foo. I've also looked at the commits for Bug 32086 which introduced the str2dn functions, which don't accept unicode. I've added a new helper function "unicode_to_utf8" (as a drop in replacement for the dreaded old "compatible_modstring" function) and use that in most places where str2dn is used now, except two places that used ldap.explode_dn before. We have to properly sanitize the handling of encoding at a later stage. My commit only tries to avoid new problems due to str2dn/dn2str/unicode. Package rebuilt and advisory updated.
Still get lots of 12.04.2017 10:18:08,941 LDAP (PROCESS): sync from ucs: Resync rejected file: /var/lib/univention-connector/s4/1491984969.714697 12.04.2017 10:18:08,949 LDAP (WARNING): sync failed, saved as rejected /var/lib/univention-connector/s4/1491984969.714697 12.04.2017 10:18:08,950 LDAP (WARNING): Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 987, in resync_rejected_ucs if self.__sync_file_from_ucs(filename, append_error=' rejected'): File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 839, in __sync_file_from_ucs object = self._object_mapping(key, object, 'ucs') File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 1768, in _object_mapping object = function(self, object, dn_mapping_stored, isUCSobject=(object_type == 'ucs')) File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/dns.py", line 233, in dns_dn_mapping target_zone_dn = unicode(ldap.dn.dn2str([target_zone_rdn] + exploded_dn[2:]), 'utf8') TypeError: decoding Unicode is not supported directly after the installation.
Ok, fixed, rebuilt, advisory updated.
OK - takeover with german ad OK - takeover with french ad OK - UCS samba4 installation OK - YAML
<http://errata.software-univention.de/ucs/4.2/2.html>