Univention Bugzilla – Bug 45779
univention-connector-list-rejected output encoding
Last modified: 2018-08-01 12:40:31 CEST
+++ This bug was initially created as a clone of Bug #23289 +++ DNs mit speziellen Zeichen (z.B. Accent ...) werden von univention-connector-list-rejected nicht korrekt ausgegeben. ====================================================================== When calling univention-connector-list-rejected Traceback (most recent call last): File "/usr/sbin/univention-connector-list-rejected", line 191, in <module> main() File "/usr/sbin/univention-connector-list-rejected", line 176, in main print "%5d: AD DN: %s" % (i, univention.connector.ad.encode_attrib(dn).encode('latin')) UnicodeEncodeError: 'latin-1' codec can't encode character u'\u200b' in position 29: ordinal not in range(256) Exitcode was 1 -------- In /var/log/univention/connector.log 25.11.2017 08:43:11,498 LDAP (ERROR ): Unknown Exception during sync_to_ucs 25.11.2017 08:43:11,498 LDAP (ERROR ): Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 1357, in sync_to_ucs result = self.modify_in_ucs(property_type, object, module, position) File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 1194, in modify_in_ucs return ucs_object.modify() and self.__modify_custom_attributes(property_type, object, ucs_object, module, position) File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 419, in modify dn = self._modify(modify_childs, ignore_license=ignore_license, response=response) File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 1082, in _modify self.lo.modify(self.dn, ml, ignore_license=ignore_license, serverctrls=serverctrls, response=response) File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 505, in modify raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) ldapError: Other (e.g., implementation specific) error: DN index delete fail -------- A dns domain transfer is no longer possible ; <<>> DiG 9.9.5-9+deb8u13A~4.2.1.201708081700-Univention <<>> @10.123.45.113 domain.local -t AXFR ; (1 server found) ;; global options: +cmd ; Transfer failed. -------- Version info UCS: 4.2-2 errata231 App Center compatibility: 4 Installed: adconnector=11.0 kopano-core=8.3.1.32 kopano-webapp=3.3.0.610 nagios=3.5 samba-memberserver=4.6 z-push-kopano=2.3.7 ad/member: true samba/role: memberserver server/role: domaincontroller_master
worth mentioning there is already a patch at the original bug.
(In reply to Nico Stöckigt from comment #1) > worth mentioning there is already a patch at the original bug. I would not use that patch but remove the encoding completely. UCS is UTF-8 based (not latin-1 anymore since years).
Isn't the problem that ad-connector still uses 'latin-1' in code, at least the check-rejects-script? What exactly happens when there are special chars in dns - are we handle this right all the time?
(In reply to Nico Stöckigt from comment #3) > Isn't the problem that ad-connector still uses 'latin-1' in code, at least > the check-rejects-script? > What exactly happens when there are special chars in dns - are we handle > this right all the time? In our OpenLDAP most attributes which are parts of an DN have a syntax can only contain UTF-8. I think this is the same in AD (but not sure). So yes, using latin-1 is wrong here: And the traceback here is the proove for it.
This whole latin-1 en+decoding handling in the connector should be checked. My impression from reading the code was that it is a pretty useless exercise. And I think we are just lucky that it somehow gives consistent results. Microsoft uses UTF-16LE in many cases IIRC, not sure about Active Directory values. Probably depends on the specific attribute.
But let's keep this focussed on the real issue at hand here: The output of the tools is not correct. Let's fix that here, rather then messing with connector internals, if possible,
Move to 4.3-0-errata. If a UCS 4.2 backport is needed, please clone this issue.
univention-ad-connector db26f32e333a34a4457322c5b30f7a370b2fa3d8 changes: _save_rejected_ucs() and _save_rejected(): both now encode_attrib the dn (latin no longer supported in sqlite) ad/__init__.py.resync_rejected(): do not try to decode Unicode univention-connector-list-rejected: do not encode('latin') the dn's univention-adsearch: removed encoding (latin) stuff tested: * UCS/AD rejects with special characters is saved in sqlite * univention-connector-list-rejected works and prints the rejects * rejects can be processed
*** Bug 45226 has been marked as a duplicate of this bug. ***
As discussed, maybe we should keep the compatible_modstring in _save_rejected in: services/univention-ad-connector/modules/univention/connector/ad/__init__.py that's how it is in the S4-Connector. Otherwise we would have to adjust the code to handle existing rejects too (which would still be utf-8).
(In reply to Arvid Requate from comment #10) > As discussed, maybe we should keep the compatible_modstring in > _save_rejected in: > > services/univention-ad-connector/modules/univention/connector/ad/__init__.py > > that's how it is in the S4-Connector. Otherwise we would have to adjust the > code to handle existing rejects too (which would still be utf-8). sqlite in 4.3 (and 4.2) accepts only unicode, we must not use utf-8 in _save_rejected but unicode (encode_attrib) and not convert to unicode in resync_rejected see Bug 47013 for s4 connector
To quote Jannek: "I concur".
<http://errata.software-univention.de/ucs/4.3/36.html>