Univention Bugzilla – Bug 45779
univention-connector-list-rejected output encoding
Last modified: 2018-08-01 12:40:31 CEST
+++ This bug was initially created as a clone of Bug #23289 +++
DNs mit speziellen Zeichen (z.B. Accent ...) werden von univention-connector-list-rejected nicht korrekt ausgegeben.
When calling univention-connector-list-rejected
Traceback (most recent call last):
File "/usr/sbin/univention-connector-list-rejected", line 191, in <module>
File "/usr/sbin/univention-connector-list-rejected", line 176, in main
print "%5d: AD DN: %s" % (i, univention.connector.ad.encode_attrib(dn).encode('latin'))
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u200b' in position 29: ordinal not in range(256)
Exitcode was 1
25.11.2017 08:43:11,498 LDAP (ERROR ): Unknown Exception during sync_to_ucs
25.11.2017 08:43:11,498 LDAP (ERROR ): Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 1357, in sync_to_ucs
result = self.modify_in_ucs(property_type, object, module, position)
File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 1194, in modify_in_ucs
return ucs_object.modify() and self.__modify_custom_attributes(property_type, object, ucs_object, module, position)
File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 419, in modify
dn = self._modify(modify_childs, ignore_license=ignore_license, response=response)
File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 1082, in _modify
self.lo.modify(self.dn, ml, ignore_license=ignore_license, serverctrls=serverctrls, response=response)
File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 505, in modify
raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
ldapError: Other (e.g., implementation specific) error: DN index delete fail
A dns domain transfer is no longer possible
; <<>> DiG 9.9.5-9+deb8u13A~22.214.171.124708081700-Univention <<>> @10.123.45.113 domain.local -t AXFR
; (1 server found)
;; global options: +cmd
; Transfer failed.
UCS: 4.2-2 errata231
App Center compatibility: 4
Installed: adconnector=11.0 kopano-core=126.96.36.199 kopano-webapp=188.8.131.520 nagios=3.5 samba-memberserver=4.6 z-push-kopano=2.3.7
worth mentioning there is already a patch at the original bug.
(In reply to Nico Stöckigt from comment #1)
> worth mentioning there is already a patch at the original bug.
I would not use that patch but remove the encoding completely. UCS is UTF-8 based (not latin-1 anymore since years).
Isn't the problem that ad-connector still uses 'latin-1' in code, at least the check-rejects-script?
What exactly happens when there are special chars in dns - are we handle this right all the time?
(In reply to Nico Stöckigt from comment #3)
> Isn't the problem that ad-connector still uses 'latin-1' in code, at least
> the check-rejects-script?
> What exactly happens when there are special chars in dns - are we handle
> this right all the time?
In our OpenLDAP most attributes which are parts of an DN have a syntax can only contain UTF-8. I think this is the same in AD (but not sure). So yes, using latin-1 is wrong here: And the traceback here is the proove for it.
This whole latin-1 en+decoding handling in the connector should be checked. My impression from reading the code was that it is a pretty useless exercise. And I think we are just lucky that it somehow gives consistent results. Microsoft uses UTF-16LE in many cases IIRC, not sure about Active Directory values. Probably depends on the specific attribute.
But let's keep this focussed on the real issue at hand here: The output of the tools is not correct. Let's fix that here, rather then messing with connector internals, if possible,
Move to 4.3-0-errata. If a UCS 4.2 backport is needed, please clone this issue.
_save_rejected_ucs() and _save_rejected():
both now encode_attrib the dn (latin no longer supported in sqlite)
do not try to decode Unicode
do not encode('latin') the dn's
removed encoding (latin) stuff
* UCS/AD rejects with special characters is saved in sqlite
* univention-connector-list-rejected works and prints the rejects
* rejects can be processed
*** Bug 45226 has been marked as a duplicate of this bug. ***
As discussed, maybe we should keep the compatible_modstring in _save_rejected in:
that's how it is in the S4-Connector. Otherwise we would have to adjust the code to handle existing rejects too (which would still be utf-8).
(In reply to Arvid Requate from comment #10)
> As discussed, maybe we should keep the compatible_modstring in
> _save_rejected in:
> that's how it is in the S4-Connector. Otherwise we would have to adjust the
> code to handle existing rejects too (which would still be utf-8).
sqlite in 4.3 (and 4.2) accepts only unicode, we must not use utf-8 in _save_rejected but unicode (encode_attrib) and not convert to unicode in resync_rejected
see Bug 47013 for s4 connector
To quote Jannek: "I concur".