Bug 45779 - univention-connector-list-rejected output encoding
univention-connector-list-rejected output encoding
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: AD Connector
UCS 4.2
Other Linux
: P5 normal (vote)
: UCS 4.3-0-errata
Assigned To: Felix Botner
Arvid Requate
:
: 45226 (view as bug list)
Depends on: 23289
Blocks: 47430
  Show dependency treegraph
 
Reported: 2017-11-27 16:12 CET by Nico Stöckigt
Modified: 2018-08-01 12:40 CEST (History)
6 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.057
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number: 2017112521000086
Bug group (optional): Internationalization
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nico Stöckigt univentionstaff 2017-11-27 16:12:14 CET
+++ This bug was initially created as a clone of Bug #23289 +++

DNs mit speziellen Zeichen (z.B. Accent ...) werden von univention-connector-list-rejected nicht korrekt ausgegeben.

======================================================================
When calling univention-connector-list-rejected

Traceback (most recent call last):
  File "/usr/sbin/univention-connector-list-rejected", line 191, in <module>
    main()
  File "/usr/sbin/univention-connector-list-rejected", line 176, in main
    print "%5d:    AD DN: %s" % (i, univention.connector.ad.encode_attrib(dn).encode('latin'))
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u200b' in position 29: ordinal not in range(256)

Exitcode was 1

--------

In /var/log/univention/connector.log

 25.11.2017 08:43:11,498 LDAP        (ERROR  ): Unknown Exception during sync_to_ucs
 25.11.2017 08:43:11,498 LDAP        (ERROR  ): Traceback (most recent call last):
   File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 1357, in sync_to_ucs
     result = self.modify_in_ucs(property_type, object, module, position)
   File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 1194, in modify_in_ucs
     return ucs_object.modify() and self.__modify_custom_attributes(property_type, object, ucs_object, module, position)
   File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 419, in modify
     dn = self._modify(modify_childs, ignore_license=ignore_license, response=response)
   File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 1082, in _modify
     self.lo.modify(self.dn, ml, ignore_license=ignore_license, serverctrls=serverctrls, response=response)
   File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 505, in modify
     raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
 ldapError: Other (e.g., implementation specific) error: DN index delete fail

--------
A dns domain transfer is no longer possible

; <<>> DiG 9.9.5-9+deb8u13A~4.2.1.201708081700-Univention <<>> @10.123.45.113 domain.local -t AXFR
; (1 server found)
;; global options: +cmd
; Transfer failed.

--------
Version info

UCS: 4.2-2 errata231
App Center compatibility: 4
Installed: adconnector=11.0 kopano-core=8.3.1.32 kopano-webapp=3.3.0.610 nagios=3.5 samba-memberserver=4.6 z-push-kopano=2.3.7
ad/member: true
samba/role: memberserver
server/role: domaincontroller_master
Comment 1 Nico Stöckigt univentionstaff 2017-11-27 16:17:18 CET
worth mentioning there is already a patch at the original bug.
Comment 2 Florian Best univentionstaff 2017-11-27 18:28:20 CET
(In reply to Nico Stöckigt from comment #1)
> worth mentioning there is already a patch at the original bug.

I would not use that patch but remove the encoding completely. UCS is UTF-8 based (not latin-1 anymore since years).
Comment 3 Nico Stöckigt univentionstaff 2017-11-28 12:01:12 CET
Isn't the problem that ad-connector still uses 'latin-1' in code, at least the check-rejects-script?
What exactly happens when there are special chars in dns - are we handle this right all the time?
Comment 4 Florian Best univentionstaff 2017-11-28 12:19:00 CET
(In reply to Nico Stöckigt from comment #3)
> Isn't the problem that ad-connector still uses 'latin-1' in code, at least
> the check-rejects-script?
> What exactly happens when there are special chars in dns - are we handle
> this right all the time?
In our OpenLDAP most attributes which are parts of an DN have a syntax can only contain UTF-8. I think this is the same in AD (but not sure). So yes, using latin-1 is wrong here: And the traceback here is the proove for it.
Comment 5 Arvid Requate univentionstaff 2017-11-30 22:16:27 CET
This whole latin-1 en+decoding handling in the connector should be checked. My impression from reading the code was that it is a pretty useless exercise. And I think we are just lucky that it somehow gives consistent results. Microsoft uses UTF-16LE in many cases IIRC, not sure about Active Directory values. Probably depends on the specific attribute.
Comment 6 Arvid Requate univentionstaff 2017-11-30 22:19:13 CET
But let's keep this focussed on the real issue at hand here: The output of the tools is not correct. Let's fix that here, rather then messing with connector internals, if possible,
Comment 7 Stefan Gohmann univentionstaff 2018-03-15 06:25:16 CET
Move to 4.3-0-errata. If a UCS 4.2 backport is needed, please clone this issue.
Comment 8 Felix Botner univentionstaff 2018-05-08 14:22:26 CEST
univention-ad-connector db26f32e333a34a4457322c5b30f7a370b2fa3d8

changes:

_save_rejected_ucs() and _save_rejected(): 
both now encode_attrib the dn (latin no longer supported in sqlite)

ad/__init__.py.resync_rejected():
do not try to decode Unicode

univention-connector-list-rejected:
do not encode('latin') the dn's

univention-adsearch:
removed encoding (latin) stuff

tested:
 * UCS/AD rejects with special characters is saved in sqlite
 * univention-connector-list-rejected works and prints the rejects
 * rejects can be processed
Comment 9 Felix Botner univentionstaff 2018-05-08 14:23:34 CEST
*** Bug 45226 has been marked as a duplicate of this bug. ***
Comment 10 Arvid Requate univentionstaff 2018-05-15 18:07:19 CEST
As discussed, maybe we should keep the compatible_modstring in _save_rejected in:

services/univention-ad-connector/modules/univention/connector/ad/__init__.py

that's how it is in the S4-Connector. Otherwise we would have to adjust the code to handle existing rejects too (which would still be utf-8).
Comment 11 Felix Botner univentionstaff 2018-05-15 19:08:15 CEST
(In reply to Arvid Requate from comment #10)
> As discussed, maybe we should keep the compatible_modstring in
> _save_rejected in:
> 
> services/univention-ad-connector/modules/univention/connector/ad/__init__.py
> 
> that's how it is in the S4-Connector. Otherwise we would have to adjust the
> code to handle existing rejects too (which would still be utf-8).

sqlite in 4.3 (and 4.2) accepts only unicode, we must not use utf-8 in _save_rejected  but unicode (encode_attrib) and not convert to unicode in resync_rejected

see Bug 47013 for s4 connector
Comment 12 Arvid Requate univentionstaff 2018-05-15 19:19:16 CEST
To quote Jannek: "I concur".
Comment 13 Arvid Requate univentionstaff 2018-05-16 17:03:56 CEST
<http://errata.software-univention.de/ucs/4.3/36.html>