Univention Bugzilla – Bug 52044
services/univention-ad-connector: Migrate to Python 3
Last modified: 2021-05-25 16:02:43 CEST
The univention-ad-connector should be Python 3 compatible. As well as the string-changes in UDM have to be integrated.
The univention-ad-connector package is currently broken due to missing dependencies (e.g. python-ldb). Due to work on samba that was expected. Please revert and rebuild the cd than this is fixed: [5.0-0 6c2956f894] Bug #52044: Do not copy broken ad-connector package Thanks!
WIP branch in fbest/52044-ad-connector-python3.
Is this ready for QA?
(In reply to Arvid Requate from comment #4) > Is this ready for QA? Well yes, except that 35 tests are still failing and two tracebacks/rejects - which might be the reason.
The AD-Connector has been migrated to Python 3. The logfile has been renamed from connector.log to connector-ad.log. The script univention-connector-list-rejected has been renamed to univention-adconnector-list-rejected (an alias/symlink for the old still exists). The Jenkins scripts to join into AD has been migrated to Python 3. univention-system-setup (13.0.3-8) 8108412f0a1b | Bug #52044: Merge branch 'fbest/52044-ad-connector-python3' into 5.0-0 51fe3930c9fe | Revert "Bug #52044: Do not copy broken ad-connector package" univention-system-setup (13.0.2-1) 03e42bbecf05 | Bug #52044: Do not copy broken ad-connector package univention-nagios-ad-connector (9.0.0-1) 8108412f0a1b | Bug #52044: Merge branch 'fbest/52044-ad-connector-python3' into 5.0-0 3990831fc6a2 | Bug #52044: rename univention-connector-list-rejected -> univention-adconnector-list-rejected univention-ad-connector (14.0.3-2) de8c1b50b7d3 | fixup! Bug #52044: flatten _object_mapping 91e1939ecdbb | Bug #52044: fix DN subtree functions c0d591b5aaa9 | Bug #52044: decode value of con_other_attribute a5ca6737e981 | Bug #52044: log mapping only once c76a6f9974b8 | Bug #52044: hide noisy unrelevant debug messages 33e6c63dd91d | Bug #52044: fix sync_to_ucs() not putting AD:mail to UCS:mailPrimaryAddress e5f6cd9e915b | Bug #52044: remove unused con_value_merge_function from mapping e75c51a78cac | fixup! Bug #52044: cleanup: assign variables for mapping 048a3f1f679c | Bug #52044: fix using well-known-sid-object-rename with kerberos ea5839ba75b8 | fixup! Bug #52044: make sure mapping.py exists during join script execution 960ffa7f8cca | Bug #52044: move directory to python 3 package 8108412f0a1b | Bug #52044: Merge branch 'fbest/52044-ad-connector-python3' into 5.0-0 a901dafa7cc8 | Bug #52044: Use ldbsearch for univention-adsearch 2203367ff121 | Bug #52044: Skip pointless entries in group cn mapping_table 7da1e0053379 | Bug #52044: According to AD docs the description attribute is single-valued for SAM managed objects 3d73650d9080 | Bug #52044: Add debug logging for _object_mapping 8bdd86cad02c | Bug #52044: flatten _object_mapping 99d410e25f65 | Bug #52044: Replace Simple_AD_Connection 3990831fc6a2 | Bug #52044: rename univention-connector-list-rejected -> univention-adconnector-list-rejected f27fb48a919c | Bug #52044: rename logfile into /var/log/univention/connector-ad.log 676c65288d7c | Bug #52044: migrate group_members_sync_to_ucs() to Python 3 a823b2791d83 | Bug #52044: ignore not existing objects in disable_user_from_ucs() 7031d8ae7720 | Bug #52044: rename function into __identify_ad_type() 66c78ddcdbca | Bug #52044: flatten resync_rejected() c736b65f28e5 | Bug #52044: migrate set_primary_group_to_ucs_user() to Python 3 e707b1d3cc30 | Bug #52044: add univention.connector.ucs.ad.main() 2a2dd7c11e58 | Bug #52044: Fix mapping_table loop and logging in samaccountname_dn_mapping 34a0c2d46a7f | Bug #52044: Fix value encoding 4c51a9f4cae0 | Bug #52044: migrate samaccountname_dn_mapping() to Python 3 ce77035a6852 | Bug #52044: use poll() implementation of S4-Connector de427922fa81 | Bug #52044: reorder imports 8d0ee0e69d7f | Bug #52044: use error handling of subtree removal from S4-Connector abf437a82ff3 | Bug #52044: make it possible to add servercontrols to modify/add ldap requests 0467d34e096c | Bug #52044: migrate attribute value comparision to Python 3 dfa9d858d95c | Bug #52044: cleanup: assign variables for mapping 72ed8d9a632a | Bug #52044: cleanup: remove unnecessary duplicated "OR" condition 4eda4d07610e | Bug #52044: unify configdb with S4-Connector 09f2bb789eaf | Bug #52044: move_in_ucs: compare DN correctly 2811e6d2407d | Bug #52044: unify decoding of objectGUID 120dfbe77177 | Bug #52044: rename variable into group_member_mapping_cache_ ae9e85843ea1 | Bug #52044: Python 3 migration: decode values f77880ad308c | Bug #52044: outsource identification into identify_udm_object() be7569c4f271 | Bug #52044: make debug level and logfile configurable via command line options 9bfa7600436b | Bug #52044: initial AD search: combine USN filters 31ac859eceed | Bug #52044: format_escaped: raise TypeError for bytes 8d7ae0243fc3 | Bug #52044: remove use of self.encode() e138521dc521 | Bug #52044: remove config encoding and unused functions ae1c47080a67 | Bug #52044: move ldap control into static definition fb5440f22867 | Bug #52044: add docstrings from S4C c31e290260ee | Bug #52044: fix exception handling 3adc085c8a40 | Bug #52044: cleanup: rename variable function name "f" a20b565d466c | Bug #52044: fix univention.debug() messages e7d6fecdd8a1 | Bug #52044: remove unnecessary ucs_no_recode (re-decoding of values) 77dd814b28a6 | Bug #52044: migrate unpickling to Python 3 d9df5dbf5b21 | Bug #52044: use samba version of decode_sid() 1112683e82b6 | Bug #52044: remove obsolete compatible_modlist / encode functions 93ec4bf78a6e | Bug #52044: remove obsolete configsaver bfe3fa2edc83 | Bug #52044: migrate univention-adsearch to Python 3 b4f8d4197085 | Bug #52044: migrate time functions to Python 3 1489120b70fc | Bug #52044: migrate proxyAdresses to Python 3 1920037a6bf9 | Bug #52044: migrate main.py to Python 3 f05fca91378f | Bug #52044: migrate password.py to Python 3 2ade5fef8c71 | Bug #52044: migrate query_config to Python 3 dd864a892177 | Bug #52044: migrate adcache.py to Python 3 e8547b71b6a6 | Bug #52044: copy fix_dn_in_search() from S4-Connector b886e4e5f3f5 | Bug #52044: remove obsolete encode_attr() functions ddb378876c11 | Bug #52044: remove obsolete BINARY_ATTRIBUTES 55e6238d706a | Bug #52044: remove unused mapping functions eebb3ef4131a | Bug #52044: remove unused old_user_dn_mapping() 81d8b15172d3 | Bug #52044: remove unneccesary database commit 5ecd35ce719f | Bug #52044: migrate Listener to Python 3 6ac578b7ef65 | Bug #52044: migrate scripts to Python 3 7488e5528e8b | Bug #52044: baseconfig → configRegistry 6c8eefd1353b | Bug #52044: depend on Python 3 packages 8c31bb536ddc | Bug #52044: migrate mapping.py to Python 3 83dd7d40bbdf | Bug #52044: change hashbangs to Python 3 4afebeb22cac | Bug #52044: make sure mapping.py exists during join script execution 382bd68f5ad3 | fixup! Bug #52044: README.md documentation db7ed2de95f2 | Bug #52044: Merge README into README.md 7654ab972e81 | Bug #52044: README.md documentation ucs-test (10.0.2-2) edcf24621652 | Bug #52044: Skip empty modlist 8108412f0a1b | Bug #52044: Merge branch 'fbest/52044-ad-connector-python3' into 5.0-0 3990831fc6a2 | Bug #52044: rename univention-connector-list-rejected -> univention-adconnector-list-rejected f27fb48a919c | Bug #52044: rename logfile into /var/log/univention/connector-ad.log 70ba9c46d18a | fixup! Bug #52044: fix 169sync_custom_mapping_nonstandard_attribs 474848a788e8 | fixup! Bug #52044: fix 169sync_custom_mapping_nonstandard_attribs 8c01b40d1445 | Bug #52044: execute 502_other_attribute_sync.py with Python 3 42a6fe24c9bf | fixup! Bug #52044: adconnector.py: convert to unicode 91da6ed9f489 | Bug #52044: adconnector.py: convert to unicode f67203f3853f | fixup! Bug #52044: fix 169sync_custom_mapping_nonstandard_attribs 806e0f247a41 | Bug #52044: fix 169sync_custom_mapping_nonstandard_attribs a9e680b45db5 | fixup! Bug #52044: migrate AD-Connector tests to Python 3 2f3320a90c22 | Bug #52044: migrate AD-Connector tests to Python 3 cd158b6010fd | Bug #52044: adjust mapping via localmapping.py changelog-5.0-0.xml e57d5d916759 | Changelog Bug #52044 8beb45b7ff99 | Bug #52044: migrate ad-join.py to Python 3 070f41fc25b5 | Bug #52044: run end section in ad-connector tests
Good job! During code review I spotted Bug #52761 in S4C and fixed it here directly to avoid a regression for ADC. The only other remarkable change I found was a sAMAccountName getting encoded as ASCII instead of UTF-8. 97ca8bfcb9 fixup! Bug #52044: unify decoding of objectGUID 852ff226ae fixup! Bug #52044: Python 3 migration: decode values b965a4c459 fixup! Bug #52044: Python 3 migration: decode values bdc8263aaf fixup! Bug #52044: add univention.connector.ucs.ad.main() Package: univention-ad-connector Version: 14.0.5-1A~5.0.0.202102082109 Branch: ucs_5.0-0
The mapping UCR template has been transferred into a real Python script: univention-ad-connector (14.0.5-5) 51c607f222f6 | Bug #52044: Transfer mapping from UCR template to real Python script changelog-5.0-0.xml 51c607f222f6 | Bug #52044: Transfer mapping from UCR template to real Python script
The "con_other" attributes are not synchronized properly any longer. E.g. a 'telephoneNumber': [b'16466950', b'188866956', b'66961'] from OpenLDAP is handled like this: (INFO ): sync_from_ucs: The following attribute has been changed: otherTelephone (INFO ): sync_from_ucs: Found a corresponding mapping definition: telephoneNumber (INFO ): sync_from_ucs: otherTelephone old_values: set() (INFO ): sync_from_ucs: otherTelephone new_values: {b'66961', b'188866956'} [...] (INFO ): sync_from_ucs: The following attribute has been changed: telephoneNumber (INFO ): sync_from_ucs: Found a corresponding mapping definition: telephoneNumber (INFO ): sync_from_ucs: telephoneNumber old_values: set() (INFO ): sync_from_ucs: telephoneNumber new_values: {b'16466950'} and then otherTelephone is split again into telephoneNumber + otherTelephone and then telephoneNumber is put twice into the modlist: (ALL ): sync_from_ucs: modlist: [(2, 'sn', [b'latxduyt']), (2, 'givenName', [b'pvobduyn']), (2, 'streetAddress', [b'vgvfduzi']), (2, 'homePhone', [b'ddvbr']), (2, 'description', [b'rjqsduyz']), (2, 'l', [b'vjjhduzn']), (2, 'telephoneNumber', [b'66961']), (2, 'otherTelephone', [b'188866956']), (2, 'pager', [b'66980']), (2, 'scriptPath', [b'diusdvaf']), (2, 'displayName', [b'pvobduyn latxduyt']), (2, 'telephoneNumber', [b'16466950']), (2, 'pager', [b'1666985']), (2, 'otherPager', [b'66991']), (2, 'profilePath', [b'aecddvaa']), (2, 'userWorkstations', [b'wjdddvak']), (2, 'postalCode', [b'aissduzt']), (2, 'mobile', [b'aywgzzkoclxpuwvfixenddvbx'])]
No regression, already reproducible in UCS 4.4-7, split off as Bug 52946.
The mapping UCR template files has been replaced with a real python module: univention.connector.ad.mapping. From now on /etc/univention/connector/ad/localmapping.py may be used to adjust the mapping. def mapping_hook(mapping): # add new type mapping['foo'] = univention.connector.property(...) # remove type mapping.pop('dc', None) # add new attribute mapping['user'].attributes['foo'] = univention.connector.attribute(...) return mapping univention-ad-connector (14.0.6-1) ea08bb1230a6 | Bug #52044: cleanup whitespace issues fba206522aaa | Bug #52044: replace UCR filter based mapping with a real python module 43e079e167e2 | Bug #52044: remove mapping.py
please create a "temporary" file /etc/univention/connector/s4/mapping.current (or wherever you like) with the current mapping (during the startup of the connector after the mapping has been processed) Seems that with the standard log level the mapping is not logged in connector-s4.log, but anyway i think an extra file is better and i'm afraid support gets crazy if we longer have such a file.
univention-ad-connector (14.0.7-1) d268742b5fdb | Bug #52044: Log mapping to /var/log/univention/connector-ad-mapping.log
46f2886027 | Document connector-ad-mapping.log
Verified: * The tests look good * Release changelog I've added a sentence about connector-ad-mapping.log to the manual. 46f2886027 | Document connector-ad-mapping.log
03581e9aee95dae5abe18ae9c4a44634cedbdefd Bug #52044: added help to univention-adsearch
UCS 5.0 has been released: https://docs.software-univention.de/release-notes-5.0-0-en.html https://docs.software-univention.de/release-notes-5.0-0-de.html If this error occurs again, please use "Clone This Bug".