Univention Bugzilla – Bug 46317
run checks after pre-hooks, allow to remove email address
Last modified: 2018-06-04 15:34:08 CEST
The ucs-school-user-import runs its consistency/syntax checks before the pre_{create,modify,move} hooks. That creates the problem that data modified in the hooks is not checked. In a customers case the email uniqueness checks triggers because the email address is moved from one user to another in a pre_modify hook. As the email address was available during the check with both users before the pre_modify hook, it was counted twice. Move the checks code after the pre_{create,modify,move} hooks, but before the actual {create,modify,move}().
Now that checks are run after each pre-hook, they can happen to a user multiple times per import-job. For example when first moving a user to a different school and then doing other modifications or when calling modify() from within a hook. To handle that, the dictionary for storing used unique attributes now saves the users DN, and thus allows the same user object to be checked multiple times. This has the nice side effect, that the error message in case of a duplicate now contains the DN of the object that already uses the same value. Furthermore it was discovered, that it wasn't possible to remove/blank an email address. There is now a subtle but important difference between user.email='' and user.email=None: * user.email='' means that there is a CSV column mapped to the email property and the CSV field was empty. That indicates the expressed will of the importer to blank the email address. The scheme mechanism will _not_ be triggered. * user.email=None means that there is _no_ CSV column mapped to the email property. In such a case the scheme mechanism will create an email address, if a scheme is configured. [4.2 731e5455] Bug #46317: store users DN for unique attributes, so the same object can be checked mutliple times, e.g. move(); modify() [4.2 4c43beb2] Bug #46317: run checks after pre_* hooks [4.2 9ac37476] Bug #46317: fix documentation [4.2 857b6b35] Bug #46317: allow to remove an email address [4.2 b2e24346] Bug #46317: fix typo [4.2 22c3d036] Bug #46317: changelog [4.2 ecb0a315] Bug #46317: handle unset properties [4.2 0d460144] Bug #46317: remove email CSV column to enable scheme generation [4.3 abd6c4b9] Bug #46317: store users DN for unique attributes, so the same object can be checked mutliple times, e.g. move(); modify() [4.3 6e8e2cdd] Bug #46317: run checks after pre_* hooks [4.3 7279b449] Bug #46317: fix documentation [4.3 093682a3] Bug #46317: allow to remove an email address [4.3 28804a6d] Bug #46317: fix typo [4.3 fa81f75c] Bug #46317: handle unset properties [4.3 8ae55d34] Bug #46317: remove email CSV column to enable scheme generation [4.3 1bc5006f] Bug #46317: changelog [4.2 8123b923] Bug #46317: advisory ucs-school-import (15.0.3-24) ucs-test-ucsschool (4.0.4-61) ucs-school-import (16.0.1-2) ucs-test-ucsschool (5.0.2-8)
These commits have been accidentally labeled for Bug #45640, but should have been for this bug: [4.2 314d3e69] Bug #45640: rewrite self._unique_ids after school move [4.2 783dd60b] Bug #45640: changelog [4.2 21473581] Bug #45640: create required attributes [4.2 046f6454] Bug #45640: fix typo [4.2 60e1f7a0] Bug #45640: don't fail on slaves [4.2 28bc2beb] Bug #45640: changelog & merge to 4.3 ucs-school-import (15.0.3-25) ucs-test-ucsschool (4.0.4-62)
Code change regarding user.email='' vs. user.email=None has been discussed with professional services and should be no problem for existing customers and is the desired behaviour. OK: code change OK: functional change REOPEN: tests → see below OK: changelog entry REOPEN: advisory → missing for U@S 4.2 OK: changes merged to 4.2-3 and 4.3-0 OK: package built http://jenkins.knut.univention.de:8080/job/UCSschool-4.3/job/Upgrade%20Singleserver/lastCompletedBuild/Config=s4,TestGroup=import1/testReport/90_ucsschool/208_import-users_iso_birthday/test/ [...] [2018-02-28 10:35:21.523622] 2018-02-28 10:35:21 DEBUG user.do_modify:294 Checking group cn=schueler-le2,cn=groups,ou=le2,dc=autotest206,dc=local for adding [2018-02-28 10:35:21.534120] 2018-02-28 10:35:21 WARNING utils.stopped_notifier:344 Starting univention-directory-notifier [2018-02-28 10:35:21.680067] 2018-02-28 10:35:21 INFO utils._run:316 Starting univention-directory-notifier (via systemctl): univention-directory-notifier.service. [2018-02-28 10:35:21.680103] 2018-02-28 10:35:21 INFO utils.stopped_notifier:351 univention-directory-notifier started [2018-02-28 10:35:21.717636] 2018-02-28 10:35:21 ERROR cmdline.main:137 Outer Exception catcher: ldapError('Type or value exists: modify/add: objectClass: value #0 already exists',) [2018-02-28 10:35:21.717668] Traceback (most recent call last): [2018-02-28 10:35:21.717681] File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 117, in main [2018-02-28 10:35:21.717692] self.do_import() [2018-02-28 10:35:21.717703] File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 95, in do_import [2018-02-28 10:35:21.717713] importer.mass_import() [2018-02-28 10:35:21.717724] File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 70, in mass_import [2018-02-28 10:35:21.717743] self.import_users() [2018-02-28 10:35:21.717754] File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 100, in import_users [2018-02-28 10:35:21.717765] user_import.create_and_modify_users(imported_users) # 90% - 100% [2018-02-28 10:35:21.717777] File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 151, in create_and_modify_users [2018-02-28 10:35:21.717788] success = user.modify(lo=self.connection) [2018-02-28 10:35:21.717799] File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 681, in modify [2018-02-28 10:35:21.717853] return super(ImportUser, self).modify(lo, validate, move_if_necessary) [2018-02-28 10:35:21.717870] File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 481, in modify [2018-02-28 10:35:21.717880] success = self.modify_without_hooks(lo, validate, move_if_necessary) [2018-02-28 10:35:21.717891] File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 690, in modify_without_hooks [2018-02-28 10:35:21.717901] return super(ImportUser, self).modify_without_hooks(lo, validate, move_if_necessary) [2018-02-28 10:35:21.717911] File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 504, in modify_without_hooks [2018-02-28 10:35:21.717921] self.do_modify(udm_obj, lo) [2018-02-28 10:35:21.717931] File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/user.py", line 298, in do_modify [2018-02-28 10:35:21.717942] return super(User, self).do_modify(udm_obj, lo) [2018-02-28 10:35:21.717955] File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 530, in do_modify [2018-02-28 10:35:21.717964] udm_obj.modify(ignore_license=1) [2018-02-28 10:35:21.717975] File "/usr/lib/pymodules/python2.7/univention/admin/handlers/users/user.py", line 1657, in modify [2018-02-28 10:35:21.717985] return super(object, self).modify(*args, **kwargs) [2018-02-28 10:35:21.718013] File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 526, in modify [2018-02-28 10:35:21.718027] dn = self._modify(modify_childs, ignore_license=ignore_license, response=response) [2018-02-28 10:35:21.718037] File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 1074, in _modify [2018-02-28 10:35:21.718048] self.lo.modify(self.dn, ml, ignore_license=ignore_license, serverctrls=serverctrls, response=response) [2018-02-28 10:35:21.718058] File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 505, in modify [2018-02-28 10:35:21.718068] raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) [2018-02-28 10:35:21.718078] ldapError: Type or value exists: modify/add: objectClass: value #0 already exists [2018-02-28 10:35:21.895151] 2018-02-28 10:35:21 INFO: run_import:340: Import process exited with exit code 1 [2018-02-28 10:35:21.895188] 2018-02-28 10:35:21 ERROR: run_import:342: As requested raising an exception due to non-zero exit code [2018-02-28 10:35:21.895203] *** Cleanup after exception: <class 'essential.importusers_cli_v2.ImportException'> Non-zero exit code 1 [...]
Bug in UDM. see Bug #45842 comment 70.
> REOPEN: tests → see below → fixed via bug 45842 → OK > REOPEN: advisory → missing for U@S 4.2 → OK
UCS@school 4.2 v9 has been released. https://docs.software-univention.de/changelog-ucsschool-4.2v9-de.html If this error occurs again, please clone this bug.