Univention Bugzilla – Bug 44993
(4.2) Feature request: Increase handling of scheme:email options during import of user
Last modified: 2017-10-16 21:33:57 CEST
+++ This bug was initially created as a clone of Bug #42130 +++ According to Feedback on Ticket#2016082421000167 related to the Import-Interface. Use case from the customer: Import user with an unique email. Procedure: The customer tried two different import configurations for the option scheme:email and didn't get the expected results: 1. "email": "<firstname>.<lastname>[COUNTER2]@<maildomain>" and 2. "email": "<username>@<maildomain>" The first approach doesn't fail but the String "[COUNTER2]" was added to the email address which wasn't the goal. The second approach failed with (Traceback 1 attached): ERROR user_import.create_and_modify_users:189 Entry #2: Email address '@schulen.de' has invalid format. Following configuration works: "email": "<name>@<maildomain>" Leads to lmustermann20@schulen.de I think its because the name of attribute from the the underlying ucs-school-lib UCS@school-User is "name". So, from my perspective a quick workaround would be the documentation of "email": "<name>@<maildomain>" and later on the implementation of both configuration options. 1. "email": "<firstname>.<lastname>[COUNTER2]@<maildomain>" A generic possibility to use the counters. 2. "email": "<username>@<maildomain>" Static mapping from the keywords "username" & "uid" to "email": "<name>@<maildomain>"
81195: add support for COUNTER variables in email address schemes in import 81196: document COUNTER variables in email address schemes in manual 81197: adapt 215_import-users_illegal_chars_in_username_v2 to API change 81195: * The bulk of the changes is to generalize the variable naming. * The StorageBackends are now initialized to support different types of names simultaneously. * The EmailHandler formats only the local part of an email address. UDM template replacements are still supported for the domain part, but are done in ImportUser.format_from_scheme() before calling EmailHandler.format_name(). 81196: In the manual <systemitem class="username"> was used for email addresses instead of <email>, to prevent the generation of clickable email addresses in the output. Branch: ucs_4.2-0 Scope: ucs-school-4.2 Package: ucs-school-import Version: 15.0.0-8A~4.2.0.201707171034 Package: ucs-test-ucsschool Version: 4.0.4-23A~4.2.0.201707171036 Import manual: http://jenkins.knut.univention.de:8080/job/UCSschool%204.2/job/UCSschool%204.2%20Manual/ws/manual/ucsschool-import-handbuch-4.2.pdf (If the email address is clickable, that's a feature of your browser. In a dedicated PDF viewer it's not.) I'll commit for Bug #42130 (4.1r2) after this one is QA'd. OK?
REOPEN: The test looks broken now: http://jenkins.knut.univention.de:8080/job/UCSschool%204.2/job/UCSschool%204.2%20Singleserver/133/ImportTests=ImportTests,SambaVersion=s4-all-components/testReport/90_ucsschool/34_import-users-legacy/test/ [2017-07-19 05:52:07.041581] 2017-07-19 05:52:07 ERROR user_import.create_and_modify_users:163 Entry #0: No email in u''. [2017-07-19 05:52:07.041598] Traceback (most recent call last): [2017-07-19 05:52:07.041609] File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 111, in create_and_modify_users [2017-07-19 05:52:07.041619] user = self.determine_add_modify_action(imported_user) [2017-07-19 05:52:07.041629] File "/usr/lib/pymodules/python2.7/ucsschool/importer/legacy/legacy_user_import.py", line 92, in determine_add_modify_action [2017-07-19 05:52:07.041638] imported_user.prepare_all(new_user=True) [2017-07-19 05:52:07.041647] File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 266, in prepare_all [2017-07-19 05:52:07.041656] self.prepare_attributes(new_user) [2017-07-19 05:52:07.041665] File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 288, in prepare_attributes [2017-07-19 05:52:07.041673] self.make_email() [2017-07-19 05:52:07.041683] File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 432, in make_email [2017-07-19 05:52:07.041691] self.email = self.unique_email_handler.format_name(self.email) [2017-07-19 05:52:07.041718] File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 463, in format_name [2017-07-19 05:52:07.041729] local_part_new = super(EmailHandler, self).format_name(local_part, max_length) [2017-07-19 05:52:07.041739] File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 366, in format_name [2017-07-19 05:52:07.041748] raise FormatError("No {} in {!r}.".format(self.attribute_name, name), name, name) [2017-07-19 05:52:07.041756] FormatError: No email in u''.
The code is OK, the test is broken: [2017-07-19 05:52:01.563035] {u'classes': {}, [2017-07-19 05:52:01.563054] u'csv': {u'delimiter': u'\t', .. [2017-07-19 05:52:01.596961] u'mapping': {u'0': u'__action', .. [2017-07-19 05:52:01.597224] u'7': u'email', .. [2017-07-19 05:52:01.597389] u'scheme': {u'email': u'<email>', [2017-07-19 05:52:06.948312] 2017-07-19 05:52:06 DEBUG base_reader.next:73 Input 1: ['A', 'bowwncur1q', 'cy95o8u8wz', '76enh204z4', 'e09jbs9ye67', '', '', '', '0', '1', '0'] -> {u'10': u'0', u'1': u'bowwncur1q', u'0': u'A', u'3': u'76enh204z4', u'2': u'cy95o8u8wz', u'5': u'', u'4': u'e09jbs9ye67', u'7': u'', u'6': u'', u'9': u'1', u'8': u'0'} [2017-07-19 05:52:06.949379] 2017-07-19 05:52:06 DEBUG csv_reader.map:192 LegacyImportStudent(name=u'bowwncur1q', school=u'e09jbs9ye67', dn=u'uid=bowwncur1q,cn=schueler,cn=users,ou=e09jbs9ye67,dc=autotest201,dc=local', old_dn=None) attributes={'record_uid': None, 'disabled': 'none', 'objectType': 'users/user', 'old_user': None, 'display_name': u'76enh204z4 cy95o8u8wz', 'source_uid': None, 'type_name': 'Student', 'in_hook': False, 'udm_properties': {}, 'type': 'legacyImportStudent', 'email': u'', '$dn$': u'uid=bowwncur1q,cn=schueler,cn=users,ou=e09jbs9ye67,dc=autotest201,dc=local', 'firstname': u'76enh204z4', 'lastname': u'cy95o8u8wz', 'entry_count': 0, 'schools': [], 'password': None, 'school': u'e09jbs9ye67', 'name': u'bowwncur1q', 'roles': ['pupil'], 'school_classes': {}, 'input_data': [], 'birthday': None, 'action': u'A'} udm_properties={} action=u'A' [2017-07-19 05:52:07.018800] 2017-07-19 05:52:07 WARNING import_user.format_from_scheme:720 Created empty 'email' from scheme '<email>' and input data {u'__is_staff': '0', 'record_uid': u'bowwncur1q', u'__action': 'A', 'disabled': 'none', 'objectType': 'users/user', 'old_user': None, 'display_name': u'76enh204z4 cy95o8u8wz', 'source_uid': u'LegacyDB', u'__activate': '1', 'type_name': 'Student', 'in_hook': False, u'__is_teacher': '0', 'udm_properties': {'overridePWHistory': '1', 'overridePWLength': '1'}, 'type': 'legacyImportStudent', u'email': u'', 'username': u'bowwncur1q', '$dn$': u'uid=bowwncur1q,cn=schueler,cn=users,ou=e09jbs9ye67,dc=autotest201,dc=local', u'firstname': u'76enh204z4', u'lastname': u'cy95o8u8wz', 'entry_count': 1L, 'overridePWLength': '1', 'schools': ['e09jbs9ye67'], 'password': 'DWwo4a/B', u'__ignore': '', u'school': 'e09jbs9ye67', 'maildomain': 'autotest201.local', 'overridePWHistory': '1', u'name': u'bowwncur1q', 'roles': ['pupil'], u'school_classes': {}, 'input_data': ['A', 'bowwncur1q', 'cy95o8u8wz', '76enh204z4', 'e09jbs9ye67', '', '', '', '0', '1', '0'], 'birthday': None, 'action': u'A'}. So that's correct: Created empty 'email' from scheme '<email>' and input data {u'email': u''}. Fixed the test in r81309.
I fixed a typo in the manual: r81315
The dry-run passed but the real import fails with an exception for me now. In the logs I especially see the following message, where the filter is obviously wrong. Probably some .lower() was done before evaluating the [COUNTER2]: 2017-07-27 00:36:30 DEBUG base.get_only_udm_obj:869 Getting ImportStudent UDM object by filter: &(!(uid=y.muesterman19))(mailPrimaryAddress=y.muestermann@school.local[counter2]) Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 135, in create_and_modify_users success = user.create(lo=self.connection) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 165, in create return super(ImportUser, self).create(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 427, in create success = self.create_without_hooks(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 440, in create_without_hooks raise ValidationError(self.errors.copy()) UserValidationError: ValidationError when adding ImportStudent(name='y.muesterman19', school='oldschool', dn='uid=y.muesterman19,cn=schueler,cn=users,ou=oldschool,dc=school,dc=local', old_dn=None) (source_uid:test record_uid: ÝlangMüstèrmánnA student.2): {'email': ['The email address is already taken by another user. Please change the email address.']}
I cannot reproduce this. Is this from 90_ucsschool/219_import_users_umlauts? That runs without errors for me. ucs-school-import 15.0.0-17 ucs-test-ucsschool 4.0.4-25 When I run real, repeated imports, no errors either: /usr/share/ucs-school-import/scripts/ucs-school-testuser-import --csvfile /tmp/importtest.csv --teachers 5 --classes 1 --create-email-addresses -v SchuleEinz /usr/share/ucs-school-import/scripts/ucs-school-user-import --conffile /usr/share/ucs-school-import/configs/ucs-school-testuser-import.json --infile /tmp/importtest.csv -v
(In reply to Daniel Tröder from comment #6) > I cannot reproduce this. > > Is this from 90_ucsschool/219_import_users_umlauts? That runs without errors > for me. No, this happened with one of my manual tests. Could you imagine how this can happen, that there is a ldap filter which contains [COUNTER2] ? I think the order in which the replacement of the template with the value is done in the wrong order. I also see often that <email> is replaced with "" if email is not set by a template yet. We should at least create a new bug for the resolution order. As discussed, please implement some ucs-test cases which use explicit the [COUNTER2] / [ALWAYSCOUNTER] scheme for email addresses in the configuration.
(In reply to Florian Best from comment #7) > I think the order in which the replacement of the template with the value is > done in the wrong order. I also see often that <email> is replaced with "" > if email is not set by a template yet. We should at least create a new bug > for the resolution order. There is: https://forge.univention.org/bugzilla/show_bug.cgi?id=42137
r82703: add test for unique email addresses, fix logging (accidentally jumbled commit order: included tests should have been commited after next commit) r82705: refactor common unique object code Package: ucs-test-ucsschool Version: 4.0.4-26A~4.2.0.201709061019
The following tests are failing: 90_ucsschool.211_import-users_empty_class_name.test 90_ucsschool.216_import-users_delete_variants.test 90_ucsschool.207_import-users_school_change.test 90_ucsschool.219_import_users_umlauts.test
I checked the tests: (In reply to Florian Best from comment #10) > The following tests are failing: > 90_ucsschool.211_import-users_empty_class_name.test Unrelated. Fixed in 8b02bf20 (adapt to changes for Bug #45044). > 90_ucsschool.216_import-users_delete_variants.test Unrelated. Fails because of Bug #45467. > 90_ucsschool.207_import-users_school_change.test Unrelated. (Fails because the s4connector/samba4 doesn't handle group membership well.) > 90_ucsschool.219_import_users_umlauts.test Test is stable for a while now.
(In reply to Daniel Tröder from comment #11) > I checked the tests: > > (In reply to Florian Best from comment #10) > > The following tests are failing: > > 90_ucsschool.211_import-users_empty_class_name.test > Unrelated. Fixed in 8b02bf20 (adapt to changes for Bug #45044). OK > > 90_ucsschool.216_import-users_delete_variants.test > Unrelated. Fails because of Bug #45467. OK > > 90_ucsschool.207_import-users_school_change.test > Unrelated. (Fails because the s4connector/samba4 doesn't handle group > membership well.) OK > > 90_ucsschool.219_import_users_umlauts.test > Test is stable for a while now. OK
UCS@school 4.2 v4 has been released. http://docs.software-univention.de/changelog-ucsschool-4.2v4-de.html If this error occurs again, please clone this bug.