Bug 44993 - (4.2) Feature request: Increase handling of scheme:email options during import of user
(4.2) Feature request: Increase handling of scheme:email options during impor...
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: Import scripts
UCS@school 4.2
Other Linux
: P5 normal (vote)
: UCS@school 4.2 v4
Assigned To: Daniel Tröder
Florian Best
:
Depends on: 42130
Blocks: 45506
  Show dependency treegraph
 
Reported: 2017-07-12 17:26 CEST by Daniel Tröder
Modified: 2017-10-16 21:33 CEST (History)
6 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional): API change, Usability
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2017-07-12 17:26:39 CEST
+++ This bug was initially created as a clone of Bug #42130 +++

According to Feedback on Ticket#2016082421000167 related to the Import-Interface.

Use case from the customer:
Import user with an unique email.

Procedure:
The customer tried two different import configurations for the option scheme:email and didn't get the expected results:
1. "email": "<firstname>.<lastname>[COUNTER2]@<maildomain>" and
2. "email": "<username>@<maildomain>"

The first approach doesn't fail but the String "[COUNTER2]" was added to the email address which wasn't the goal.
The second approach failed with (Traceback 1 attached):
ERROR user_import.create_and_modify_users:189  Entry #2: Email address '@schulen.de' has invalid format. 

Following configuration works:
"email": "<name>@<maildomain>"
Leads to lmustermann20@schulen.de
I think its because the name of attribute from the the underlying ucs-school-lib UCS@school-User is "name".

So, from my perspective a quick workaround would be the documentation of "email": "<name>@<maildomain>" and later on the implementation of both configuration options.

1. "email": "<firstname>.<lastname>[COUNTER2]@<maildomain>"
A generic possibility to use the counters.

2. "email": "<username>@<maildomain>"
Static mapping from the keywords "username" & "uid" to "email": "<name>@<maildomain>"
Comment 1 Daniel Tröder univentionstaff 2017-07-17 10:40:47 CEST
81195: add support for COUNTER variables in email address schemes in import
81196: document COUNTER variables in email address schemes in manual
81197: adapt 215_import-users_illegal_chars_in_username_v2 to API change

81195:
* The bulk of the changes is to generalize the variable naming.
* The StorageBackends are now initialized to support different types of names simultaneously.
* The EmailHandler formats only the local part of an email address. UDM template replacements are still supported for the domain part, but are done in ImportUser.format_from_scheme() before calling EmailHandler.format_name().

81196: In the manual <systemitem class="username"> was used for email addresses instead of <email>, to prevent the generation of clickable email addresses in the output.


Branch: ucs_4.2-0
Scope: ucs-school-4.2

Package: ucs-school-import
Version: 15.0.0-8A~4.2.0.201707171034

Package: ucs-test-ucsschool
Version: 4.0.4-23A~4.2.0.201707171036

Import manual: http://jenkins.knut.univention.de:8080/job/UCSschool%204.2/job/UCSschool%204.2%20Manual/ws/manual/ucsschool-import-handbuch-4.2.pdf (If the email address is clickable, that's a feature of your browser. In a dedicated PDF viewer it's not.)

I'll commit for Bug #42130 (4.1r2) after this one is QA'd. OK?
Comment 2 Florian Best univentionstaff 2017-07-20 15:05:15 CEST
REOPEN: The test looks broken now:

http://jenkins.knut.univention.de:8080/job/UCSschool%204.2/job/UCSschool%204.2%20Singleserver/133/ImportTests=ImportTests,SambaVersion=s4-all-components/testReport/90_ucsschool/34_import-users-legacy/test/

[2017-07-19 05:52:07.041581] 2017-07-19 05:52:07 ERROR user_import.create_and_modify_users:163  Entry #0: No email in u''.
[2017-07-19 05:52:07.041598] Traceback (most recent call last):
[2017-07-19 05:52:07.041609]   File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 111, in create_and_modify_users
[2017-07-19 05:52:07.041619]     user = self.determine_add_modify_action(imported_user)
[2017-07-19 05:52:07.041629]   File "/usr/lib/pymodules/python2.7/ucsschool/importer/legacy/legacy_user_import.py", line 92, in determine_add_modify_action
[2017-07-19 05:52:07.041638]     imported_user.prepare_all(new_user=True)
[2017-07-19 05:52:07.041647]   File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 266, in prepare_all
[2017-07-19 05:52:07.041656]     self.prepare_attributes(new_user)
[2017-07-19 05:52:07.041665]   File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 288, in prepare_attributes
[2017-07-19 05:52:07.041673]     self.make_email()
[2017-07-19 05:52:07.041683]   File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 432, in make_email
[2017-07-19 05:52:07.041691]     self.email = self.unique_email_handler.format_name(self.email)
[2017-07-19 05:52:07.041718]   File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 463, in format_name
[2017-07-19 05:52:07.041729]     local_part_new = super(EmailHandler, self).format_name(local_part, max_length)
[2017-07-19 05:52:07.041739]   File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 366, in format_name
[2017-07-19 05:52:07.041748]     raise FormatError("No {} in {!r}.".format(self.attribute_name, name), name, name)
[2017-07-19 05:52:07.041756] FormatError: No email in u''.
Comment 3 Daniel Tröder univentionstaff 2017-07-21 15:02:38 CEST
The code is OK, the test is broken:

[2017-07-19 05:52:01.563035] {u'classes': {},
[2017-07-19 05:52:01.563054]  u'csv': {u'delimiter': u'\t',
..
[2017-07-19 05:52:01.596961]           u'mapping': {u'0': u'__action',
..
[2017-07-19 05:52:01.597224]                        u'7': u'email',
..
[2017-07-19 05:52:01.597389]  u'scheme': {u'email': u'<email>',

[2017-07-19 05:52:06.948312] 2017-07-19 05:52:06 DEBUG base_reader.next:73  Input 1: ['A', 'bowwncur1q', 'cy95o8u8wz', '76enh204z4', 'e09jbs9ye67', '', '', '', '0', '1', '0'] -> {u'10': u'0', u'1': u'bowwncur1q', u'0': u'A', u'3': u'76enh204z4', u'2': u'cy95o8u8wz', u'5': u'', u'4': u'e09jbs9ye67', u'7': u'', u'6': u'', u'9': u'1', u'8': u'0'}

[2017-07-19 05:52:06.949379] 2017-07-19 05:52:06 DEBUG csv_reader.map:192  LegacyImportStudent(name=u'bowwncur1q', school=u'e09jbs9ye67', dn=u'uid=bowwncur1q,cn=schueler,cn=users,ou=e09jbs9ye67,dc=autotest201,dc=local', old_dn=None) attributes={'record_uid': None, 'disabled': 'none', 'objectType': 'users/user', 'old_user': None, 'display_name': u'76enh204z4 cy95o8u8wz', 'source_uid': None, 'type_name': 'Student', 'in_hook': False, 'udm_properties': {}, 'type': 'legacyImportStudent', 'email': u'', '$dn$': u'uid=bowwncur1q,cn=schueler,cn=users,ou=e09jbs9ye67,dc=autotest201,dc=local', 'firstname': u'76enh204z4', 'lastname': u'cy95o8u8wz', 'entry_count': 0, 'schools': [], 'password': None, 'school': u'e09jbs9ye67', 'name': u'bowwncur1q', 'roles': ['pupil'], 'school_classes': {}, 'input_data': [], 'birthday': None, 'action': u'A'} udm_properties={} action=u'A'

[2017-07-19 05:52:07.018800] 2017-07-19 05:52:07 WARNING import_user.format_from_scheme:720  Created empty 'email' from scheme '<email>' and input data {u'__is_staff': '0', 'record_uid': u'bowwncur1q', u'__action': 'A', 'disabled': 'none', 'objectType': 'users/user', 'old_user': None, 'display_name': u'76enh204z4 cy95o8u8wz', 'source_uid': u'LegacyDB', u'__activate': '1', 'type_name': 'Student', 'in_hook': False, u'__is_teacher': '0', 'udm_properties': {'overridePWHistory': '1', 'overridePWLength': '1'}, 'type': 'legacyImportStudent', u'email': u'', 'username': u'bowwncur1q', '$dn$': u'uid=bowwncur1q,cn=schueler,cn=users,ou=e09jbs9ye67,dc=autotest201,dc=local', u'firstname': u'76enh204z4', u'lastname': u'cy95o8u8wz', 'entry_count': 1L, 'overridePWLength': '1', 'schools': ['e09jbs9ye67'], 'password': 'DWwo4a/B', u'__ignore': '', u'school': 'e09jbs9ye67', 'maildomain': 'autotest201.local', 'overridePWHistory': '1', u'name': u'bowwncur1q', 'roles': ['pupil'], u'school_classes': {}, 'input_data': ['A', 'bowwncur1q', 'cy95o8u8wz', '76enh204z4', 'e09jbs9ye67', '', '', '', '0', '1', '0'], 'birthday': None, 'action': u'A'}. 


So that's correct:

Created empty 'email' from scheme '<email>' and input data {u'email': u''}.

Fixed the test in r81309.
Comment 4 Florian Best univentionstaff 2017-07-21 16:52:09 CEST
I fixed a typo in the manual: r81315
Comment 5 Florian Best univentionstaff 2017-07-27 00:40:38 CEST
The dry-run passed but the real import fails with an exception for me now.

In the logs I especially see the following message, where the filter is obviously wrong. Probably some .lower() was done before evaluating the [COUNTER2]:

2017-07-27 00:36:30 DEBUG base.get_only_udm_obj:869  Getting ImportStudent UDM object by filter: &(!(uid=y.muesterman19))(mailPrimaryAddress=y.muestermann@school.local[counter2])

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 135, in create_and_modify_users
    success = user.create(lo=self.connection)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 165, in create
    return super(ImportUser, self).create(lo, validate)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 427, in create
    success = self.create_without_hooks(lo, validate)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 440, in create_without_hooks
    raise ValidationError(self.errors.copy())
UserValidationError: ValidationError when adding ImportStudent(name='y.muesterman19', school='oldschool', dn='uid=y.muesterman19,cn=schueler,cn=users,ou=oldschool,dc=school,dc=local', old_dn=None) (source_uid:test record_uid: ÝlangMüstèrmánnA student.2): {'email': ['The email address is already taken by another user. Please change the email address.']}
Comment 6 Daniel Tröder univentionstaff 2017-08-03 08:41:49 CEST
I cannot reproduce this.

Is this from 90_ucsschool/219_import_users_umlauts? That runs without errors for me.

ucs-school-import        15.0.0-17
ucs-test-ucsschool       4.0.4-25

When I run real, repeated imports, no errors either:

/usr/share/ucs-school-import/scripts/ucs-school-testuser-import --csvfile /tmp/importtest.csv --teachers 5 --classes 1 --create-email-addresses -v SchuleEinz

/usr/share/ucs-school-import/scripts/ucs-school-user-import --conffile /usr/share/ucs-school-import/configs/ucs-school-testuser-import.json --infile /tmp/importtest.csv -v
Comment 7 Florian Best univentionstaff 2017-09-04 15:31:37 CEST
(In reply to Daniel Tröder from comment #6)
> I cannot reproduce this.
> 
> Is this from 90_ucsschool/219_import_users_umlauts? That runs without errors
> for me.
No, this happened with one of my manual tests. Could you imagine how this can happen, that there is a ldap filter which contains [COUNTER2] ?
I think the order in which the replacement of the template with the value is done in the wrong order. I also see often that <email> is replaced with "" if email is not set by a template yet. We should at least create a new bug for the resolution order. 

As discussed, please implement some ucs-test cases which use explicit the [COUNTER2] / [ALWAYSCOUNTER] scheme for email addresses in the configuration.
Comment 8 Daniel Tröder univentionstaff 2017-09-05 08:55:36 CEST
(In reply to Florian Best from comment #7)
> I think the order in which the replacement of the template with the value is
> done in the wrong order. I also see often that <email> is replaced with ""
> if email is not set by a template yet. We should at least create a new bug
> for the resolution order. 
There is: https://forge.univention.org/bugzilla/show_bug.cgi?id=42137
Comment 9 Daniel Tröder univentionstaff 2017-09-06 10:23:42 CEST
r82703: add test for unique email addresses, fix logging (accidentally jumbled commit order: included tests should have been commited after next commit)
r82705: refactor common unique object code

Package: ucs-test-ucsschool
Version: 4.0.4-26A~4.2.0.201709061019
Comment 10 Florian Best univentionstaff 2017-09-15 12:32:44 CEST
The following tests are failing:
 90_ucsschool.211_import-users_empty_class_name.test
 90_ucsschool.216_import-users_delete_variants.test
 90_ucsschool.207_import-users_school_change.test
 90_ucsschool.219_import_users_umlauts.test
Comment 11 Daniel Tröder univentionstaff 2017-10-11 15:27:58 CEST
I checked the tests:

(In reply to Florian Best from comment #10)
> The following tests are failing:
>  90_ucsschool.211_import-users_empty_class_name.test
Unrelated. Fixed in 8b02bf20 (adapt to changes for Bug #45044).

>  90_ucsschool.216_import-users_delete_variants.test
Unrelated. Fails because of Bug #45467.

>  90_ucsschool.207_import-users_school_change.test
Unrelated. (Fails because the s4connector/samba4 doesn't handle group membership well.)

>  90_ucsschool.219_import_users_umlauts.test
Test is stable for a while now.
Comment 12 Florian Best univentionstaff 2017-10-12 15:22:24 CEST
(In reply to Daniel Tröder from comment #11)
> I checked the tests:
> 
> (In reply to Florian Best from comment #10)
> > The following tests are failing:
> >  90_ucsschool.211_import-users_empty_class_name.test
> Unrelated. Fixed in 8b02bf20 (adapt to changes for Bug #45044).
OK
> >  90_ucsschool.216_import-users_delete_variants.test
> Unrelated. Fails because of Bug #45467.
OK
> >  90_ucsschool.207_import-users_school_change.test
> Unrelated. (Fails because the s4connector/samba4 doesn't handle group
> membership well.)
OK
> >  90_ucsschool.219_import_users_umlauts.test
> Test is stable for a while now.
OK
Comment 13 Sönke Schwardt-Krummrich univentionstaff 2017-10-16 21:32:03 CEST
UCS@school 4.2 v4 has been released.

http://docs.software-univention.de/changelog-ucsschool-4.2v4-de.html

If this error occurs again, please clone this bug.