Bug 42503 - ucs-school-import: ImportUser.make_email() does not honor config[mandatory_attributes]
ucs-school-import: ImportUser.make_email() does not honor config[mandatory_at...
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: Import scripts
UCS@school 4.1 R2
Other Linux
: P5 normal (vote)
: UCS@school 4.1 R2 vXXX
Assigned To: Daniel Tröder
Florian Best
: interim-3
Depends on:
Blocks: 45550 45553
  Show dependency treegraph
 
Reported: 2016-09-26 13:15 CEST by Daniel Tröder
Modified: 2017-10-17 12:47 CEST (History)
1 user (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 3: Simply Wrong: The implementation doesn't match the docu
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 1: Nuisance – not a big deal but noticeable
User Pain: 0.034
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2016-09-26 13:15:29 CEST
ImportUser.make_email() raises an exception if no email address is either supplied in the input data or can be created from a scheme.

It ignores config[mandatory_attributes]: if there is no "email" in the list, no exception should be raised, but the attribute should be left empty.
Comment 1 Daniel Tröder univentionstaff 2016-09-26 13:18:10 CEST
I set "How will those affected feel about the bug?" to "nuisance", because a simple workaround exists: supply mailPrimaryAddress="" for all users to get empty email addresses.
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2016-09-27 09:15:03 CEST
I will set interim keyword later on.
Comment 3 Daniel Tröder univentionstaff 2016-10-17 10:33:38 CEST
When fixing this bug, please check the ucs-test scripts for then-unneeded handler_set(['mail/hosteddomains=..']).
Comment 4 Daniel Tröder univentionstaff 2016-11-02 17:00:09 CET
After fixing this bug, please revert r74028.
Comment 5 Sönke Schwardt-Krummrich univentionstaff 2016-11-07 12:15:42 CET
(In reply to Daniel Tröder from comment #4)
> After fixing this bug, please revert r74028.

Fixed last commit. So a simple revert will not work.
Comment 6 Daniel Tröder univentionstaff 2016-11-16 13:01:23 CET
r74469: code and advisory
r74471: reverted workarounds in ucs-test-ucsschool
Comment 7 Daniel Tröder univentionstaff 2016-11-16 13:30:52 CET
r74475: post-autopep8 fixes, whitespace-only
Comment 8 Florian Best univentionstaff 2016-11-24 17:52:19 CET
OK: YAML
OK: I could import a user which has no email set
REOPEN: But after the import always this traceback appears:

# cat foo.csv
"Schulen","Benutzertyp","Vorname","Nachname","Klassen","Beschreibung","Telefon","recordUID"
"oldschool","student","Ingward","Heckmann","oldschool-1,oldschool-a","A student.","+99-218-122014","ingward"


# /usr/share/ucs-school-import/scripts/ucs-school-user-import -i foo.csv --sourceUID=foo -u teacher -s oldschool
------ UCS@school import tool starting ------
Reading configuration from '/usr/share/ucs-school-import/configs/global_defaults.json'...
Reading configuration from '/var/lib/ucs-school-import/configs/global.json'...
Reading configuration from '/usr/share/ucs-school-import/configs/user_import_defaults.json'...
Reading configuration from '/var/lib/ucs-school-import/configs/user_import.json'...
2016-11-16 03:16:49 INFO  cmdline.main:111  ------ UCS@school import tool configured ------
2016-11-16 03:16:49 INFO  cmdline.main:112  Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaults.json', '/var/lib/ucs-school-impo
rt/configs/user_import.json'].
2016-11-16 03:16:49 INFO  cmdline.main:113  Using command line arguments: {'input': {'filename': 'foo.csv'}, 'school': 'oldschool', 'user_role': 'teacher', 'sourceUID': 'foo'}
2016-11-16 03:16:49 INFO  cmdline.main:114  Configuration is:
{u'activate_new_users': {u'default': True},
 u'classes': {},
 u'csv': {u'header_lines': 1,
          u'incell-delimiter': {u'default': u','},
          u'mapping': {u'Nachname': u'lastname', u'Vorname': u'firstname'}},
 u'dry_run': False,
 u'factory': u'ucsschool.importer.default_user_import_factory.DefaultUserImportFactory',
 u'input': {u'filename': 'foo.csv', u'type': u'csv'},
 u'logfile': u'/var/log/univention/ucs-school-import.log',
 u'maildomain': None,
 u'mandatory_attributes': [u'firstname', u'lastname', u'name', u'school'],
 u'no_delete': False,
 u'output': {u'new_user_passwords': None,
             u'user_import_summary': u'/var/lib/ucs-school-import/user_import_summary_%Y-%m-%d_%H:%M:%S.csv'},
 u'password_length': 15,
 u'scheme': {u'email': u'<firstname>[0].<lastname>@<maildomain>',
             u'recordUID': u'<username>',
             u'username': {u'allow_rename': False,
                           u'default': u'<:umlauts><firstname>[0].<lastname><:lower>[COUNTER2]'}},
 u'school': 'oldschool',
 u'sourceUID': 'foo',
 u'tolerate_errors': 0,
 u'user_deletion': {u'delete': True, u'expiration': 0},
 u'user_role': 'teacher',
 u'verbose': True}
2016-11-16 03:16:49 INFO  cmdline.do_import:94  ------ Starting mass import... ------
2016-11-16 03:16:49 WARNING utils.stopped_notifier:294  Stopping univention-directory-notifier
2016-11-16 03:16:54 INFO  utils._run:288  Stopping univention-directory-notifier daemon: .
ok: down: univention-directory-notifier: 0s
done.
2016-11-16 03:16:54 INFO  utils.stopped_notifier:306  univention-directory-notifier stopped
2016-11-16 03:16:54 INFO  mass_import.import_users:94  ------ Importing users... ------
2016-11-16 03:16:54 INFO  user_import.read_input:75  ------ Starting to read users from input data... ------
2016-11-16 03:16:54 DEBUG base_reader.next:73  Input 2: ['oldschool', 'student', 'Ingward', 'Heckmann', 'oldschool-1,oldschool-a', 'A student.', '+99-218-122014', 'ingward'] -> {u'Schulen': u'oldschool', u'Benutzertyp': u'student', u'recordUID': u'ingward', u'Nachname': u'
Heckmann', u'Telefon': u'+99-218-122014', u'Beschreibung': u'A student.', u'Vorname': u'Ingward', u'Klassen': u'oldschool-1,oldschool-a'}
2016-11-16 03:16:54 DEBUG csv_reader.map:195  ImportTeacher(name=None, school=None, dn=None) attributes={'$dn$': None, 'display_name': u'Ingward Heckmann', 'record_uid': None, 'firstname': u'Ingward', 'lastname': u'Heckmann', 'type_name': 'Teacher', 'school': None, 'name':
 None, 'disabled': None, 'email': None, 'birthday': None, 'type': 'importTeacher', 'schools': [], 'password': None, 'source_uid': None, 'school_classes': {}, 'objectType': 'users/user'} udm_properties={} action=None
2016-11-16 03:16:54 WARNING import_user.format_from_scheme:681  Created empty 'recordUID' from scheme '<username>' and input data {'$dn$': None, u'firstname': u'Ingward', u'lastname': u'Heckmann', 'disabled': None, 'birthday': None, 'schools': [], 'password': None, 'source
_uid': None, 'name': None, 'school': None, 'display_name': u'Ingward Heckmann', 'record_uid': None, 'type_name': 'Teacher', 'school_classes': {}, 'objectType': 'users/user', 'type': 'importTeacher', 'email': None}. 
2016-11-16 03:16:54 INFO  user_import.read_input:79  Done reading 1. user: ImportTeacher(name=None, school=None, dn=None)
2016-11-16 03:16:54 INFO  user_import.read_input:87  ------ Read 1 users from input data. ------
2016-11-16 03:16:54 INFO  user_import.detect_users_to_delete:210  ------ Detecting which users to delete... ------
2016-11-16 03:16:54 DEBUG user_import.detect_users_to_delete:250  users_to_delete=[]
2016-11-16 03:16:54 INFO  user_import.delete_users:265  ------ Deleting 0 users... ------
2016-11-16 03:16:54 INFO  user_import.delete_users:281  ------ Deleted 0 users. ------
2016-11-16 03:16:54 INFO  user_import.create_and_modify_users:101  ------ Creating / modifying users... ------
2016-11-16 03:16:54 DEBUG user_import.create_and_modify_users:106  Creating / modifying user ImportTeacher(name=None, school=None, dn=None)...
2016-11-16 03:16:54 DEBUG base.get_only_udm_obj:869  Getting ImportTeacher UDM object by filter: (&(objectClass=ucsschoolType)(ucsschoolSourceUID=foo)(ucsschoolRecordUID=))
2016-11-16 03:16:54 WARNING import_user.format_from_scheme:681  Created empty 'recordUID' from scheme '<username>' and input data {'$dn$': None, u'firstname': u'Ingward', u'lastname': u'Heckmann', 'disabled': None, 'birthday': None, 'schools': [], 'password': None, 'source
_uid': 'foo', 'name': None, 'school': None, 'display_name': u'Ingward Heckmann', 'record_uid': u'', 'type_name': 'Teacher', 'school_classes': {}, 'objectType': 'users/user', 'type': 'importTeacher', 'email': None}. 
2016-11-16 03:16:54 INFO  user_import.create_and_modify_users:121  Adding ImportTeacher(name='i.heckmann5', school='oldschool', dn='uid=i.heckmann5,cn=lehrer,cn=users,ou=oldschool,dc=school,dc=local', old_dn=None) (source_uid:foo record_uid:) attributes={'$dn$': 'uid=i.hec
kmann5,cn=lehrer,cn=users,ou=oldschool,dc=school,dc=local', 'display_name': 'Ingward Heckmann', 'record_uid': u'', 'firstname': 'Ingward', 'lastname': 'Heckmann', 'type_name': 'Teacher', 'school': 'oldschool', 'name': 'i.heckmann5', 'disabled': 'none', 'email': None, 'birt
hday': None, 'type': 'importTeacher', 'schools': ['oldschool'], 'password': 'ENQMX9duK0pb1B,', 'source_uid': 'foo', 'school_classes': {}, 'objectType': 'users/user'} udm_properties={'overridePWHistory': '1', 'overridePWLength': '1'}...
2016-11-16 03:16:54 INFO  pyhooks_loader.get_plugins:53  Searching for plugins in: /usr/share/ucs-school-import/pyhooks...
2016-11-16 03:16:54 INFO  pyhooks_loader.get_plugins:61  Found plugins: []
2016-11-16 03:16:54 INFO  import_user.setup_pyhooks:730  Registered hooks: {}.
2016-11-16 03:16:54 DEBUG base.call_hooks:373  /usr/share/ucs-school-import/hooks/user_create_pre.d not found or empty.
2016-11-16 03:16:54 DEBUG base.get_only_udm_obj:869  Getting ImportTeacher UDM object by filter: username=i.heckmann5
2016-11-16 03:16:54 INFO  base.create_without_hooks:435  Creating ImportTeacher(name='i.heckmann5', school='oldschool', dn='uid=i.heckmann5,cn=lehrer,cn=users,ou=oldschool,dc=school,dc=local', old_dn=None)
2016-11-16 03:16:54 DEBUG base.cache:210  Initializing ('School', ('name', 'oldschool'))
2016-11-16 03:16:54 DEBUG base.get_udm_object:666  Getting School UDM object by dn: ou=oldschool,dc=school,dc=local
2016-11-16 03:16:54 DEBUG base.cache:210  Initializing ('Group', ('name', 'Domain Users oldschool'), ('school', 'oldschool'))
2016-11-16 03:16:54 DEBUG base.call_hooks:373  /usr/share/ucs-school-import/hooks/group_create_pre.d not found or empty.
2016-11-16 03:16:54 DEBUG base.get_only_udm_obj:869  Getting Group UDM object by filter: name=Domain Users oldschool
2016-11-16 03:16:54 DEBUG base.cache:210  Initializing ('Group', ('name', 'lehrer-oldschool'), ('school', 'oldschool'))
2016-11-16 03:16:54 DEBUG base.get_only_udm_obj:869  Getting Group UDM object by filter: name=lehrer-oldschool
2016-11-16 03:16:54 DEBUG base.from_dn:852  Looking up School with dn 'ou=oldschool,dc=school,dc=local'
2016-11-16 03:16:54 DEBUG base.get_all:737  Getting all AnyComputer of oldschool with filter 'univentionService=Windows Profile Server'
2016-11-16 03:16:55 INFO  base.create_without_hooks:457  ImportTeacher(name='i.heckmann5', school='oldschool', dn='uid=i.heckmann5,cn=lehrer,cn=users,ou=oldschool,dc=school,dc=local') successfully created
2016-11-16 03:16:55 DEBUG base.get_only_udm_obj:869  Getting ImportTeacher UDM object by filter: (&(objectClass=ucsschoolType)(ucsschoolSourceUID=foo)(ucsschoolRecordUID=))
2016-11-16 03:16:55 WARNING utils.stopped_notifier:316  Starting univention-directory-notifier
2016-11-16 03:16:55 INFO  utils._run:288  Starting Univention Directory Notifier daemon.
ok: run: univention-directory-notifier: (pid 20322) 0s, normally down
done.
2016-11-16 03:16:55 INFO  utils.stopped_notifier:323  univention-directory-notifier started
2016-11-16 03:16:55 ERROR cmdline.main:137  Outer Exception catcher: noObject('No user with source_uid=foo and record_uid= found.',)
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 117, in main
    self.do_import()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 95, in do_import
    importer.mass_import()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 70, in mass_import
    self.import_users()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 99, in import_users
    user_import.create_and_modify_users(imported_users)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 131, in create_and_modify_users
    success = user.create(lo=self.connection)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 150, in create
    return super(ImportUser, self).create(lo, validate)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 429, in create
    self.call_hooks('post', 'create')
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 124, in call_hooks
    user = self.get_by_import_id(self._lo, self.source_uid, self.record_uid)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 166, in get_by_import_id
    raise noObject("No user with source_uid={0} and record_uid={1} found.".format(source_uid, record_uid))
noObject: No user with source_uid=foo and record_uid= found.

→ I am not sure why this happens, everything else seems to work. Please decide what to do with it.
Comment 9 Daniel Tröder univentionstaff 2016-11-25 08:24:41 CET
(In reply to Florian Best from comment #8)
> REOPEN: But after the import always this traceback appears:
> 
> # cat foo.csv
> "Schulen","Benutzertyp","Vorname","Nachname","Klassen","Beschreibung",
> "Telefon","recordUID"
> "oldschool","student","Ingward","Heckmann","oldschool-1,oldschool-a","A
> student.","+99-218-122014","ingward"
> 
> 
> # /usr/share/ucs-school-import/scripts/ucs-school-user-import -i foo.csv
> --sourceUID=foo -u teacher -s oldschool
[..]
>  u'scheme': {u'email': u'<firstname>[0].<lastname>@<maildomain>',
>              u'recordUID': u'<username>',
>              u'username': {u'allow_rename': False,
>                            u'default':
> u'<:umlauts><firstname>[0].<lastname><:lower>[COUNTER2]'}},
[..]
> noObject: No user with source_uid=foo and record_uid= found.
> 
> → I am not sure why this happens, everything else seems to work. Please
> decide what to do with it.

There is already a in separate bug for this problem: Bug #42137.
Comment 10 Florian Best univentionstaff 2016-11-25 11:00:21 CET
Yes!
Comment 11 Florian Best univentionstaff 2016-12-01 18:30:34 CET
This introduces a syntax error in r74475 causing:

2016-12-01 18:13:09 ERROR cmdline.main:137  Outer Exception catcher: TypeError('__init__() takes at least 3 arguments (1 given)',)
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 117, in main
    self.do_import()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 95, in do_import
    importer.mass_import()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 70, in mass_import
    self.import_users()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 99, in import_users
    user_import.create_and_modify_users(imported_users)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 151, in create_and_modify_users
    sys.exc_info()[2]
TypeError: __init__() takes at least 3 arguments (1 given)
Comment 12 Florian Best univentionstaff 2016-12-01 18:31:21 CET
And another one in:
ucs-school-import/modules/ucsschool/importer/reader/csv_reader.py
Comment 13 Daniel Tröder univentionstaff 2016-12-02 10:11:49 CET
r74926: fix syntax error from r74475
Comment 14 Florian Best univentionstaff 2016-12-02 10:29:13 CET
OK: syntax errors
Comment 15 Sönke Schwardt-Krummrich univentionstaff 2016-12-12 13:10:13 CET
UCS@school 4.1 R2 v9 has been released.

http://docs.software-univention.de/changelog-ucsschool-4.1R2v9-de.html