Univention Bugzilla – Bug 41544
Removal of user accounts before add/modification to prevent problems with reused values
Last modified: 2016-10-04 13:24:47 CEST
The import script should remove user accounts before adding/modify new/existing objects to prevent problems with reused values: root@master65:/usr/share/ucs-school-import/scripts/test# ../ucs-school-user-import -c import.json -i import.csv --set scheme:rid="<username>|<firstname>|<lastname>|<email>" 2016-06-01 01:25:13 INFO cmdline.main:108 ------ UCS@school import tool configured ------ 2016-06-01 01:25:13 INFO cmdline.main:109 Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaults.json', '/var/lib/ucs-school-import/configs/user_import.json', 'import.json']. 2016-06-01 01:25:13 INFO cmdline.main:110 Using command line arguments: {'input': {'filename': 'import.csv'}, 'scheme': {'rid': '<username>|<firstname>|<lastname>|<email>'}, 'infile': 'import.csv'} 2016-06-01 01:25:13 INFO cmdline.main:111 Configuration is: {u'activate_new_users': {u'default': True}, u'classes': {}, u'csv': {u'header_lines': 1, u'incell-delimiter': {u'default': u','}, u'mapping': {u'Beschreibung': u'description', u'E-Mail': u'email', u'Gruppen': u'school_classes', u'Nach': u'lastname', u'OUs': u'schools', u'Vor': u'firstname'}}, u'dry_run': False, u'factory': u'ucsschool.importer.default_user_import_factory.DefaultUserImportFactory', 'infile': 'import.csv', u'input': {u'filename': 'import.csv', u'type': u'csv'}, u'logfile': u'/var/log/univention/ucsschool-import.log', u'maildomain': None, u'mandatory_attributes': [u'firstname', u'lastname', u'name', u'school'], u'no_delete': False, u'outdated_users': {u'deactivate': False, u'delete': True, u'waiting_period': 0}, u'output': {u'new_user_passwords': u'/var/lib/ucs-school-import/new_user_passwords.csv', u'user_import_summary': u'/var/lib/ucs-school-import/user_import_summary.csv'}, u'password_length': 15, u'scheme': {u'email': u'<firstname>[0].<lastname>[COUNTER2]@<maildomain>', u'rid': '<username>|<firstname>|<lastname>|<email>', u'username': {u'allow_rename': False, u'default': u'<:umlauts><firstname>[0].<lastname>[COUNTER2]'}}, u'school': None, u'sourceUID': u'FirstDB', u'tolerate_errors': 0, u'user_deletion': {u'delete': True, u'expiration': 0}, u'user_role': u'student', u'verbose': True} 2016-06-01 01:25:13 INFO cmdline.do_import:91 ------ Starting mass import... ------ 2016-06-01 01:25:13 WARNING utils.stopped_notifier:168 Stopping univention-directory-notifier 2016-06-01 01:25:18 INFO utils._run:162 Stopping univention-directory-notifier daemon: . ok: down: univention-directory-notifier: 0s done. 2016-06-01 01:25:18 INFO utils.stopped_notifier:180 univention-directory-notifier stopped 2016-06-01 01:25:18 INFO mass_import.import_users:93 ------ Importing users... ------ 2016-06-01 01:25:19 INFO pyhooks_loader.get_plugins:53 Searching for plugins in: /usr/share/ucs-school-import/pyhooks... 2016-06-01 01:25:19 INFO pyhooks_loader.get_plugins:80 Found plugins: {} 2016-06-01 01:25:19 INFO user_import.read_input:79 ------ Starting to read users from input data... ------ 2016-06-01 01:25:19 DEBUG base_reader.next:73 Input 2: ['Musta', '2345', 'Maxi', 'gsmitte', 'gsmitte-1A', 'Mein zweiter Schueler', 'addi@nstx.local'] -> {u'Testmuell': u'2345', u'Gruppen': u'gsmitte-1A', u'Nach': u'Musta', u'Beschreibung': u'Mein zweiter Schueler', u'E-Mail': u'addi@nstx.local', u'Vor': u'Maxi', u'OUs': u'gsmitte'} 2016-06-01 01:25:19 DEBUG base.cache:209 Initializing ('SchoolClass', ('name', 'gsmitte-1A'), ('school', 'gsmitte')) 2016-06-01 01:25:19 DEBUG csv_reader.map:186 ImportStudent(name=None, school=None, dn=None) attributes={'$dn$': None, 'display_name': u'Maxi Musta', 'record_uid': None, 'firstname': u'Maxi', 'lastname': u'Musta', 'type_name': 'Student', 'school': None, 'name': None, 'disabled': None, 'email': u'addi@nstx.local', 'birthday': None, 'type': 'importStudent', 'schools': u'gsmitte', 'password': 'xxxxxxxxxx', 'source_uid': None, 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={u'description': u'Mein zweiter Schueler'} 2016-06-01 01:25:19 INFO user_import.read_input:83 Done reading 1. user: ImportStudent(name=None, school=None, dn=None) 2016-06-01 01:25:19 INFO user_import.read_input:91 ------ Read 1 users from input data. ------ 2016-06-01 01:25:19 INFO user_import.create_and_modify_users:109 ------ Creating / modifying users... ------ 2016-06-01 01:25:19 DEBUG user_import.create_and_modify_users:112 Creating / modifying user ImportStudent(name=None, school=None, dn=None)... 2016-06-01 01:25:19 DEBUG base.get_only_udm_obj:851 Getting ImportStudent UDM object by filter: (&(objectClass=ucsschoolType)(ucsschoolSourceUID=FirstDB)(ucsschoolRecordUID=|Maxi|Musta|addi@nstx.local)) 2016-06-01 01:25:19 INFO user_import.create_and_modify_users:128 Adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid:|Maxi|Musta|addi@nstx.local) attributes={'$dn$': 'uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', 'display_name': 'Maxi Musta', 'record_uid': u'|Maxi|Musta|addi@nstx.local', 'firstname': 'Maxi', 'lastname': 'Musta', 'type_name': 'Student', 'school': 'gsmitte', 'name': 'M.Musta', 'disabled': 'none', 'email': u'addi@nstx.local', 'birthday': None, 'type': 'importStudent', 'schools': ['gsmitte'], 'password': 'xxxxxxxxxx', 'source_uid': u'FirstDB', 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={'overridePWHistory': '1', u'description': u'Mein zweiter Schueler', 'overridePWLength': '1'}... 2016-06-01 01:25:19 DEBUG base.call_hooks:363 /usr/share/ucs-school-import/hooks/user_create_pre.d not found or empty. 2016-06-01 01:25:19 DEBUG base.get_only_udm_obj:851 Getting ImportStudent UDM object by filter: username=M.Musta 2016-06-01 01:25:19 INFO base.create_without_hooks:425 Creating ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) 2016-06-01 01:25:19 DEBUG base.cache:209 Initializing ('School', ('name', 'gsmitte')) 2016-06-01 01:25:19 DEBUG base.get_udm_object:648 Getting School UDM object by dn: ou=gsmitte,dc=nstx,dc=local 2016-06-01 01:25:19 DEBUG base.get_only_udm_obj:851 Getting ImportStudent UDM object by filter: &(!(uid=M.Musta))(mailPrimaryAddress=addi@nstx.local) 2016-06-01 01:25:19 ERROR user_import.create_and_modify_users:174 Entry #0: ValidationError when adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid: |Maxi|Musta|addi@nstx.local): {'email': ['The email address is already taken by another user. Please change the email address.']} Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 136, in create_and_modify_users success = user.create(lo=self.connection) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 417, in create success = self.create_without_hooks(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 618, in create_without_hooks success = super(ImportUser, self).create_without_hooks(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 430, in create_without_hooks raise ValidationError(self.errors.copy()) UserValidationError: ValidationError when adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid: |Maxi|Musta|addi@nstx.local): {'email': ['The email address is already taken by another user. Please change the email address.']} 2016-06-01 01:25:19 WARNING utils.stopped_notifier:190 Starting univention-directory-notifier 2016-06-01 01:25:19 INFO utils._run:162 Starting Univention Directory Notifier daemon. ok: run: univention-directory-notifier: (pid 25455) 0s, normally down done. 2016-06-01 01:25:19 INFO utils.stopped_notifier:197 univention-directory-notifier started 2016-06-01 01:25:19 ERROR cmdline.main:119 More than 0 errors. Exiting. Errors: 2016-06-01 01:25:19 ERROR cmdline.main:121 0: ValidationError when adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid: |Maxi|Musta|addi@nstx.local): {'email': ['The email address is already taken by another user. Please change the email address.']}
I don't think the users should be removed. Otherwise all settings are lost which aren't done via the import script (e.g. that the user is enabled for owncloud, specific UMC service providers, has another additional mail address)!
(In reply to Florian Best from comment #1) > I don't think the users should be removed. Otherwise all settings are lost > which aren't done via the import script (e.g. that the user is enabled for > owncloud, specific UMC service providers, has another additional mail > address)! To avoid misunderstandings: the import has 3 phases/actions: add, modify, delete Currently the "delete" phase is the last action performed by the import. This may be an problem, e.g. if important attributes like mailPrimaryAddress shall be reused by new accounts → "creating new accounts" is currently performed prior to "deleting outdated accounts". The intention of this bug is to reorder the actions to delete → modify (→) add This way, there should be no conflict and the size of LDAP is usually smaller during modification phase → better performace.
Fixed in r71303. (In reply to Sönke Schwardt-Krummrich from comment #2) > the import has 3 phases/actions: add, modify, delete The import is run from mass_import.MassImport.import_users() In short it is: imported_users = user_import.read_input() # CSV to ImportUser objects users_to_delete = detect_users_to_delete() delete_users(users_to_delete) create_and_modify_users(imported_users)
r72521: added test_reuse_email_attribute() to 90_ucsschool/34_import-users_via_cli_v2 that creates a user with a specific email address, and then creates another user with the same email address, letting the import remove the first one.
OK: removal is done prior to modifications OK: sounds reasonable OK: could not find any regressions OK: YAML
UCS@school 4.1 R2 v5 has been released. http://docs.software-univention.de/changelog-ucsschool-4.1R2v5-de.html If this error occurs again, please clone this bug.