Bug 41544 - Removal of user accounts before add/modification to prevent problems with reused values
Removal of user accounts before add/modification to prevent problems with reu...
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: Import scripts
UCS@school 4.1 R2
Other Linux
: P5 normal (vote)
: UCS@school 4.1 R2 vXXX
Assigned To: Daniel Tröder
Florian Best
: interim-1
Depends on:
Blocks: 42090
  Show dependency treegraph
 
Reported: 2016-06-13 12:30 CEST by Sönke Schwardt-Krummrich
Modified: 2016-10-04 13:24 CEST (History)
1 user (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.257
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2016-06-13 12:30:39 CEST
The import script should remove user accounts before adding/modify new/existing objects to prevent problems with reused values:

root@master65:/usr/share/ucs-school-import/scripts/test# ../ucs-school-user-import -c import.json -i import.csv --set scheme:rid="<username>|<firstname>|<lastname>|<email>"
2016-06-01 01:25:13 INFO  cmdline.main:108  ------ UCS@school import tool configured ------
2016-06-01 01:25:13 INFO  cmdline.main:109  Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaults.json', '/var/lib/ucs-school-import/configs/user_import.json', 'import.json'].
2016-06-01 01:25:13 INFO  cmdline.main:110  Using command line arguments: {'input': {'filename': 'import.csv'}, 'scheme': {'rid': '<username>|<firstname>|<lastname>|<email>'}, 'infile': 'import.csv'}
2016-06-01 01:25:13 INFO  cmdline.main:111  Configuration is:
{u'activate_new_users': {u'default': True},
 u'classes': {},
 u'csv': {u'header_lines': 1,
          u'incell-delimiter': {u'default': u','},
          u'mapping': {u'Beschreibung': u'description',
                       u'E-Mail': u'email',
                       u'Gruppen': u'school_classes',
                       u'Nach': u'lastname',
                       u'OUs': u'schools',
                       u'Vor': u'firstname'}},
 u'dry_run': False,
 u'factory': u'ucsschool.importer.default_user_import_factory.DefaultUserImportFactory',
 'infile': 'import.csv',
 u'input': {u'filename': 'import.csv', u'type': u'csv'},
 u'logfile': u'/var/log/univention/ucsschool-import.log',
 u'maildomain': None,
 u'mandatory_attributes': [u'firstname', u'lastname', u'name', u'school'],
 u'no_delete': False,
 u'outdated_users': {u'deactivate': False,
                     u'delete': True,
                     u'waiting_period': 0},
 u'output': {u'new_user_passwords': u'/var/lib/ucs-school-import/new_user_passwords.csv',
             u'user_import_summary': u'/var/lib/ucs-school-import/user_import_summary.csv'},
 u'password_length': 15,
 u'scheme': {u'email': u'<firstname>[0].<lastname>[COUNTER2]@<maildomain>',
             u'rid': '<username>|<firstname>|<lastname>|<email>',
             u'username': {u'allow_rename': False,
                           u'default': u'<:umlauts><firstname>[0].<lastname>[COUNTER2]'}},
 u'school': None,
 u'sourceUID': u'FirstDB',
 u'tolerate_errors': 0,
 u'user_deletion': {u'delete': True, u'expiration': 0},
 u'user_role': u'student',
 u'verbose': True}
2016-06-01 01:25:13 INFO  cmdline.do_import:91  ------ Starting mass import... ------
2016-06-01 01:25:13 WARNING utils.stopped_notifier:168  Stopping univention-directory-notifier
2016-06-01 01:25:18 INFO  utils._run:162  Stopping univention-directory-notifier daemon: .
ok: down: univention-directory-notifier: 0s
done.

2016-06-01 01:25:18 INFO  utils.stopped_notifier:180  univention-directory-notifier stopped
2016-06-01 01:25:18 INFO  mass_import.import_users:93  ------ Importing users... ------
2016-06-01 01:25:19 INFO  pyhooks_loader.get_plugins:53  Searching for plugins in: /usr/share/ucs-school-import/pyhooks...
2016-06-01 01:25:19 INFO  pyhooks_loader.get_plugins:80  Found plugins: {}
2016-06-01 01:25:19 INFO  user_import.read_input:79  ------ Starting to read users from input data... ------
2016-06-01 01:25:19 DEBUG base_reader.next:73  Input 2: ['Musta', '2345', 'Maxi', 'gsmitte', 'gsmitte-1A', 'Mein zweiter Schueler', 'addi@nstx.local'] -> {u'Testmuell': u'2345', u'Gruppen': u'gsmitte-1A', u'Nach': u'Musta', u'Beschreibung': u'Mein zweiter Schueler', u'E-Mail': u'addi@nstx.local', u'Vor': u'Maxi', u'OUs': u'gsmitte'}
2016-06-01 01:25:19 DEBUG base.cache:209  Initializing ('SchoolClass', ('name', 'gsmitte-1A'), ('school', 'gsmitte'))
2016-06-01 01:25:19 DEBUG csv_reader.map:186  ImportStudent(name=None, school=None, dn=None) attributes={'$dn$': None, 'display_name': u'Maxi Musta', 'record_uid': None, 'firstname': u'Maxi', 'lastname': u'Musta', 'type_name': 'Student', 'school': None, 'name': None, 'disabled': None, 'email': u'addi@nstx.local', 'birthday': None, 'type': 'importStudent', 'schools': u'gsmitte', 'password': 'xxxxxxxxxx', 'source_uid': None, 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={u'description': u'Mein zweiter Schueler'}
2016-06-01 01:25:19 INFO  user_import.read_input:83  Done reading 1. user: ImportStudent(name=None, school=None, dn=None)
2016-06-01 01:25:19 INFO  user_import.read_input:91  ------ Read 1 users from input data. ------
2016-06-01 01:25:19 INFO  user_import.create_and_modify_users:109  ------ Creating / modifying users... ------
2016-06-01 01:25:19 DEBUG user_import.create_and_modify_users:112  Creating / modifying user ImportStudent(name=None, school=None, dn=None)...
2016-06-01 01:25:19 DEBUG base.get_only_udm_obj:851  Getting ImportStudent UDM object by filter: (&(objectClass=ucsschoolType)(ucsschoolSourceUID=FirstDB)(ucsschoolRecordUID=|Maxi|Musta|addi@nstx.local))
2016-06-01 01:25:19 INFO  user_import.create_and_modify_users:128  Adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid:|Maxi|Musta|addi@nstx.local) attributes={'$dn$': 'uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', 'display_name': 'Maxi Musta', 'record_uid': u'|Maxi|Musta|addi@nstx.local', 'firstname': 'Maxi', 'lastname': 'Musta', 'type_name': 'Student', 'school': 'gsmitte', 'name': 'M.Musta', 'disabled': 'none', 'email': u'addi@nstx.local', 'birthday': None, 'type': 'importStudent', 'schools': ['gsmitte'], 'password': 'xxxxxxxxxx', 'source_uid': u'FirstDB', 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={'overridePWHistory': '1', u'description': u'Mein zweiter Schueler', 'overridePWLength': '1'}...
2016-06-01 01:25:19 DEBUG base.call_hooks:363  /usr/share/ucs-school-import/hooks/user_create_pre.d not found or empty.
2016-06-01 01:25:19 DEBUG base.get_only_udm_obj:851  Getting ImportStudent UDM object by filter: username=M.Musta
2016-06-01 01:25:19 INFO  base.create_without_hooks:425  Creating ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None)
2016-06-01 01:25:19 DEBUG base.cache:209  Initializing ('School', ('name', 'gsmitte'))
2016-06-01 01:25:19 DEBUG base.get_udm_object:648  Getting School UDM object by dn: ou=gsmitte,dc=nstx,dc=local
2016-06-01 01:25:19 DEBUG base.get_only_udm_obj:851  Getting ImportStudent UDM object by filter: &(!(uid=M.Musta))(mailPrimaryAddress=addi@nstx.local)
2016-06-01 01:25:19 ERROR user_import.create_and_modify_users:174  Entry #0: ValidationError when adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid: |Maxi|Musta|addi@nstx.local): {'email': ['The email address is already taken by another user. Please change the email address.']}
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 136, in create_and_modify_users
    success = user.create(lo=self.connection)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 417, in create
    success = self.create_without_hooks(lo, validate)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 618, in create_without_hooks
    success = super(ImportUser, self).create_without_hooks(lo, validate)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 430, in create_without_hooks
    raise ValidationError(self.errors.copy())
UserValidationError: ValidationError when adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid: |Maxi|Musta|addi@nstx.local): {'email': ['The email address is already taken by another user. Please change the email address.']}
2016-06-01 01:25:19 WARNING utils.stopped_notifier:190  Starting univention-directory-notifier
2016-06-01 01:25:19 INFO  utils._run:162  Starting Univention Directory Notifier daemon.
ok: run: univention-directory-notifier: (pid 25455) 0s, normally down
done.
2016-06-01 01:25:19 INFO  utils.stopped_notifier:197  univention-directory-notifier started
2016-06-01 01:25:19 ERROR cmdline.main:119  More than 0 errors. Exiting. Errors:
2016-06-01 01:25:19 ERROR cmdline.main:121  0: ValidationError when adding ImportStudent(name='M.Musta', school='gsmitte', dn='uid=M.Musta,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid: |Maxi|Musta|addi@nstx.local): {'email': ['The email address is already taken by another user. Please change the email address.']}
Comment 1 Florian Best univentionstaff 2016-06-23 10:26:46 CEST
I don't think the users should be removed. Otherwise all settings are lost which aren't done via the import script (e.g. that the user is enabled for owncloud, specific UMC service providers, has another additional mail address)!
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2016-06-29 10:11:45 CEST
(In reply to Florian Best from comment #1)
> I don't think the users should be removed. Otherwise all settings are lost
> which aren't done via the import script (e.g. that the user is enabled for
> owncloud, specific UMC service providers, has another additional mail
> address)!

To avoid misunderstandings:
the import has 3 phases/actions: add, modify, delete

Currently the "delete" phase is the last action performed by the import. This may be an problem, e.g. if important attributes like mailPrimaryAddress shall be reused by new accounts → "creating new accounts" is currently performed prior to "deleting outdated accounts".

The intention of this bug is to reorder the actions to delete → modify (→) add

This way, there should be no conflict and the size of LDAP is usually smaller during modification phase → better performace.
Comment 3 Daniel Tröder univentionstaff 2016-08-01 15:08:24 CEST
Fixed in r71303.


(In reply to Sönke Schwardt-Krummrich from comment #2)
> the import has 3 phases/actions: add, modify, delete
The import is run from mass_import.MassImport.import_users()

In short it is:

imported_users = user_import.read_input()  # CSV to ImportUser objects

users_to_delete = detect_users_to_delete()

delete_users(users_to_delete)

create_and_modify_users(imported_users)
Comment 4 Daniel Tröder univentionstaff 2016-09-12 16:10:27 CEST
r72521: added test_reuse_email_attribute() to 90_ucsschool/34_import-users_via_cli_v2 that creates a user with a specific email address, and then creates another user with the same email address, letting the import remove the first one.
Comment 5 Florian Best univentionstaff 2016-09-22 19:02:25 CEST
OK: removal is done prior to modifications
OK: sounds reasonable
OK: could not find any regressions
OK: YAML
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2016-10-04 13:24:47 CEST
UCS@school 4.1 R2 v5 has been released.

http://docs.software-univention.de/changelog-ucsschool-4.1R2v5-de.html

If this error occurs again, please clone this bug.