Univention Bugzilla – Bug 41242
replace user import part from legacy import-script with new import-tool
Last modified: 2016-09-30 12:11:59 CEST
As discussed with Daniel and Stefan, it should be no problem if the replacement outputs different messages resp. logs its messages into a logfile (instead of stdout). In the release notes (not only the changelog.xml) it has to be mentioned explicitly.
r69732 creates a drop-in replacement for the user part of the legacy import script. 90_ucsschool/essential/importusers.py got a tiny modification so the import script can adapt to the ucs-tests broken csv generation. ucs-test/90_ucsschool/34_import-users_via_cli runs the user generation part successfully, but there seems to be a problem with "hooks" - I'll look into it tomorrow: Traceback (most recent call last): File "34_import-users_via_cli", line 14, in <module> import_users_basics(use_cli_api=True, use_python_api=False) File "/usr/share/ucs-test/90_ucsschool/essential/importusers.py", line 645, in import_users_basics create_and_verify_users(use_cli_api, use_python_api, school_name, 3, 3, 3, 3) File "/usr/share/ucs-test/90_ucsschool/essential/importusers.py", line 570, in create_and_verify_users import_file.run_import(user_import) File "/usr/share/ucs-test/90_ucsschool/essential/importusers.py", line 303, in run_import raise UserHookResult() essential.importusers.UserHookResult
There is currently no migration from existing Users without RID+SID to the new import tool. I see two options: * migration tool, must be run once on existing user base * automatic migration during import, means doubling all searches
(In reply to Daniel Tröder from comment #2) > ucs-test/90_ucsschool/34_import-users_via_cli runs the user generation part > successfully, but there seems to be a problem with "hooks" - I'll look into > it tomorrow: The ImportUser classes do indeed not support hooks (yet). They have to overwrite build_hook_line(). On it.
With r69763 90_ucsschool/34_import-users_via_cli seems to work (running without error since 30min...). The question (comment 3) is still open: what about migrating?
(In reply to Daniel Tröder from comment #3) > There is currently no migration from existing Users without RID+SID to the > new import tool. > > I see two options: > * migration tool, must be run once on existing user base > * automatic migration during import, means doubling all searches No automatic migration. The required steps have to be documented and depend mostly on the import data.
The drop-in replacement script just passed 90_ucsschool/34_import-users_via_cli. Not benchmarking, just for orientation: real 47m58.965s user 13m0.488s sys 1m32.600s
r69870: added logfile output to: DEBUG: /var/log/univention/ucsschool-import.log >=INFO: /var/log/univention/ucsschool-import.info
(In reply to Daniel Tröder from comment #3) > There is currently no migration from existing Users without RID+SID to the > new import tool. s/RID/RUID/ s/SID/SUID/ RID/SID are abbrevs from Active Directory
1) → please revert the changeset 69740 in ucs-test-ucsschool 1b) self.args.settings["csv"] = {"mapping": {"11": "password"}} → This should be default in legacy configuration file. It should not depend on the argument "ucs-test" since the old import_user scripts supports this option too. 2) # ls /var/log/univention/ucs*import.log /var/log/univention/ucs-school-import.log /var/log/univention/ucsschool-import.log → ucsschool-import.log has been created from the new import_user script → please use ucs-school-import.log 3) s/ucs@school/UCS@school/g → incorrect wording in file headers, log entries, ... 4) "More than 0 errors"? 2016-05-31 15:56:33 ERROR cmdline.main:115 More than 0 errors. Exiting. Errors: 2016-05-31 15:56:33 ERROR cmdline.main:117 0: ValidationError when adding LegacyImportStudent(name=u'banton1', school='gsnord', dn=None) (source_uid:LegacyDB record_uid: bAnton1): {'school': ['The school "gsnord" does not exist. Please choose an existing one or create it.']} [...] 2016-05-31 16:15:19 ERROR cmdline.main:115 More than 0 errors. Exiting. Errors: 2016-05-31 16:15:19 ERROR cmdline.main:117 1: User LegacyImportStudent(name=u'bAnton1', school=u'gsnord', dn=None) (source_uid:LegacyDB record_uid: bAnton1) exist, but input demands 'A'. → The message sounds a bit unusual → The legacy-import should not stop, if it is unable to import a single user but continue with the next import line → "tolerate_errors": int [1]: number of non-fatal UcsSchoolImportErrors to tolerate before aborting Was anything of the above a non-fatal error? → please change the default to "unlimited" for legacy user import 5) Traceback in import script if username in CSV file contains uppercase letters I tried to modify my freshly created user and got a traceback. "bAnton1" is the username of the CSV file - in the LDAP the username is "banton1" Somewhere during the import, the username is converted to lowercase letters. → according to syntax.py valid usernames are defined by this: re.compile('(?u)(^\w[\w-.]*\w$)|\w*$') → UDM is currently unable to handle different cases correctly, so we should use the username from LDAP if possible during modification, so UDM will not crash → other ideas are welcome! 2016-05-31 16:34:37 INFO utils.stopped_notifier:197 univention-directory-notifier started 2016-05-31 16:34:37 ERROR cmdline.main:124 Outer Exception catcher: uidAlreadyUsed(": u'bAnton1' 'banton1'",) Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 112, in main self.do_import() File "/usr/lib/pymodules/python2.7/ucsschool/importer/legacy/legacy_user_import_cmdline.py", line 60, in do_import importer.import_users() File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 93, in import_users user_import.create_and_modify_users(imported_users) File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 142, in create_and_modify_users success = user.modify(lo=self.connection) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 461, in modify success = self.modify_without_hooks(lo, validate, move_if_necessary) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 615, in modify_without_hooks success = super(ImportUser, self).modify_without_hooks(lo, validate, move_if_necessary) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 484, in modify_without_hooks self.do_modify(udm_obj, lo) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/user.py", line 281, in do_modify return super(User, self).do_modify(udm_obj, lo) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 510, in do_modify udm_obj.modify(ignore_license=1) File "/usr/lib/pymodules/python2.7/univention/admin/handlers/users/user.py", line 1564, in modify return super(object, self).modify(*args, **kwargs) File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 361, in modify return self._modify(modify_childs,ignore_license=ignore_license) File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 751, in _modify self._ldap_pre_modify() File "/usr/lib/pymodules/python2.7/univention/admin/handlers/users/user.py", line 1927, in _ldap_pre_modify raise univention.admin.uexceptions.uidAlreadyUsed, ': %r %r' % (username, oldusername) uidAlreadyUsed: : u'bAnton1' 'banton1' 6) /var/lib/ucs-school-import/user_import_summary.csv and /var/lib/ucs-school-import/new_user_passwords.csv are always overwritten during each run of import_user. → why not adding a timestamp to the filename? 7) /var/lib/ucs-school-import/new_user_passwords.csv → we should not store password by default in cleartext on the harddisk → please disable this by default QA still in progress
(In reply to Sönke Schwardt-Krummrich from comment #10) > 1) → please revert the changeset 69740 in ucs-test-ucsschool Done in r70072. > 1b) self.args.settings["csv"] = {"mapping": {"11": "password"}} > 2) # ls /var/log/univention/ucs*import.log > → please use ucs-school-import.log > 3) s/ucs@school/UCS@school/g > → incorrect wording in file headers, log entries, ... > 4) "More than 0 errors"? > → please change the default to "unlimited" for legacy user import Done in r70067. > 5) Traceback in import script if username in CSV file contains uppercase > letters > I tried to modify my freshly created user and got a traceback. > "bAnton1" is the username of the CSV file - in the LDAP the username is > "banton1" > Somewhere during the import, the username is converted to lowercase letters. The legacy import does lower() on usernames, and the legacy version of the new import too. But there was a bug preventing that from happening. Fixed in r70067. The non-legacy version of the new import script does not lowercase usernames from input though. > 6) /var/lib/ucs-school-import/user_import_summary.csv and > /var/lib/ucs-school-import/new_user_passwords.csv are always overwritten > during each run of import_user. > → why not adding a timestamp to the filename? > > 7) /var/lib/ucs-school-import/new_user_passwords.csv > → we should not store password by default in cleartext on the harddisk > → please disable this by default Done in r70071
8) Add column 11 with "password" to configuration → The code should be ok with it, if a column is empty/not every line contains as much columns as configured. In the following test, I added "password" as column 11 but gave a CSV file with only 10 columns. 2016-06-01 00:03:10 INFO cmdline.main:106 ------ ucs@school import tool configured ------ 2016-06-01 00:03:10 INFO cmdline.main:107 Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaults.json', '/var/lib/ucs-school-import/configs/user_import.json', '/usr/share/ucs-school-import/configs/user_import_legacy_defaults.json', '/var/lib/ucs-school-import/configs/user_import_legacy.json']. 2016-06-01 00:03:10 INFO cmdline.main:108 Using command line arguments: {'input': {'filename': 'gsnord6.txt'}, 'output': {'passwords': None}} 2016-06-01 00:03:10 INFO cmdline.main:109 Configuration is: {u'activate_new_users': {u'default': True}, u'classes': {}, u'csv': {u'header_lines': 0, u'incell-delimiter': {u'default': u','}, u'mapping': {u'0': u'__action', u'1': u'name', u'10': u'__is_staff', u'11': u'password', u'2': u'lastname', u'3': u'firstname', u'4': u'school', u'5': u'school_classes', u'6': u'__ignore', u'7': u'email', u'8': u'__is_teacher', u'9': u'__activate'}}, u'dry_run': False, u'factory': u'ucsschool.importer.legacy.legacy_csv_user_import_factory.LegacyCsvUserImportFactory', u'input': {u'filename': 'gsnord6.txt', u'type': u'csv'}, u'logfile': u'/var/log/univention/ucsschool-import.log', u'maildomain': None, u'mandatory_attributes': [u'firstname', u'lastname', u'name', u'school'], u'no_delete': False, u'outdated_users': {u'deactivate': False, u'delete': False, u'waiting_period': 45}, u'output': {u'new_user_passwords': u'/var/lib/ucs-school-import/new_user_passwords.csv', 'passwords': None, u'user_import_summary': u'/var/lib/ucs-school-import/user_import_summary.csv'}, u'password_length': 8, u'scheme': {u'email': u'<email>', u'rid': u'<name>', u'username': {u'allow_rename': False, u'default': u'<name>[COUNTER2]'}}, u'school': None, u'sourceUID': u'LegacyDB', u'tolerate_errors': 9999999, u'user_deletion': {u'delete': True, u'expiration': 0}, u'user_role': None, u'verbose': True} [...] 2016-06-01 00:03:16 INFO utils.stopped_notifier:180 univention-directory-notifier stopped 2016-06-01 00:03:16 INFO mass_import.import_users:90 ------ Importing users... ------ 2016-06-01 00:03:16 INFO pyhooks_loader.get_plugins:53 Searching for plugins in: /usr/share/ucs-school-import/pyhooks... 2016-06-01 00:03:16 INFO pyhooks_loader.get_plugins:80 Found plugins: {} 2016-06-01 00:03:16 INFO user_import.read_input:79 ------ Starting to read users from input data... ------ 2016-06-01 00:03:16 DEBUG base_reader.next:73 Input 1: ['A', 'banton8', 'Wagner', 'Anton', 'gsnord', 'gsnord-4r', '', 'bAnton8@nstx.local', '0', '1', '0'] -> {u'10': u'0', u'1': u'banton8', u'0': u'A', u'3': u'Anton', u'2': u'Wagner', u'5': u'gsnord-4r', u'4': u'gsnord', u'7': u'bAnton8@nstx.local', u'6': u'', u'9': u'1', u'8': u'0'} 2016-06-01 00:03:16 WARNING utils.stopped_notifier:190 Starting univention-directory-notifier 2016-06-01 00:03:16 INFO utils._run:162 Starting Univention Directory Notifier daemon. ok: run: univention-directory-notifier: (pid 20028) 0s, normally down done. 2016-06-01 00:03:16 INFO utils.stopped_notifier:197 univention-directory-notifier started 2016-06-01 00:03:16 ERROR cmdline.main:124 Outer Exception catcher: KeyError(u'11',) Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 112, in main self.do_import() File "/usr/lib/pymodules/python2.7/ucsschool/importer/legacy/legacy_user_import_cmdline.py", line 60, in do_import importer.import_users() File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 92, in import_users imported_users = user_import.read_input() File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 82, in read_input import_user = self.reader.next() File "/usr/lib/pymodules/python2.7/ucsschool/importer/reader/base_reader.py", line 75, in next cur_import_user = self.map(input_dict, cur_user_roles) File "/usr/lib/pymodules/python2.7/ucsschool/importer/reader/csv_reader.py", line 161, in map if self.handle_input(k, v, input_data[k], import_user): KeyError: u'11' 9) Deleting non-existing users is currently no error I tried to delete banton9 twice in 2 seperate attempts. There was an error message, but no error entry within the statistic part of the log: 2016-06-01 00:15:47 ERROR legacy_user_import.detect_users_to_delete:54 No user with source_uid=LegacyDB and record_uid=banton9 found. 2016-06-01 00:15:47 INFO user_import.delete_users:257 ------ Deleting users... ------ 2016-06-01 00:15:47 INFO user_import.delete_users:265 Deleting 0 users... 2016-06-01 00:15:47 INFO user_import.delete_users:284 ------ Deleted 0 users. ------ 2016-06-01 00:15:47 INFO user_import.log_stats:327 ------ User import statistics ------ 2016-06-01 00:15:47 INFO user_import.log_stats:328 Read users from input data: 5 2016-06-01 00:15:47 INFO user_import.log_stats:335 Created LegacyImportStudent: 0 2016-06-01 00:15:47 INFO user_import.log_stats:338 Modified LegacyImportStudent: 3 2016-06-01 00:15:47 INFO user_import.log_stats:340 [u'banton5', u'banton6', u'banton7'] 2016-06-01 00:15:47 INFO user_import.log_stats:341 Deleted LegacyImportStudent: 0 2016-06-01 00:15:47 INFO user_import.log_stats:344 Errors: 1 2016-06-01 00:15:47 INFO user_import.log_stats:346 Entry #: Error description 2016-06-01 00:15:47 INFO user_import.log_stats:349 1: banton8: User LegacyImportStudent(name=u'banton8', school=u'gsnord', dn=None) (source_uid:LegacyDB record_uid: banton8) exist, but input demands 'A'. 10) exit codes The process should exit with a non-zero exitcode if at least one error occured. Not well thought out idea: Exit code 1: fatal error Exit code 2: at least one non-fatal error
(In reply to Sönke Schwardt-Krummrich from comment #12) > 8) Add column 11 with "password" to configuration > → The code should be ok with it, if a column is empty/not every line > contains as much columns as configured. Done in r70067 (for 1b above) > 9) Deleting non-existing users is currently no error > I tried to delete banton9 twice in 2 seperate attempts. There was an error > message, but no error entry within the statistic part of the log: The legacy script does not produce an error for deleting non-existing users and exists with 0. That behavior was copied. Should it be different for the non-legacy user import? > 10) exit codes > The process should exit with a non-zero exitcode if at least one error > occured. > Not well thought out idea: > Exit code 1: fatal error > Exit code 2: at least one non-fatal error r70084: exit code is 0 if all went well, 1 if a fatal error occured, 2 if at least one non-fatal error occured Regarding 2) from above: A TimedRotatingFileHandler is used for logging. It will rename once per day the logfile (e.g. ucs-school-import.log.2016-06-08) and keep 10 copies.
11) Moving users to another school does not work A test user was located in "gsnord" and I did a s/gsnord/gsmitte/g on the import line and tried to import it: 2016-06-01 01:28:04 INFO user_import.create_and_modify_users:128 Modifying LegacyImportStudent(name=u'banton11', school='gsmitte', dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') (source_uid:LegacyDB record_uid:banton11) attributes={'$dn$': 'uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local', 'display_name': 'Antonia Richterin', 'record_uid': u'banton11', 'firstname': 'Antonia', 'lastname': 'Richterin', 'type_name': 'Student', 'school': 'gsmitte', 'name': u'banton11', 'disabled': 'none', 'email': u'bAnton11@nstx.local', 'birthday': None, 'type': 'legacyImportStudent', 'schools': ['gsmitte'], 'password': 'xxxxxxxxxx', 'source_uid': u'LegacyDB', 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={}... 2016-06-01 01:28:04 DEBUG base.call_hooks:353 /usr/share/ucs-school-import/hooks/user_modify_pre.d not found or empty. 2016-06-01 01:28:04 INFO base.modify_without_hooks:467 Modifying LegacyImportStudent(name=u'banton11', school='gsmitte', dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') 2016-06-01 01:28:04 DEBUG base.cache:209 Initializing ('School', ('name', 'gsmitte')) 2016-06-01 01:28:04 DEBUG base.get_udm_object:636 Getting School UDM object by dn: ou=gsmitte,dc=nstx,dc=local 2016-06-01 01:28:04 DEBUG base.get_only_udm_obj:838 Getting LegacyImportStudent UDM object by filter: &(!(uid=banton11))(mailPrimaryAddress=bAnton11@nstx.local) 2016-06-01 01:28:04 ERROR user_import.create_and_modify_users:174 Entry #0: ValidationError when modifying LegacyImportStudent(name=u'banton11', school='gsmitte', dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') (source_uid:LegacyDB record_uid: banton11): {'name': ['The name is already used somewhere outside the school. It may not be taken twice and has to be changed.']} Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 142, in create_and_modify_users success = user.modify(lo=self.connection) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 461, in modify success = self.modify_without_hooks(lo, validate, move_if_necessary) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 621, in modify_without_hooks success = super(ImportUser, self).modify_without_hooks(lo, validate, move_if_necessary) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 475, in modify_without_hooks raise ValidationError(self.errors.copy()) UserValidationError: ValidationError when modifying LegacyImportStudent(name=u'banton11', school='gsmitte', dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') (source_uid:LegacyDB record_uid: banton11): {'name': ['The name is already used somewhere outside the school. It may not be taken twice and has to be changed.']} Then I tried to import the line via the *OLD* legacy import script, which returned the following: Processing line 1: M banton11 Richterin Antonia gsmitte gsmitte-1A bAnton11@nstx.local 0 1 0 deleted: uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local WARNING: user to modify not found, verify ou for school nr gsmitte creating object uid=banton11,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local created: uid=banton11,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local I.e. the *OLD* script removed and recreated the user. Is it a big code change, to achieve the old behaviour?
(In reply to Daniel Tröder from comment #13) > > 9) Deleting non-existing users is currently no error > > I tried to delete banton9 twice in 2 seperate attempts. There was an error > > message, but no error entry within the statistic part of the log: > The legacy script does not produce an error for deleting non-existing users > and exists with 0. That behavior was copied. > Should it be different for the non-legacy user import? For non-legacy I would suspect this as an error. > > 10) exit codes > > The process should exit with a non-zero exitcode if at least one error > > occured. > > Not well thought out idea: > > Exit code 1: fatal error > > Exit code 2: at least one non-fatal error > r70084: exit code is 0 if all went well, 1 if a fatal error occured, 2 if at > least one non-fatal error occured Great. > Regarding 2) from above: > A TimedRotatingFileHandler is used for logging. It will rename once per day > the logfile (e.g. ucs-school-import.log.2016-06-08) and keep 10 copies. Please keep all copies. Sometimes we have to analyse changes that have been made months ago (> 6 months). Detailed logs like this would help a lot.
(In reply to Sönke Schwardt-Krummrich from comment #14) > 11) Moving users to another school does not work > A test user was located in "gsnord" and I did a s/gsnord/gsmitte/g on the > import line and tried to import it: > > 2016-06-01 01:28:04 INFO user_import.create_and_modify_users:128 Modifying > LegacyImportStudent(name=u'banton11', school='gsmitte', > dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') > (source_uid:LegacyDB record_uid:banton11) attributes={'$dn$': > 'uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local', > 'display_name': 'Antonia Richterin', 'record_uid': u'banton11', 'firstname': > 'Antonia', 'lastname': 'Richterin', 'type_name': 'Student', 'school': > 'gsmitte', 'name': u'banton11', 'disabled': 'none', 'email': > u'bAnton11@nstx.local', 'birthday': None, 'type': 'legacyImportStudent', > 'schools': ['gsmitte'], 'password': 'xxxxxxxxxx', 'source_uid': u'LegacyDB', > 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} > udm_properties={}... > 2016-06-01 01:28:04 DEBUG base.call_hooks:353 > /usr/share/ucs-school-import/hooks/user_modify_pre.d not found or empty. > 2016-06-01 01:28:04 INFO base.modify_without_hooks:467 Modifying > LegacyImportStudent(name=u'banton11', school='gsmitte', > dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') > 2016-06-01 01:28:04 DEBUG base.cache:209 Initializing ('School', ('name', > 'gsmitte')) > 2016-06-01 01:28:04 DEBUG base.get_udm_object:636 Getting School UDM object > by dn: ou=gsmitte,dc=nstx,dc=local > 2016-06-01 01:28:04 DEBUG base.get_only_udm_obj:838 Getting > LegacyImportStudent UDM object by filter: > &(!(uid=banton11))(mailPrimaryAddress=bAnton11@nstx.local) > 2016-06-01 01:28:04 ERROR user_import.create_and_modify_users:174 Entry #0: > ValidationError when modifying LegacyImportStudent(name=u'banton11', > school='gsmitte', > dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') > (source_uid:LegacyDB record_uid: banton11): {'name': ['The name is already > used somewhere outside the school. It may not be taken twice and has to be > changed.']} > Traceback (most recent call last): > File > "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", > line 142, in create_and_modify_users > success = user.modify(lo=self.connection) > File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line > 461, in modify > success = self.modify_without_hooks(lo, validate, move_if_necessary) > File > "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", > line 621, in modify_without_hooks > success = super(ImportUser, self).modify_without_hooks(lo, validate, > move_if_necessary) > File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line > 475, in modify_without_hooks > raise ValidationError(self.errors.copy()) > UserValidationError: ValidationError when modifying > LegacyImportStudent(name=u'banton11', school='gsmitte', > dn='uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local') > (source_uid:LegacyDB record_uid: banton11): {'name': ['The name is already > used somewhere outside the school. It may not be taken twice and has to be > changed.']} > > Then I tried to import the line via the *OLD* legacy import script, which > returned the following: > > Processing line 1: M banton11 Richterin Antonia gsmitte > gsmitte-1A bAnton11@nstx.local 0 1 0 > deleted: uid=banton11,cn=schueler,cn=users,ou=gsnord,dc=nstx,dc=local > WARNING: user to modify not found, verify ou for school nr gsmitte > creating object uid=banton11,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local > created: uid=banton11,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local > > I.e. the *OLD* script removed and recreated the user. Is it a big code > change, to achieve the old behaviour? No - that would be easy. AFAIK it would also not be a problem to detect and do the proper school-move... I think it did work once...
12) tested with existing customer hooks and real input data → the mail address is optional; so import of the user should not fail if no mail address has been specified (to be precise: the empty string ''). 2016-06-01 02:31:32 DEBUG base_reader.next:73 Input 1: ['A', 'olivia284', 'Karlsfeld', 'Olivia-Maxima', 'gsmitte', 'gsmitte-10C', '', '', '0', '1', '0', '', '', '', '', 'olivia284@nstx.local', 'gsmitte-Kurs-10-BIO'] -> {u'11': u'', u'10' : u'0', u'13': u'', u'12': u'', u'15': u'olivia284@nstx.local', u'14': u'', u'16': u'gsmitte-Kurs-10-BIO', u'1': u'olivia284', u'0': u'A', u'3': u'Olivia-Maxima', u'2': u'Karlsfeld', u'5': u'gsmitte-10C', u'4': u'gsmitte', u'7': u'', u'6': u'', u'9': u'1', u'8': u'0'} 2016-06-01 02:31:32 DEBUG base.cache:209 Initializing ('SchoolClass', ('name', 'gsmitte-10C'), ('school', 'gsmitte')) 2016-06-01 02:31:32 DEBUG csv_reader.map:186 LegacyImportStudent(name=u'olivia284', school=u'gsmitte', dn=None) attributes={'$dn$': u'uid=olivia284,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', 'display_name': u'Olivia-Maxima Karlsfeld', 'record_uid': None, 'firstname': u'Olivia-Maxima', 'lastname': u'Karlsfeld', 'type_name': 'Student', 'school': u'gsmitte', 'name': u'olivia284', 'disabled': 'none', 'email': u'', 'birthday': None, 'type': 'legacyImportStudent', 'schools': None, 'password': 'xxxxxxxxxx', 'source_uid': None, 'school_classes': {'gsmitte': ['gsmitte-10C']}, 'objectType': 'users/user'} udm_properties={} 2016-06-01 02:31:32 INFO user_import.read_input:83 Done reading 1. user: LegacyImportStudent(name=u'olivia284', school=u'gsmitte', dn=None) 2016-06-01 02:31:32 INFO user_import.read_input:91 ------ Read 1 users from input data. ------ 2016-06-01 02:31:32 INFO user_import.create_and_modify_users:109 ------ Creating / modifying users... ------ 2016-06-01 02:31:32 DEBUG user_import.create_and_modify_users:112 Creating / modifying user LegacyImportStudent(name=u'olivia284', school=u'gsmitte', dn=None)... 2016-06-01 02:31:32 DEBUG base.get_only_udm_obj:838 Getting LegacyImportStudent UDM object by filter: (&(objectClass=ucsschoolType)(ucsschoolSourceUID=LegacyDB)(ucsschoolRecordUID=olivia284)) 2016-06-01 02:31:32 ERROR user_import.create_and_modify_users:174 Entry #1: Could not create 'email' from scheme '<email>' and input data {'$dn$': u'uid=olivia284,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', u'__is_staff': '0', u'firstname': 'Olivia-Maxima', u'lastname': 'Karlsfeld', u'__action': 'A', 'disabled': 'none', 'birthday': None, 'record_uid': u'olivia284', 'overridePWLength': '1', 'schools': ['gsmitte'], 'overridePWHistory': '1', 'password': u'`2q;=K;C', 'source_uid': u'LegacyDB', 'objectType': 'users/user', u'__ignore': '', u'school': 'gsmitte', 'maildomain': 'nstx.local', 'display_name': 'Olivia-Maxima Karlsfeld', u'name': u'olivia284', u'__activate': '1', 'type_name': 'Student', u'school_classes': {'gsmitte': ['gsmitte-10C']}, u'__is_teacher': '0', 'type': 'legacyImportStudent', u'email': u''}. Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 113, in create_and_modify_users user = self.determine_add_modify_action(imported_user) File "/usr/lib/pymodules/python2.7/ucsschool/importer/legacy/legacy_user_import.py", line 83, in determine_add_modify_action imported_user.prepare_all(new_user=True) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 171, in prepare_all self.prepare_attributes(new_user) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 193, in prepare_attributes self.make_email() File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 321, in make_email self.email = self.format_from_scheme("email", self.config["scheme"]["email"], maildomain=maildomain).lower() File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 586, in format_from_scheme entry=self.entry_count, import_user=self) FormatError: Could not create 'email' from scheme '<email>' and input data {'$dn$': u'uid=olivia284,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', u'__is_staff': '0', u'firstname': 'Olivia-Maxima', u'lastname': 'Karlsfeld', u'__action': 'A', 'disabled': 'none', 'birthday': None, 'record_uid': u'olivia284', 'overridePWLength': '1', 'schools': ['gsmitte'], 'overridePWHistory': '1', 'password': u'`2q;=K;C', 'source_uid': u'LegacyDB', 'objectType': 'users/user', u'__ignore': '', u'school': 'gsmitte', 'maildomain': 'nstx.local', 'display_name': 'Olivia-Maxima Karlsfeld', u'name': u'olivia284', u'__activate': '1', 'type_name': 'Student', u'school_classes': {'gsmitte': ['gsmitte-10C']}, u'__is_teacher': '0', 'type': 'legacyImportStudent', u'email': u''}. 13) tested with existing customer hooks and real input data → the legacy parsing is not correct! In this case, the column 16 contains comma-separated group names and it looks like the whole line was split at commata instead of tabs. 2016-06-01 02:36:49 INFO user_import.read_input:79 ------ Starting to read users from input data... ------ 2016-06-01 02:36:49 DEBUG base_reader.next:73 Input 1: ['A\tolivia284\tKarlsfeld\tOlivia-Maxima\tgsmitte\tgsmitte-10C\t\t\t0\t1\t0\t\t\t\t\tolivia284@nstx.local\tgsmitte-Kurs-10-BIO', 'gsmitte-Kurs-10-ENG'] -> {u'1': u'gsmitte-Kurs-10-ENG', u'0': u'A\tolivia284\tKarlsfeld\tOlivia-Maxima\tgsmitte\tgsmitte-10C\t\t\t0\t1\t0\t\t\t\t\tolivia284@nstx.local\tgsmitte-Kurs-10-BIO'} If I remove the comma vom column 16, it works (see item 12 above).
(In reply to Daniel Tröder from comment #16) > (In reply to Sönke Schwardt-Krummrich from comment #14) > > I.e. the *OLD* script removed and recreated the user. Is it a big code > > change, to achieve the old behaviour? > No - that would be easy. AFAIK it would also not be a problem to detect and > do the proper school-move... I think it did work once... Then I would prefer the proper school-move.
Tested so far: - adding users - once → OK - twice → fails as expected → OK - with lower-case username → OK - with mixed-case username → OK - without mail address → FAIL - with comma in values → FAIL - modifying users - once → OK - twice in the same file → fails as expected → OK - with lower-case username → OK - with mixed-case username → OK - removing users - once → OK - twice → no real error, but does not show up in statistics - LDAP values match with input data - add → OK - modify → OK - remove → OK - hooks - UNTESTED due to errors above
(In reply to Sönke Schwardt-Krummrich from comment #15) > (In reply to Daniel Tröder from comment #13) > > > 9) Deleting non-existing users is currently no error > > > I tried to delete banton9 twice in 2 seperate attempts. There was an error > > > message, but no error entry within the statistic part of the log: > > The legacy script does not produce an error for deleting non-existing users > > and exists with 0. That behavior was copied. > > Should it be different for the non-legacy user import? > > For non-legacy I would suspect this as an error. New and legacy will both create an error, if the user cannot be deleted from LDAP. In the case of the legacy import, it had logged an error, that a user marked with a 'D' in the input data could not be found in the LDAP. The new import does not expect an action in the input. If it has, the method detect_users_to_delete() of UserImport must be adapted accordingly (like in LegacyUserImport). > > Regarding 2) from above: > > A TimedRotatingFileHandler is used for logging. It will rename once per day > > the logfile (e.g. ucs-school-import.log.2016-06-08) and keep 10 copies. > > Please keep all copies. Sometimes we have to analyse changes that have been > made months ago (> 6 months). Detailed logs like this would help a lot. With r70106 logfiles are kept for some millenia. (In reply to Sönke Schwardt-Krummrich from comment #17) > 12) tested with existing customer hooks and real input data > → the mail address is optional; so import of the user should not fail if no > mail address has been specified (to be precise: the empty string ''). The error is created if the result of a formatting is empty. Was meant as a heuristic, has been converted to a warning (r70131). > 13) tested with existing customer hooks and real input data > → the legacy parsing is not correct! In this case, the column 16 contains > comma-separated group names and it looks like the whole line was split at > commata instead of tabs. > > If I remove the comma vom column 16, it works (see item 12 above). The CSV-dialect is detected by Pythons csv library. A specific dialect can only be forced by subclassing, but the delimiter can be configured in the configuration. I added to user_import_legacy_defaults.json (r70131): "delimiter": "\t" Please retry after copying it from /usr/.. to /var/.. or by setting it on the command line: "--set csv:delimiter='\t'".
(In reply to Sönke Schwardt-Krummrich from comment #18) > (In reply to Daniel Tröder from comment #16) > > (In reply to Sönke Schwardt-Krummrich from comment #14) > > > I.e. the *OLD* script removed and recreated the user. Is it a big code > > > change, to achieve the old behaviour? > > No - that would be easy. AFAIK it would also not be a problem to detect and > > do the proper school-move... I think it did work once... > > Then I would prefer the proper school-move. School move implemented in r70140.
r70142: honor return code of libs school move
(In reply to Sönke Schwardt-Krummrich from comment #19) > Tested so far: > - adding users > - once → OK > - twice → fails as expected → OK > - with lower-case username → OK > - with mixed-case username → OK > - without mail address → FAIL → OK: retested > - with comma in values → FAIL → OK: retested > - modifying users > - once → OK > - twice in the same file → fails as expected → OK > - with lower-case username → OK > - with mixed-case username → OK > - removing users > - once → OK > - twice → no real error, but does not show up in statistics > - LDAP values match with input data > - add → OK > - modify → OK > - remove → OK > - hooks > - UNTESTED due to errors above → tested with legacy CLI hooks - moving users to another OU (previous versions did a remove+add!) - worked quite well, but a wrong class membership remains → Bug 41593 → VERIFIED
UCS@school 4.1 R2 has been released: http://docs.software-univention.de/release-notes-ucsschool-4.1R2v1-de.pdf If this error occurs again, please use "Clone This Bug".