Univention Bugzilla – Bug 49562
User Import fails with unknown reason
Last modified: 2019-08-19 10:25:50 CEST
Importing csv based users, teachers and employees based on doc (see URL). I remember the API-import requires header lines but the doc does not mention them. At least, print some useable error message! ========================================== root@master:~# head -n 2 user.csv A LeMAT.verhtbarLe verhtbare Tufling SchuleLena LeMAT.verhtbarLe@schulen.ucs 0 1 1 A LeMAr.CodegeneLe Codegenerierungsphasen reuevollerer SchuleLena LeMAr.CodegeneLe@schulen.ucs 0 1 1 ========================================== root@master:~# /usr/share/ucs-school-import/scripts/import_user user.csv infile is: user.csv 2019-05-28 14:07:50 INFO cmdline.prepare_import:132 ------ UCS@school import tool starting ------ 2019-05-28 14:07:50 INFO configuration.read:92 Reading configuration from '/usr/share/ucs-school-import/configs/global_defaults.json'... 2019-05-28 14:07:50 INFO configuration.read:92 Reading configuration from '/var/lib/ucs-school-import/configs/global.json'... 2019-05-28 14:07:50 INFO configuration.read:92 Reading configuration from '/usr/share/ucs-school-import/configs/user_import_defaults.json'... 2019-05-28 14:07:50 INFO configuration.read:92 Reading configuration from '/var/lib/ucs-school-import/configs/user_import.json'... 2019-05-28 14:07:50 INFO configuration.read:92 Reading configuration from '/usr/share/ucs-school-import/configs/user_import_legacy_defaults.json'... 2019-05-28 14:07:50 INFO configuration.read:92 Reading configuration from '/var/lib/ucs-school-import/configs/user_import_legacy.json'... 2019-05-28 14:07:50 INFO configuration._set_username_maxlength:65 UCRV ucsschool/username/max_length: None 2019-05-28 14:07:50 INFO configuration._set_username_maxlength:73 Set value of configuration key username:max_length:default to default value of UCR variable ucsschool/username/max_length: 20. 2019-05-28 14:07:50 INFO configuration._set_username_maxlength:82 Set value of configuration key username:max_length:student to username:max_length:default reduced by length of the exam-prefix ('exam-'): 15. 2019-05-28 14:07:50 INFO pyhooks_loader.get_hook_classes:110 Searching for hooks of type 'ConfigPyHook' in: /usr/share/ucs-school-import/pyhooks... 2019-05-28 14:07:50 INFO pyhooks_loader.get_hook_classes:125 Found hook classes: 2019-05-28 14:07:50 INFO pyhooks_loader.get_hook_objects:163 Loaded hooks: {}. 2019-05-28 14:07:50 INFO configuration.setup_configuration:58 Finished reading configuration, starting checks... 2019-05-28 14:07:50 INFO pyhooks_loader.get_hook_classes:110 Searching for hooks of type 'ConfigurationChecks' in: /usr/share/ucs-school-import/checks... 2019-05-28 14:07:50 INFO pyhooks_loader.get_hook_classes:124 Hook class 'SingleSourcePartialImportConfigurationChecks' filtered out by is_module_in_config(). 2019-05-28 14:07:50 INFO pyhooks_loader.get_hook_classes:125 Found hook classes: DefaultConfigurationChecks 2019-05-28 14:07:50 INFO cmdline.prepare_import:138 ------ UCS@school import tool configured ------ 2019-05-28 14:07:50 INFO cmdline.prepare_import:139 Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaults.json', '/var/lib/ucs-school-import/configs/user_import.json', '/usr/share/ucs-school-import/configs/user_import_legacy_defaults.json', '/var/lib/ucs-school-import/configs/user_import_legacy.json']. 2019-05-28 14:07:50 INFO cmdline.prepare_import:140 Using command line arguments: {'input': {'filename': 'user.csv'}, 'output': {'new_user_passwords': None}} 2019-05-28 14:07:50 INFO cmdline.prepare_import:141 Configuration is: {u'activate_new_users': {u'default': True}, u'classes': {}, u'configuration_checks': [u'defaults'], u'csv': {u'allowed_missing_columns': [], u'delimiter': u'\t', u'header_lines': 0, u'incell-delimiter': {u'default': u','}, u'mapping': {u'0': u'__action', u'1': u'name', u'10': u'__is_staff', u'11': u'password', u'12': u'__ignore', u'13': u'__ignore', u'14': u'__ignore', u'15': u'__ignore', u'16': u'__ignore', u'17': u'__ignore', u'18': u'__ignore', u'19': u'__ignore', u'2': u'lastname', u'20': u'__ignore', u'21': u'__ignore', u'22': u'__ignore', u'23': u'__ignore', u'24': u'__ignore', u'25': u'__ignore', u'26': u'__ignore', u'27': u'__ignore', u'28': u'__ignore', u'29': u'__ignore', u'3': u'firstname', u'30': u'__ignore', u'4': u'school', u'5': u'school_classes', u'6': u'__ignore', u'7': u'email', u'8': u'__is_teacher', u'9': u'__activate'}}, u'deletion_grace_period': {u'deactivation': 0, u'deletion': 0}, u'dry_run': False, u'factory': u'ucsschool.importer.legacy.legacy_csv_user_import_factory.LegacyCsvUserImportFactory', u'hooks_dir_pyhook': u'/usr/share/ucs-school-import/pyhooks', u'input': {u'filename': 'user.csv', u'type': u'csv'}, u'logfile': u'/var/log/univention/ucs-school-import.log', u'maildomain': None, u'mandatory_attributes': [u'firstname', u'lastname', u'name', u'record_uid', u'school', u'source_uid'], u'no_delete': False, u'normalize': {u'firstname': False, u'lastname': False}, u'output': {u'new_user_passwords': None, u'user_import_summary': u'/var/lib/ucs-school-import/summary/%Y/%m/user_import_summary_%Y-%m-%d_%H:%M:%S.csv'}, u'password_length': 8, u'scheme': {u'record_uid': u'<name>'}, u'school': None, u'school_classes_keep_if_empty': True, u'source_uid': u'LegacyDB', u'tolerate_errors': -1, u'user_role': None, u'username': {u'allowed_special_chars': u'.', 'max_length': {'default': 20, 'student': 15}}, u'verbose': True} 2019-05-28 14:07:50 INFO utils.stopped_notifier:535 Stopping univention-directory-notifier 2019-05-28 14:07:56 INFO utils._run:529 Stopping univention-directory-notifier (via systemctl): univention-directory-notifier.service. 2019-05-28 14:07:56 INFO utils.stopped_notifier:547 univention-directory-notifier stopped 2019-05-28 14:07:56 INFO mass_import.import_users:110 ------ Importing users... ------ 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_classes:110 Searching for hooks of type 'PreReadPyHook' in: /usr/share/ucs-school-import/pyhooks... 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_classes:125 Found hook classes: 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_objects:163 Loaded hooks: {}. 2019-05-28 14:07:56 INFO user_import.read_input:102 ------ Starting to read users from input data... ------ 2019-05-28 14:07:56 DEBUG csv_reader.read:145 Reading 'user.csv' with encoding 'us-ascii'. 2019-05-28 14:07:56 ERROR user_import.read_input:112 Error reading 1. user: Columns configured in csv:mapping missing: [u'10']. Header found: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'] Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 105, in read_input import_user = self.reader.next() File "/usr/lib/pymodules/python2.7/ucsschool/importer/reader/base_reader.py", line 84, in next input_dict = self.import_users.next() File "/usr/lib/pymodules/python2.7/ucsschool/importer/reader/csv_reader.py", line 169, in read missing_columns, self.fieldnames)) ConfigurationError: Columns configured in csv:mapping missing: [u'10']. Header found: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'] 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_classes:110 Searching for hooks of type 'PostReadPyHook' in: /usr/share/ucs-school-import/pyhooks... 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_classes:125 Found hook classes: 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_objects:163 Loaded hooks: {}. 2019-05-28 14:07:56 INFO user_import.read_input:115 ------ Read 0 users from input data. ------ 2019-05-28 14:07:56 INFO legacy_user_import.detect_users_to_delete:48 ------ Detecting which users to delete... ------ 2019-05-28 14:07:56 INFO user_import.delete_users:364 ------ Deleting 0 users... ------ 2019-05-28 14:07:56 INFO user_import.delete_users:400 ------ Deleted 0 users. ------ 2019-05-28 14:07:56 INFO user_import.create_and_modify_users:131 ------ Creating / modifying users... ------ 2019-05-28 14:07:56 INFO user_import.create_and_modify_users:216 ------ Created 0 users, modified 0 users. ------ 2019-05-28 14:07:56 INFO user_import.log_stats:585 ------ User import statistics ------ 2019-05-28 14:07:56 INFO user_import.log_stats:618 Read users from input data: 0 2019-05-28 14:07:56 INFO user_import.log_stats:618 Errors: 1 2019-05-28 14:07:56 INFO user_import.log_stats:618 Entry | User | Error description 2019-05-28 14:07:56 INFO user_import.log_stats:618 ------------------------------------- 2019-05-28 14:07:56 INFO user_import.log_stats:618 0 | <No name> | Columns configured in csv:mapping missing: [u'10']. Header found: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'] 2019-05-28 14:07:56 INFO user_import.log_stats:619 ------ End of user import statistics ------ 2019-05-28 14:07:56 INFO mass_import.import_users:136 ------ Writing user import summary to /var/lib/ucs-school-import/summary/2019/05/user_import_summary_2019-05-28_14:07:56.csv... ------ 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_classes:110 Searching for hooks of type 'ResultPyHook' in: /usr/share/ucs-school-import/pyhooks... 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_classes:125 Found hook classes: 2019-05-28 14:07:56 INFO pyhooks_loader.get_hook_objects:163 Loaded hooks: {}. 2019-05-28 14:07:56 INFO mass_import.import_users:140 ------ Importing users done. ------ 2019-05-28 14:07:56 INFO utils.stopped_notifier:557 Starting univention-directory-notifier 2019-05-28 14:07:56 INFO utils._run:529 Starting univention-directory-notifier (via systemctl): univention-directory-notifier.service. 2019-05-28 14:07:56 INFO utils.stopped_notifier:564 univention-directory-notifier started 2019-05-28 14:07:56 WARNING cmdline.main:158 Import finished normally but with errors.
It's a documentation error. The docs will be fixed though: Bug #49318. The correct doc is https://docs.software-univention.de/ucsschool-import-handbuch-4.4.html *** This bug has been marked as a duplicate of bug 49318 ***
The fix implements changes in the scenario document with bug 49318 . It has nothing to do with the issue here. I followed the documentation from the handbook. (see https://docs.software-univention.de/ucsschool-handbuch-4.4.html#school:setup:cli) If there is an official documentation for the importer it should (at least) be referred to in the handbook instead of showing outdated documentation.
The bugs scope grew beyond the title. The main manual now references both old and new imports. See current (still unpublished) doc build: http://jenkins.knut.univention.de:8080/job/UCSschool-4.4/job/Handbook/29/artifact/webroot/ucsschool-handbuch-4.4.html#school:setup:cli:importusers It references the manual for the new import: http://jenkins.knut.univention.de:8080/job/UCSschool-4.4/job/Handbook/29/artifact/webroot/ucsschool-import-handbuch-4.4.html and the legacy import: http://jenkins.knut.univention.de:8080/job/UCSschool-4.4/job/Handbook/29/artifact/webroot/ucsschool-import-legacy-4.4.html
Well, all docs seems to be written and fixed but they are not yet published. Is this bug then resolved/fixed at this stage?
(In reply to Christian Völker from comment #4) > Is this bug then resolved/fixed at this stage? No, but it will be when the other bug is published (→ CLOSED). So setting this one to be a duplicate of Bug #49318. *** This bug has been marked as a duplicate of bug 49318 ***