Univention Bugzilla – Bug 50406
ucs-school-purge-expired-users fails if SiSoPi config checks are present in config
Last modified: 2020-03-25 16:09:32 CET
ucs-school-purge-expired-users fails if the "sisopi" configuration check is present in the user import configuration. Traceback (most recent call last): File "/usr/share/ucs-school-import/scripts/ucs-school-purge-expired-users", line 180, in <module> sys.exit(main()) File "/usr/share/ucs-school-import/scripts/ucs-school-purge-expired-users", line 141, in main config = uic.setup_config() File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 86, in setup_config self.config = setup_configuration(configs, **self.args.settings) File "/usr/lib/pymodules/python2.7/ucsschool/importer/configuration.py", line 59, in setup_configuration run_configuration_checks(config) File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/configuration_checks.py", line 116, in run_configuration_checks method() File "/usr/share/ucs-school-import/checks/sisopi.py", line 44, in test_00_required_config_keys raise InitialisationError('No {!r} was specified in the configuration.'.format(attr)) ucsschool.importer.exceptions.InitialisationError: No 'school' was specified in the configuration.
It still doesn't work but I'm now getting this Traceback instead: 2020-02-16 04:50:03 ERROR cmdline.setup_config:88 Error setting up or checking the configuration: No 'school' was specified in the configuration. Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 86, in setup_config self.config = setup_configuration(configs, **self.args.settings) File "/usr/lib/pymodules/python2.7/ucsschool/importer/configuration.py", line 59, in setup_configuration run_configuration_checks(config) File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/configuration_checks.py", line 116, in run_configuration_checks method() File "/usr/share/ucs-school-import/checks/sisopi.py", line 44, in test_00_required_config_keys raise InitialisationError('No {!r} was specified in the configuration.'.format(attr)) InitialisationError: No 'school' was specified in the configuration.
AFAICS the 'school' setting is not relevant for the ucs-school-purge-expired-users script. So if the static configuration does not have a value for it, a dummy value can be used to satisfy the check test_00_required_config_keys. The fix would thus be a one liner adding: --- uic.args.settings['school'] = 'foo' --- in line 140.
Instead of faking a missing value, we decided to add the possibility to generally disable certain checks of the UCS@school import. It was decided to not document this feature publicly, as its *usage by customers is not be supported*. Documentation for internal purposes should be created thought. For this specific bug, the check "test_00_required_config_keys" of the "SingleSourcePartialImportConfigurationChecks" class was deactivated in the purge-user script (and a developer-targeted helper module). Changes to the ucs-school-import and ucs-test-ucsschool packages have been commit to the branch dtroeder/50406_sisopi_purge: [dtroeder/50406_sisopi_purge] 886db5509 Bug #50406: add support for disabling checks [dtroeder/50406_sisopi_purge] e26c36e13 Bug #50406: disable test_00_required_config_keys check for SiSoPi users [dtroeder/50406_sisopi_purge] ba71e1fb1 Bug #50406: improve ucsschool.lib usage [dtroeder/50406_sisopi_purge] 1a0ed75a8 Bug #50406: add test for 'disabled_checks' feature QA: please reopen for changelog, package build and advisory
In [dtroeder/50406_sisopi_purge] ba71e1fb1 Bug #50406: improve ucsschool.lib usage: m = re.match(r".*,ou=(\w+),dc=uni,dc=dtr$", dn) You put code specific to your testing environment. Please refactor. Otherwise it looks good. I much prefer the addition of that 'hidden feature' over any fake attributes.
What's the best practice/recommended way to use this feature to get ucs-school-purge-expired-users working again? I'm asking because most of my "SiSoPi" customers are using the UMC module for user imports. Currently I'd say the best practice would be to have a separate import config file, which is identical to the "normal" one apart from the sisopi check being deactivated with the functionality introduced here and passing it via -c to ucs-school-purge-expired-users. What do you think? In other words: How should I use this feature to get ucs-school-purge-expired-users working without changing anything for my actual import / the config it uses?
Fixed (and improved) the code. Used --fixup, so the commit "improve ucsschool.lib usage" was updated. Merged to 4.4 and build. [4.4] ffd20929c Bug #50406: add support for disabling checks [4.4] 006576a93 Bug #50406: disable test_00_required_config_keys check for SiSoPi users [4.4] 5cf056644 Bug #50406: improve ucsschool.lib usage [4.4] 1fb191032 Bug #50406: add test for 'disabled_checks' feature [4.4] 1d4b11c02 Bug #50406: Merge branch 'dtroeder/50406_sisopi_purge' into 4.4 [4.4] 76cb64ffe Bug #50406: changelog, advisory [4.4] 0815dfbed Bug #50406: advisory update [4.4] a2e57170d Bug #50406: changelog ucs-school-import (17.0.28) ucs-test-ucsschool (6.0.104)
(In reply to Valentin Heidelberger from comment #6) > What's the best practice/recommended way to use this feature to get > ucs-school-purge-expired-users working again? The ucs-school-purge-expired-users script will start working again automatically after the update - no interaction is required. > I'm asking because most of my "SiSoPi" customers are using the UMC module > for user imports. Currently I'd say the best practice would be to have a > separate import config file, which is identical to the "normal" one apart > from the sisopi check being deactivated with the functionality introduced > here and passing it via -c to ucs-school-purge-expired-users. What do you > think? > > In other words: How should I use this feature to get > ucs-school-purge-expired-users working without changing anything for my > actual import / the config it uses? Don't do anything, just update.
I have a failing test. Installed testing packages on a pristine UCS@school SingleServer and ran the tests. I could not find out where this is coming from. apt-cache policy ucs-school-import ucs-school-import: Installed: 17.0.28A~4.4.0.202003171318 apt-cache policy ucs-test-ucsschool ucs-test-ucsschool: Installed: 6.0.104A~4.4.0.202003171321 ./249_import_user_disabled_checks -f 2020-03-16 09:33:06 INFO importusers_cli_v2.setup_testenv:280 *** Creating mail domain 'realm1.intranet'... Creating mail/domain object with /usr/sbin/udm-test mail/domain create --position cn=domain,cn=mail,dc=realm1,dc=intranet --ignore_exists --set name=realm1.intranet 2020-03-16 09:33:08 INFO importusers_cli_v2.create_ous:268 Creating OUs... 2020-03-16 09:33:08 INFO ucs_test_school.check_name_edudc:339 *** It is not allowed to set the master as name_edudc ==> resetting name_edudc to None 2020-03-16 09:33:08 INFO ucs_test_school.create_multiple_ous:463 *** Chose 1/1 OUs from cache for arguments (None, None, None, '', None, False): [('8555w', 'ou=8555w,dc=realm1,dc=intranet')]. 2020-03-16 09:33:08 INFO importusers_cli_v2.create_ous:274 Created OUs: ['8555w']. 2020-03-16 09:33:08 INFO 249_import_user_disabled_checks.test:35 *** 1/2 Starting import without "disabled_checks" set... 2020-03-16 09:33:08 INFO importusers_cli_v2.create_csv_file:452 Header row = ['role', 'Nach', 'Vor', 'record_uid', 'Gruppen', 'Beschreibung', 'OUs', 'Benutzername', 'E-Mail'] 2020-03-16 09:33:08 INFO importusers_cli_v2.create_csv_file:465 Person data = {'Gruppen': '8555w-19iw', 'role': 'student', 'record_uid': 'record_uid-cbh3ckfavl', 'Vor': 'dp1302sgaz', 'E-Mail': 'je6fviqvca@realm1.intranet', 'OUs': '8555w', 'Nach': 'z51p4fus1c', 'Benutzername': 'je6fviqvca', 'Beschreibung': None} 2020-03-16 09:33:08 INFO importusers_cli_v2.create_csv_file:465 Person data = {'Gruppen': '8555w-45ob,8555w-28vz,8555w-82kw', 'role': 'teacher', 'record_uid': 'record_uid-pvocyy98sa', 'Vor': 'e3lm4y76hc', 'E-Mail': 'i9fbzxdy4a@realm1.intranet', 'OUs': '8555w', 'Nach': 'ubj3njzqny', 'Benutzername': 'i9fbzxdy4a', 'Beschreibung': None} 2020-03-16 09:33:08 INFO importusers_cli_v2.create_csv_file:465 Person data = {'Gruppen': '', 'role': 'staff', 'record_uid': 'record_uid-zsywk1vr0l', 'Vor': 'z1tefpwucn', 'E-Mail': 'h5mjxstd0n@realm1.intranet', 'OUs': '8555w', 'Nach': 'g0hg7samoh', 'Benutzername': 'h5mjxstd0n', 'Beschreibung': None} 2020-03-16 09:33:08 INFO importusers_cli_v2.create_csv_file:465 Person data = {'Gruppen': '8555w-53xf', 'role': 'teacher_and_staff', 'record_uid': 'record_uid-qiit1uswop', 'Vor': 'en6dfprpcn', 'E-Mail': 'z3z3fri1ts@realm1.intranet', 'OUs': '8555w', 'Nach': 'jhh42lu6hl', 'Benutzername': 'z3z3fri1ts', 'Beschreibung': None} 2020-03-16 09:33:08 INFO importusers_cli_v2.create_config_json:429 Config: {'username': {'max_length': {'default': 40}}, 'tolerate_errors': 0, 'maildomain': 'realm1.intranet', 'verbose': True, 'source_uid': 'source_uid-oqupp11hjf', 'user_role': None, 'factory': 'ucsschool.importer.default_user_import_factory.DefaultUserImportFactory', 'classes': {}, 'user_deletion': 0, 'input': {'type': 'csv', 'filename': '/tmp/34_import-users_via_cli_v2.WaBrEt/users.4pJ2Pd'}, 'scheme': {'username': {'default': '<:umlauts><firstname>[0].<lastname>[COUNTER2]'}, 'record_uid': '<firstname>;<lastname>;<email>', 'email': '<:umlauts><firstname>[0].<lastname>@<maildomain>'}, 'csv': {'mapping': {'record_uid': 'record_uid', 'Gruppen': 'school_classes', 'Nach': 'lastname', 'Benutzername': 'name', 'Beschreibung': 'description', 'E-Mail': 'email', 'role': '__role', 'Vor': 'firstname', 'OUs': 'schools'}}} 2020-03-16 09:33:08 INFO importusers_cli_v2.run_import:504 Starting import: ['/usr/share/ucs-school-import/scripts/ucs-school-user-import', '-v', '-c', '/tmp/34_import-users_via_cli_v2.WaBrEt/config.wE8to3'] Traceback (most recent call last): File "/usr/share/ucs-school-import/scripts/ucs-school-user-import", line 37, in <module> from ucsschool.importer.frontend.user_import_cmdline import UserImportCommandLine ImportError: No module named importer.frontend.user_import_cmdline 2020-03-16 09:33:08 INFO importusers_cli_v2.run_import:508 Import process exited with exit code 1 2020-03-16 09:33:08 ERROR importusers_cli_v2.run_import:510 As requested raising an exception due to non-zero exit code 2020-03-16 09:33:08 INFO 249_import_user_disabled_checks.test:52 *** OK - import stopped. 2020-03-16 09:33:08 INFO 249_import_user_disabled_checks.test:54 *** 2/2 Starting import with "disabled_checks" set... 2020-03-16 09:33:08 INFO importusers_cli_v2.run_import:504 Starting import: ['/usr/share/ucs-school-import/scripts/ucs-school-user-import', '-v', '-c', '/tmp/34_import-users_via_cli_v2.WaBrEt/config.wE8to3', '--set', 'disabled_checks=test_username_max_length,test_deprecated_user_deletion'] Traceback (most recent call last): File "/usr/share/ucs-school-import/scripts/ucs-school-user-import", line 37, in <module> from ucsschool.importer.frontend.user_import_cmdline import UserImportCommandLine ImportError: No module named importer.frontend.user_import_cmdline 2020-03-16 09:33:08 INFO importusers_cli_v2.run_import:508 Import process exited with exit code 1 2020-03-16 09:33:08 ERROR importusers_cli_v2.run_import:510 As requested raising an exception due to non-zero exit code 2020-03-16 09:33:08 ERROR ucs_test_school.__exit__:161 *** Cleanup after exception: <class 'univention.testing.ucsschool.importusers_cli_v2.ImportException'> ImportException('Non-zero exit code 1',) Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/testing/ucsschool/importusers_cli_v2.py", line 295, in run self.test() File "249_import_user_disabled_checks", line 59, in test '--set', 'disabled_checks=test_username_max_length,test_deprecated_user_deletion' File "/usr/lib/pymodules/python2.7/univention/testing/ucsschool/importusers_cli_v2.py", line 511, in run_import raise ImportException('Non-zero exit code %r' % (exitcode,)) ImportException: Non-zero exit code 1 2020-03-16 09:33:08 INFO ucs_test_school.cleanup:244 Performing UCSTestSchool cleanup... 2020-03-16 09:33:08 INFO ucs_test_school.cleanup:269 Performing UDM cleanup... Performing UCSTestUDM cleanup... removing DN: cn=realm1.intranet,cn=domain,cn=mail,dc=realm1,dc=intranet Cleanup: wait for replication and drs removal trying to restart UDM CLI server sending signal 15 to process 26281 (['/usr/bin/python2.7', '/usr/share/univention-directory-manager-tools/univention-cli-server']) process already terminated UCSTestUDM cleanup done 2020-03-16 09:33:10 INFO ucs_test_school.cleanup:271 Performing UCR cleanup... Unsetting mail/hosteddomains Waiting for replication... 2020-03-16 09:33:11 INFO ucs_test_school.cleanup:276 UCSTestSchool cleanup done 2020-03-16 09:33:11 INFO importusers_cli_v2.cleanup:400 Performing CLI_Import_v2_Tester cleanup... 2020-03-16 09:33:11 INFO importusers_cli_v2.cleanup:401 Purging '/tmp/34_import-users_via_cli_v2.WaBrEt' 2020-03-16 09:33:11 INFO importusers_cli_v2.cleanup:195 Performing ImportTestbase cleanup... Done: replication complete. Performing UCSTestUDM cleanup... Cleanup: wait for replication and drs removal trying to restart UDM CLI server UCSTestUDM cleanup done 2020-03-16 09:33:14 INFO importusers_cli_v2.cleanup:197 Reverting UCR... 2020-03-16 09:33:14 INFO importusers_cli_v2.cleanup:199 ImportTestbase cleanup done 2020-03-16 09:33:14 INFO importusers_cli_v2.cleanup:409 CLI_Import_v2_Tester cleanup done Traceback (most recent call last): File "249_import_user_disabled_checks", line 65, in <module> Test().run() File "/usr/lib/pymodules/python2.7/univention/testing/ucsschool/importusers_cli_v2.py", line 295, in run self.test() File "249_import_user_disabled_checks", line 59, in test '--set', 'disabled_checks=test_username_max_length,test_deprecated_user_deletion' File "/usr/lib/pymodules/python2.7/univention/testing/ucsschool/importusers_cli_v2.py", line 511, in run_import raise ImportException('Non-zero exit code %r' % (exitcode,)) univention.testing.ucsschool.importusers_cli_v2.ImportException: Non-zero exit code 1 Starting 1 ucs-test at 2020-03-16 09:33:14 to /dev/null UCS 4.4-4-e499 ucs-test 9.0.3-179A~4.4.0.202003171539 Test 'disabled_checks' feature.......................... Test failed
The ucs-school-import package was missing the build depenedency for python-support. [4.4] 07ec37cd8 Bug #50406: add missing dependency [4.4] 7983d67aa Bug #50406: advisory update ucs-school-import (17.0.29)
Most of the QA was done by Ole, see comment 5 and comment 9. Tests are green now -> OK Updated version in yaml [4.4 5c057e76f] Bug #50406: yaml version -> verified
UCS@school 4.4 v5 has been released. https://docs.software-univention.de/changelog-ucsschool-4.4v5-de.html If this error occurs again, please clone this bug.