Univention Bugzilla – Bug 53649
Traceback during import, when deactivating a user instead of deleting and mandatory attributes are involved
Last modified: 2021-11-24 12:11:27 CET
The following traceback occurs, if an imported user should be deleted after a specified amount of time. 'deletion_grace_period' is set to e.g. 90 in the json file. Short extract from the json: ----------------------------------------------------------------- "mapping": { "vorname": "firstname", "familienname": "lastname", "klasse": "school_classes", "geburtsdatum": "birthday", "schueler_stamm_id": "employeeNumber" } }, "maildomain": "edukl.xyz", "scheme": { "record_uid": "<school>-<employeeNumber>", "username": { "default": "<firstname>[0:3]<lastname>[0:5]<:lower>[COUNTER2]" }, "email": "<username>@<school>.<maildomain>" }, "activate_new_users": { "default": true }, "deletion_grace_period": { "deactivation": 0, "deletion": 90 }, "mandatory_attributes": ["firstname", "lastname", "name", "record_uid", "school", "source_uid", "birthday", "employeeNumber"], "no_delete": false, ------------------------------------------ It seems the mandatory attribute employeeNumber is the problem, but the filter could not match at all? Getting ImportStudent UDM object by filter: &(!(uid=flodietr))(mailPrimaryAddress=flodietr@one.schein.qa) The user should be deactivated, and is not deleted, and should not at this moment. ------ Read 12 users from input data. ------ ------ Detecting which users to delete... ------ Searching with filter='(&(&(objectClass=ucsschoolStudent)(!(objectClass=ucsschoolExam)))(ucsschoolSourceUID=edoosys-one-student)(ucsschoolRecordUID=*))' users_to_delete=[(u'edoosys-one-student', u'one-40282097/3333/0174/0bb96e62/7bde', [])] ------ Deleting 1 users... ------ Getting ImportUser UDM object by filter: (&(&(objectClass=ucsschoolStudent)(!(objectClass=ucsschoolExam)))(ucsschoolSourceUID=edoosys-one-student)(ucsschoolRecordUID=one- 40282097/3333/0174/0bb96e62/7bde)) UDM object uid=flodietr,cn=schueler,cn=users,ou=one,dc=schein,dc=qa is not ImportUser, but actually ImportStudent Deactivating user ImportStudent(name='flodietr', school='one', dn='uid=flodietr,cn=schueler,cn=users,ou=one,dc=schein,dc=qa')... Setting deletion grace date of ImportStudent(name='flodietr', school='one', dn='uid=flodietr,cn=schueler,cn=users,ou=one,dc=schein,dc=qa') to '2021-11-09'... Searching for hooks of type 'UserPyHook' in: /usr/share/ucs-school-import/pyhooks... Found hook classes: Loaded hooks: {}. Starting ImportStudent.call_hooks('pre', 'modify', lo('cn=admin,dc=schein,dc=qa')) for ImportStudent(name='flodietr', school='one', dn='uid=flodietr,cn=schueler,cn=users, ou=one,dc=schein,dc=qa'). Hook directory /usr/share/ucs-school-import/hooks/user_modify_pre.d not found or empty. Searching for hooks of type 'Hook' in: /var/lib/ucs-school-lib/hooks... Found hook classes: Loaded hooks: {}. Modifying ImportStudent(name='flodietr', school='one', dn='uid=flodietr,cn=schueler,cn=users,ou=one,dc=schein,dc=qa') Getting School UDM object by dn: ou=one,dc=schein,dc=qa Getting ImportStudent UDM object by filter: &(!(uid=flodietr))(mailPrimaryAddress=flodietr@one.schein.qa) Error in entry #0: Mandatory attribute u'employeeNumber' does not exist. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/ucsschool/importer/mass_import/user_import.py", line 480, in delete_users success = self.do_delete(user) File "/usr/lib/python2.7/dist-packages/ucsschool/importer/mass_import/user_import.py", line 614, in do_delete success = user.modify(lo=self.connection) File "/usr/lib/python2.7/dist-packages/ucsschool/importer/models/import_user.py", line 1056, in modify res = super(ImportUser, self).modify(lo, validate, move_if_necessary) File "/usr/lib/python2.7/dist-packages/ucsschool/lib/models/base.py", line 674, in modify success = self.modify_without_hooks(lo, validate, move_if_necessary) File "/usr/lib/python2.7/dist-packages/ucsschool/importer/models/import_user.py", line 1092, in modify_without_hooks return super(ImportUser, self).modify_without_hooks(lo, validate, move_if_necessary) File "/usr/lib/python2.7/dist-packages/ucsschool/lib/models/base.py", line 689, in modify_without_hooks self.validate(lo, validate_unlikely_changes=True) File "/usr/lib/python2.7/dist-packages/ucsschool/importer/models/import_user.py", line 1238, in validate import_user=self, MissingMandatoryAttribute: Mandatory attribute u'employeeNumber' does not exist. More than 0 errors. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/ucsschool/importer/mass_import/mass_import.py", line 124, in import_users user_import.delete_users(users_to_delete) # 0% - 10% File "/usr/lib/python2.7/dist-packages/ucsschool/importer/mass_import/user_import.py", line 500, in delete_users self._add_error(exc) File "/usr/lib/python2.7/dist-packages/ucsschool/importer/mass_import/user_import.py", line 781, in _add_error "More than {} errors.".format(self.config["tolerate_errors"]), self.errors TooManyErrors: More than 0 errors. ------ User import statistics ------ Read users from input data: 0 Errors: 2 Entry | User | Error description ------------------------------------- 0 | flodietr | Mandatory attribute u'employeeNumber' does not exist. 0 | <No name> | More than 0 errors. ------ End of user import statistics ------ ------ Writing new users passwords to /data/01/tmp/import/hbg/output/2021/08/user/edoosys/student/11_141429_passwd.csv... ------ Searching for hooks of type 'ResultPyHook' in: /usr/share/ucs-school-import/pyhooks... Found hook classes: Loaded hooks: {}. ------ Importing users done. ------ The user has its employeeNumber: univention-ldapsearch -LLL uid=flodietr dn employeeNumber dn: uid=flodietr,cn=schueler,cn=users,ou=one,dc=schein,dc=qa employeeNumber: 40282097/3333/0174/0bb96e62/7bde I could reproduce it in my testenvironment, if you need the files, they are in the ticket or in my environment.
I haven't looked into the code, but my guess is, that the ImportUser object isn't prepared completely before deletion → ImportUser.prepare_all() → ImportUser.make_udm_properties(). That'd be a bug, because a) udm_properties is incomplete and thus see above, and b) pre-/post-delete hook could depend on it.
So the search filter for the object is okay here, or is this a consequential problem caused by the incomplete prepared object? Getting ImportStudent UDM object by filter: &(!(uid=flodietr))(mailPrimaryAddress=flodietr@one.schein.qa) So a deactivated student is still there so the object could never be found?
(In reply to Christina Scheinig from comment #2) > So the search filter for the object is okay here, or is this a consequential > problem caused by the incomplete prepared object? > > Getting ImportStudent UDM object by filter: > &(!(uid=flodietr))(mailPrimaryAddress=flodietr@one.schein.qa) This is just a test if the email address of the user is already taken by another user. That code path completed successfully and is not part of the problem. The traceback happens later.
(In reply to Daniel Tröder from comment #1) > I haven't looked into the code, but my guess is, that the ImportUser object > isn't prepared completely before deletion → ImportUser.prepare_all() → > ImportUser.make_udm_properties(). > That'd be a bug, because a) udm_properties is incomplete and thus see above, > and b) pre-/post-delete hook could depend on it. So I guess the bug creation is okay. Can I do some analysis to find a solution or a workaround?
This bug had to layers. Both have been addressed: First since the user in this case is not deleted but deactivated a modify operation, instead of a delete operation, is executed. That triggered a validation of the user. We decided to deactivate the validation in that case, since in most cases it should not be necessary to repair the user object if it is going to be deleted at a later date anyways. If a validation is still needed it should be possible to write a pre remove hook that validates the user object without to much effort. Testcase: 253_import-users_deactivate.py Second even if the user object was valid, like in this case, the validation failed because the "udm_properties" attribute was empty. The "udm_properties" attribute is directly filled by the "map" function from the importer reader class (e.g ucs-school-import/modules/ucsschool/importer/reader/csv_reader.py). I found no way to get a list of udm properties which should be loaded, without supplying input data and being reader class implementation independent. I added a new method "get_imported_udm_property_names" to the BaseReader and CsvReader to be able to get a list of udm_properties which will be set during the import. This function is now used to load udm_properties from ldap for users that will be deleted. Any custom reader classes needs to implement this new function if udm_properties should be loaded for users marked for deletion. Testcase: 254_import-users_remove_udm_properties_in_pyhooks.py Sorry, two commits use the ucs 5.0 bug number :( [4.4 9dda1d796] Bug #53649: fix traceback deactivating an importuser [4.4 00ebc3faf] Bug #53649: Load udm_properties for users to be deleted [4.4 782d066e2] Bug #53649: Clear description to test disabled validation [4.4 ce644e809] Bug #53649: test that udm_properties are loaded in remove hooks [4.4 1a133abb7] Bug #53816: changelog [4.4 587ad1188] Bug #53816: yaml
I reverted the wait time for the s4 connector back to 60s seconds. The test "253_import-users_deactivate.py" was flaky [4.4 6f3a673c8] Bug #53649: revert s4 sleep to 60s
OK: Tests fail before update OK: Tests pass after update OK: Reproduce problem OK: Manual import tests work with delete hook OK: Advisory&Changelog
Errata updates for UCS@school 4.4 v9 have been released. https://docs.software-univention.de/changelog-ucsschool-4.4v9-de.html If this error occurs again, please clone this bug.