Univention Bugzilla – Bug 42137
[4.3] ucs-school-import: dependency solver for formatting ImportUser properties
Last modified: 2018-06-04 15:27:43 CEST
Created attachment 7927 [details] dependency solver for formatting ImportUser properties This bug was originally created because currently recordUID and sourceUID are created before any other ImportUser attribute. This prevents the usage of other attributes that must be calculated (are not in the CSV input) in them (for example "<username>"), if not given in the CSV. That is actually a more generell problem: all properties can be created from other properties - including those also being generated. But the order of property creation/formatting is currently fixed, effectively allowing formatting only from data given in the CSV input. The attached completely untested(!) patch implements a recursive dependency solver for all properties.
The dependency solver will parse schemes and check if properties have not been set yet. In such a case it will run the relevant make_*() methods. It takes care not to run into an infinite recursion. If a recursion is detected, the formatting of the respective attribute will be incomplete, but the import will not abort. QA: should an infinite recursion produce an error? IMO it always means a configuration error. Two tests were added: one to check the functionality (using heavily interdependent properties) and one to check the recursion detection. [4.3] 3d6b22ee Bug #42137: add recursive dependency solver for formatting ImportUser properties [4.3] 66d15008 Bug #42137: changelog [4.3] 5d4ae96c Bug #42137: tests for format dependency solver [4.3] ddd227bc Bug #42137: changelog [4.3] 55654f7d Bug #42137: advisory ucs-school-import (16.0.1-13) ucs-test-ucsschool (5.0.2-34)
(In reply to Daniel Tröder from comment #1) > The dependency solver will parse schemes and check if properties have not > been set yet. In such a case it will run the relevant make_*() methods. It > takes care not to run into an infinite recursion. If a recursion is > detected, the formatting of the respective attribute will be incomplete, but > the import will not abort. > > QA: should an infinite recursion produce an error? IMO it always means a > configuration error. I think raising a configuration error is the correct behaviour here. Working with incomplete/missing/wrong information is IMHO more harmful than aborting the import gracefully.
The import will now abort if 1) a recursion is detected when resolving a scheme or 2) no attribute/method could be found to fill a requested attribute in the scheme. A test was adapted to check that if an import was aborted due to recursion/missing/unknown attribute no users were created or deleted. [4.3 dfb4e5c3] Bug #42137: raise exception instead of log warning [4.3 44603ce7] Bug #42137: test no user is created or deleted in case of dependency recursion [4.3 b47e19a7] Bug #42137: changelog [4.3 c9dc9b08] Bug #42137: advisory update ucs-school-import (16.0.1-17) ucs-test-ucsschool (5.0.2-37)
The dependency resolver doesn't work with udm properties. In 232_import-users_format_dependency_solver I changed the line: "config.update_entry('scheme:roomNumber', '<street>-<email>[0:5]')," to: "config.update_entry('scheme:roomNumber', '<phone>-<email>[0:5]')," The test fails: 2018-04-10 14:34:58 ERROR user_import.create_and_modify_users:174 Entry #2: Cannot find data provider for dependency u'phone' for formatting of property u'roomNumber' with scheme u'<phone>-<email>[0:5]'. Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 119, in create_and_modify_users user = self.determine_add_modify_action(imported_user) File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 208, in determine_add_modify_action imported_user.prepare_all(new_user=True) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 412, in prepare_all self.prepare_udm_properties() File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 454, in prepare_udm_properties self.udm_properties[k] = self.format_from_scheme(k, v) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 948, in format_from_scheme self.solve_format_dependencies(prop_name, scheme, **kwargs) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 913, in solve_format_dependencies entry_count=self.entry_count, import_user=self InitialisationError: Cannot find data provider for dependency u'phone' for formatting of property u'roomNumber' with scheme u'<phone>-<email>[0:5]'.
[4.3 b933aa44] Bug #42137: handle dependency resolution for UDM properties [4.3 be33ec0e] Bug #42137: test formatting dependencies with and without UDM properties [4.3 eff4402c] Bug #42137: changelog [4.3 377e7d12] Bug #42137: advisory update ucs-school-import (16.0.1-21) ucs-test-ucsschool (5.0.2-40)
Created attachment 9496 [details] debug output for dependency solver In case you want to see the dependency solver at work, use this patch.
[4.3] 15d0a454 Bug #42137: query UDM for LDAP mapping without using LDAP and don't cache result [4.3] fb1e0450 Bug #42137: changelog [4.3] 713641a6 Bug #42137: advisory update
(In reply to Daniel Tröder from comment #7) > [4.3] 15d0a454 Bug #42137: query UDM for LDAP mapping without using LDAP and > don't cache result > [4.3] fb1e0450 Bug #42137: changelog > [4.3] 713641a6 Bug #42137: advisory update What about extended attributes? They are defined in LDAP. → REOPEN
(In reply to Sönke Schwardt-Krummrich from comment #8) > (In reply to Daniel Tröder from comment #7) > > [4.3] 15d0a454 Bug #42137: query UDM for LDAP mapping without using LDAP and > > don't cache result > > [4.3] fb1e0450 Bug #42137: changelog > > [4.3] 713641a6 Bug #42137: advisory update > > What about extended attributes? They are defined in LDAP. → REOPEN In our tests it work. @Daniel could you add an extended attribute to the test? (Or add a comment if there is already one)
The attributes school, record_uid, source_uid are extended attributes and are resolved. Anyway - I added a fresh extended attributes that is not already mapped by the ucsschool.lib. [4.3] 7c85fb23 Bug #42137: create, format and depend on not-ucsschool-lib-mapped extended attribute [4.3] 835a1406 Bug #42137: changelog ucs-test-ucsschool (5.0.2-44)
Tests look good -> OK jenkins -> OK YAML -> OK Import works -> OK -> Verified
UCS@school 4.3 v3 has been released. https://docs.software-univention.de/changelog-ucsschool-4.3v3-de.html If this error occurs again, please clone this bug.