Univention Bugzilla – Bug 52853
Migrate ou.create_post hooks to school lib
Last modified: 2023-05-26 15:38:47 CEST
As preparation for the creation of schools in kelvin, the shell hooks for ou.create_post must be migrated to the ucs school lib. Those include e.g. the creation of the marketplace. This has to be implemented in 4.4 and kelvin
Hooks to be migrated: 40dhcp_dns_marktplatz_ucsschoolrole Implemented in c8881a902 60schoolexam-master Implemented in c8881a902 53importgroup_import_api_school_update Implemented in c8881a902 50update_ucr_policy → Will not be migrated to python because the package ucs-school-ucc-integration will be out of maintenance next month. 10self_service_whitelist Will be implemented as a listener, because the hook sets an ucr-v. In the other hooks, UCR is only used to get. When implementing this in feature/kelvin these have to be available in the container. [twenzel/4.4/52853_migrate_shell_hooks] c8881a902 Bug #52853: migrate 60schoolexam-master to school lib [twenzel/4.4/52853_migrate_shell_hooks] 1028ec477 Bug #52853: migrate 40dhcp_dns_marktplatz_ucsschoolrole to school lib
53importgroup_import_api_school_update → I think the http API can't be updated like this, a listener cf. 10self_service_whitelist might be a better solution. I started migrating the school.py (& the missing share.py). Please check the marked code locations share.py. [twenzel/kelvin/52853_migrate_shell_hooks] 69f185315 Bug #52853: migrate school py [twenzel/kelvin/52853_migrate_shell_hooks] 85eaa5e03 Bug #52853: migrate share py
Shell hooks (10self_service_whitelite to listener, update_http_api) are moved into listeners. d1278879c was just a try to move into a python hook, which was then better implemented as a listener in b31a102bf. [twenzel/4.4/52853_migrate_shell_hooks] b31a102bf Bug #52853: move 10self_service_whitelite to listener [twenzel/4.4/52853_migrate_shell_hooks] d1278879c Bug #52853: trying to move to python hook [twenzel/4.4/52853_migrate_shell_hooks] e744297a2 Bug #52853: moving update_http_api into school_creation.py listener [twenzel/4.4/52853_migrate_shell_hooks] 11c37755b Bug #52853: 10self_service_whitelist hook migrated to listener module in python
I think that's it. I left 85eaa5e03 Bug #52853: migrate share py in the branch for now, but I suggest dropping it and picking the one in the dtroeder/pre-commit branch instead to keep things consistent.
I added some more tests: [twenzel/4.4/52853_migrate_shell_hooks] 4248bc244 Bug #52853: add ucs tests for migrated hooks and dropped the part with the share.py migration 85eaa5e03 since this is done in Bug 52817 (...and saved changes locally just in case)
The package ucs-school-import cannot be installed/updated due to the following error: dh_install: Cannot find (any matches for) "usr/share/ucs-school-import-http-api/hooks/ou_create_post.d/*" (tried in "." and "debian/tmp") dh_install: ucs-school-import-http-api missing files: usr/share/ucs-school-import-http-api/hooks/ou_create_post.d/* dh_install: missing files, aborting debian/rules:86: die Regel für Ziel „binary“ scheiterte make: *** [binary] Fehler 255 dpkg-buildpackage: Fehler: Fehler-Exitstatus von debian/rules binary war 2 Since you removed the folder from the package, you have to adapt the install script.
01_test_school_removal_ucr.py needs to be executable and it also fails on my VM with: Traceback (most recent call last): File "01_test_school_removal_ucr.py", line 42, in <module> test_school_creation() File "01_test_school_removal_ucr.py", line 27, in test_school_creation assert value in ucr_value AssertionError 01_test_school_creation_http_api_update.py fails on my VM with: Traceback (most recent call last): File "01_test_school_creation_http_api_update.py", line 28, in <module> test_school_creation() File "01_test_school_creation_http_api_update.py", line 24, in test_school_creation assert name in School.objects.all().values_list("name", flat=True) AssertionError Starting 1 ucs-test at 2021-03-22 08:51:43 to /dev/null UCS 4.4-7-e924 ucs-test 9.0.7-25A~4.4.0.202103021459 Modify ucrv on school creation/deletion........................................................................................................ Test failed 01_test_school_creation_ucr.py fails on my VM with: Traceback (most recent call last): File "01_test_school_creation_ucr.py", line 33, in <module> test_school_creation() File "01_test_school_creation_ucr.py", line 29, in test_school_creation assert value in ucr_value AssertionError
Thanks for the QA (!) comment 6 → was fixed in [twenzel/4.4/52853_migrate_shell_hooks] 3644be6ea Bug #52853: add hook dirs comment 7 → was fixed in [twenzel/4.4/52853_migrate_shell_hooks] db861d4fc Bug #52853: switched ucr to lib_ucr for the tests more documentation was added in [twenzel/4.4/52853_migrate_shell_hooks] ae91d0255 Bug #52853: more docstrings and logging [twenzel/4.4/52853_migrate_shell_hooks] 74c6839da Bug #52853: clarify documentation, commented out http_api upon removal of school [twenzel/4.4/52853_migrate_shell_hooks] 599b54c72 Bug #52853: more documentation, permissions and renaming to add_or_remove_ucrv_value As discussed, we have to split up the listener code because the code originated from ucs-school-self-service and ucs-school-import. This was done in [twenzel/4.4/52853_migrate_shell_hooks] 89152749b Bug #52853: split listener in packages
Before I forget → 50update_ucr_policy is still in the ucc-package, because it was not migrated (because it will not be supported anymore next month). We will have to remove this code.
Migrated changes to kelvin feature branch [twenzel/kelvin/52853_migrate_shell_hooks] aaaa37ab4 Bug #52853: add documentation & await and corrected a spelling mistake [twenzel/4.4/52853_migrate_shell_hooks] 3c3805641 fixup! Bug #52853: more docstrings and logging
cherry picked & squashed changes for kelvin with [feature/kelvin] 532f27a90 Bug #52853: migrate ou shell hooks
The newly added tests are still failing and we have to investigate and resolve this before the new schools/POST resource can be released. As a note: After all is RESOLVED here we want to rebase feature/kelvin once to clean up the branch.
Tests are green. The Kelvin API now depends [feature/kelvin] 2465f8c61 Bug #52853: fix hooks and tests [feature/kelvin] 890983a3e Bug #52853: create ou tests [feature/kelvin] 2465f8c61 Bug #52853: fix hooks and tests A new Docker image was built.
I added the kelvin documentation, a missing test for kelvin and migrated improvements from feature/kelvin to twenzel/4.4/52853_migrate_shell_hooks [twenzel/4.4/52853_migrate_shell_hooks] 6c0903652 Bug #52853: migrate changes in kelvin branch to 4.4 [feature/kelvin] 20f04d258 Bug #52853: add documentation [feature/kelvin] 28c7f00d2 Bug #52853: fix set_ucsschool_role_for_dc [feature/kelvin] e24221484 Bug #52853: add test for ucsschool-roles
I squashed the commits in 4.4 to make the QA easier & already rebased the feature branch to 4.4 [twenzel/4.4/52853_migrate_shell_hooks] e1865528c Bug #52853: replace len squashme [twenzel/4.4/52853_migrate_shell_hooks] c772373d9 Bug #52853: ucs tests [twenzel/4.4/52853_migrate_shell_hooks] 865e43ebd Bug #52853: 10self_service_whitelist & 70http_api_school_create hook migrated to listener modules in python [twenzel/4.4/52853_migrate_shell_hooks] 1921f9dcf Bug #52853: migrate 40dhcp_dns_marktplatz_ucsschoolrole, 53importgroup_create & 60schoolexam-master to school lib
Hooks that have to be replaced: * 10self_service_whitelist * 40dhcp_dns_marktplatz_ucsschoolrole - 40dhcpsearchbase_create - 45dhcpdns_create - 52marktplatz_create - 70server_uscschoolRole * 53importgroup_import_api_school_update - 53importgroup_create - 70http_api_school_create * 60schoolexam-master * OK: 10self_service_whitelist school_creation_self_service.py * REOPEN: The Python logging API uses ("%s", ..) for placing values in the log, evaluating them only if the line actually gets logged. Do not use logger.info("... '{}'".format(arg)), instead write logger.info("... %r", arg) * REOPEN: the Python module is not available at installation time: -------------------------------------------------------------------------- Calling joinscript 67ucs-school-selfservice-support.inst ... 2021-04-19 14:15:30.988871773+02:00 (in joinscript_init) Traceback (most recent call last): File "/usr/share/ucs-school-lib/modify_ucr_list", line 36, in <module> from ucsschool.lib.models.utils import add_or_remove_ucrv_value ImportError: cannot import name add_or_remove_ucrv_value [..] Trigger für python-support (1.0.15.18.201403132013) werden verarbeitet ... -------------------------------------------------------------------------- * REOPEN: the join script does not exist on error (||die), so it won't be automatically reexecuted after an error. * OK: 40dhcpsearchbase_create is replaced by School.create_dhcp_search_base() * REOPEN: Upon "CreateError" the code continues. The original code quit the function upon error. Anyway: the new code would crash in the next code block. * OK: 45dhcpdns_create is replaced by School.create_dhcp_dns_policy() * OK: 52marktplatz_create is replaced by School.create_market_place_share() * REOPEN: "if MarketplaceShare.get_all()" and "MarketplaceShare().exists()" do the same and are thus duplicate code. * OK: 70server_uscschoolRole is replaced by School.set_ucsschool_role_for_dc() * OK: 53importgroup_create is replaced by School create_import_group() * REOPEN: is the "group.save()" in line 961 really required, or is the one 3 lines below enough? * OK: 70http_api_school_create is replaced by the listener module school_creation_http_api_update.py * REOPEN: remove() is not implemented (but marked as "# TODO") in the listener module school_creation_http_api_update.py. Simply call "self._update_http_api()". HINT: exec_cmd() returns a 3-tuple. Instead of receiving it in a single variable and accessing it by index, receive it in a 3-tuple and use the individual variables: returncode, stdout, stderr = exec_cmd(...) * OK: 60schoolexam-master is replaced by School.create_exam_group() * REOPEN: an unused import was added to the otherwise unmodified ucs-school-import script * REOPEN: do not import the UMC in ucs-school-lib/python/models/school.py! It will massively raise the import time. Use "from univention.admin.uldap import getAdminConnection". * REOPEN: the additional "import univention.config_registry" in ucs-school-lib/python/models/utils.py is not necessary. Just append "handler_set" 2 lines below. * OK: old hook files have are removed by package update * OK: listener modules are installed and initialized * OK: manual test: the script /usr/.../create_ou created an OU with all expected objects and the hooks were executed
First part of corrections: [twenzel/4.4/52853_migrate_shell_hooks] f4b82bb27 Bug #52853: qa remarks (part 1) still missing/ unfinished: REOPEN: The Python logging API uses ("%s", ..) for placing values in the log, evaluating them only if the line actually gets logged. Do not use logger.info("... '{}'".format(arg)), instead write logger.info("... %r", arg) REOPEN: the Python module is not available at installation time REOPEN: the join script does not exist on error (||die), so it won't be automatically reexecuted after an error.
the remaining parts were commited in [twenzel/4.4/52853_migrate_shell_hooks] 3774b3a2a Bug #52853: qa remarks (part 2)
As discussed, the changes were merged, build, changelogs & yamls were added. [4.4] ca1862712 Bug #52853: update yamls [4.4] 65c8abb73 Bug #52853: add changelogs and advisories [4.4] 8b7819256 Bug #52853: Merge branch 'twenzel/4.4/52853_migrate_shell_hooks' into 4.4 [4.4] eb81f2057 Bug #52853: migrate ou_create/post hooks to ucs-school-lib Package: ucs-school-lib Version: 12.2.18A~4.4.0.202104191927 Branch: ucs_4.4-0 Scope: ucs-school-4.4 Package: ucs-school-import Version: 17.0.54A~4.4.0.202104191929 Branch: ucs_4.4-0 Scope: ucs-school-4.4 Package: ucs-school-selfservice-support Version: 3.0.0-3A~4.4.0.202104191933 Branch: ucs_4.4-0 Scope: ucs-school-4.4 Package: ucs-school-umc-exam Version: 9.0.1-51A~4.4.0.202104191935 Branch: ucs_4.4-0 Scope: ucs-school-4.4 Package: ucs-test-ucsschool Version: 6.0.209A~4.4.0.202104191937 Branch: ucs_4.4-0 Scope: ucs-school-4.4 Projekt Publish U@S 4.4 scope to testing has been started
I added a commit and rebuild, update-yaml to handle the Debian Python module build: [4.4] 82718e44e Bug #52853: update timestamps for update-python-modules [4.4] 5689bb61c Bug #52853: advisories: bump and wording
Lots of tests are failing: https://jenkins.knut.univention.de:8181/job/UCSschool-4.4/job/Install%20Singleserver/lastBuild/#showFailuresLink
Thanks for fixing some of the failing tests in [4.4] ae2992131 Bug #52853: fix imports There is still one problem which occurs on replication nodes: (2021-04-20 02:50:55.590786) self.call_hooks("post", "create") (2021-04-20 02:50:55.590837) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/school.py", line 751, in call_hooks (2021-04-20 02:50:55.590948) lo, pos = getAdminConnection() (2021-04-20 02:50:55.591004) File "/usr/lib/python2.7/dist-packages/univention/admin/uldap.py", line 166, in We can't use self.get_machine_connection() because we need this access level, e.g. for setting the roles of computers/domaincontroller_master
The code for shell hooks was only executed on primary nodes and backups. → as discussed this code is skipped here (like before) Package: ucs-school-lib Version: 12.2.21A~4.4.0.202104201631 Branch: ucs_4.4-0 Scope: ucs-school-4.4 Projekt Publish U@S 4.4 scope to testing was started too.
moved hook code to create_without_hooks Package: ucs-school-lib Version: 12.2.22A~4.4.0.202104210959 Branch: ucs_4.4-0 Scope: ucs-school-4.4 multiple tests were fixed with Package: ucs-test-ucsschool Version: 6.0.213A~4.4.0.202104211004 Branch: ucs_4.4-0 Scope: ucs-school-4.4
Released as erratum for UCS@school 4.4 v9. If you wish to reopen this bug, please clone it instead.
Sorry - wrong bug. This package has NOT been released.
OK: code merge OK: automated tests and Jenkins
Errata updates for UCS@school 4.4 v9 have been released. https://docs.software-univention.de/changelog-ucsschool-4.4v9-de.html If this error occurs again, please clone this bug.