Univention Bugzilla – Bug 49557
add support for Python based hooks to ucsschool.lib model classes
Last modified: 2021-05-26 11:28:48 CEST
Add support for Python based hooks to the ucsschool.lib model classes (like the ImportUser class has). Python based hooks execute much faster that those executed by subprocess.call(["run-parts", ...]) and allow access to the Python object being created/modified/moved/removed.
I have also had a case were this functionality would have saved a lot of effort. One customer wanted to have an import hook to modify newly created classes. This hook was not supposed to be run when creating a class in the UMC.
PyHooks for all UCS@school objects would normally also be executed when the UMC is used. If there is a use cases for distinguishing between callers, then the hook classes could have a Boolean switch for that. Something like: class SomeHook(GroupHook): run_in_import: bool = True run_in_umc: bool = False def post_create(): ...
An implementation can be found in dtroeder/ucsschool.lib.pyhooks.
Link to backlog: https://taiga.knut.univention.de/project/oschwieg-ucs-5/us/2378
New branch (as old one was not mergeable anymore): dtroeder/ucsschool.lib.pyhooks_4.4 fe49631f7 Bug #49557: sort dirs 8400c5eec Bug #49557: close fd of imported module be8d7e9fc Bug #49557: make debugging of errors during module loading easier 292be08bb Bug #49557: assert may be optimized away bb85c63e1 Bug #49557: improve decoupling by not importing PyHook classes directly 1 4592ca530 Bug #49557: improve decoupling by not importing PyHook classes directly 2 694b3283c Bug #49557: move hook execution code to utils, call_hooks always returns a bool 5714cbb9e Bug #49557: add LDAP connection argument to call_hooks() c652e6e86 Bug #49557: adapt to changed call_hooks() arguments 9a6f088ce Bug #49557: some small fixes f521b2e26 Bug #49557: mypy 39dc2349a Bug #49557: add support for Python based hooks, add hook example 139e4890e Bug #49557: make ExamUserPyHook a subclass of ImportPyHook, so ImportPyHookLoader.init_hook() won't raise a TypeError a56ec98bc Bug #49557: make FormatPyHook a subclass of ImportPyHook, so ImportPyHookLoader.init_hook() won't raise a TypeError a43e8f094 Bug #49557: avoid unnecessary initialization of import configuration system 9a36e5d0d Bug #49557: handle import errors 634084b28 Bug #49557: test for general ucsschool pyhooks 9cde9007b Bug #49557: unrelated linter fixes Hooks will be loaded for a specific class (declared with the hook class' "model" attribute) and their subclasses. The test in 90_ucsschool/83_python_hooks.py now creates a hook for every UCS@school object type and verifies that it ran. TODOs: * Documentation * Migration to feature/kelvin branch * Migration to 5.0 branch
Work on documentation has begun: Repo "doc-common": [master 34bbeef] Bug #49557: add link to UCS@school CLI import docs Repo "ucsschool": [dtroeder/ucsschool.lib.pyhooks_4.4 0962828ae] Bug #49557: add documentation on Python hooks (WIP)
Documentation is done: [dtroeder/ucsschool.lib.pyhooks_4.4] d2ee56b64 Bug #49557: add documentation on Python hooks [dtroeder/ucsschool.lib.pyhooks_4.4] 914eaf846 Bug #49557: fix TypeError The Jenkins job to check for syntax errors does not run on branches, so that'll have to wait until merged to 4.4. But please QA the text in the branch. When looking for good log output for the documentation I stumbled upon this: The TypeError happens only when get_logger() is used directly. As usually get_stream_handler() or get_file_handler() are used, the missing str() hadn't been discovered before.
Doc build for branch: https://jenkins.knut.univention.de:8181/view/Doku/job/BuildDocBookBranchUCSSchool/64/artifact/webroot/
QA → "Reopen" OK Code review 4.4 OK Doku 4.4 OK Tests pass OK Example Hooks work OK in own hook OK self.lo -> reading/writing with self.get_udm_object(self.lo) works OK self.ucr -> reading ucr["ldap/base"] OK self.logger -> logging works as expected OK are called at the places we expect OK test new hooks exam_user_not_ox_user-hook which is derived of ExamUserPyHook which is derived of ImportPyHook still works → Todo recursion handled: I saw some code in kelvin, but not in 4.4. Did you mean to catch something like this? def post_create(self, obj: SchoolClass): self.post_create(obj) OK Doku feature/kelvin OK Code review feature/kelvin OK test functionality in kelvin → Test does not pass. We need to discuss this. 2020-11-22 03:19:22 INFO 83_python_hooks._test_move:612 ** OK 5/162 move() of model 'AnyDHCPService'. ok test_AnyDHCPService_remove (__main__.TestPythonHooks) ... 2020-11-22 03:19:22 DEBUG 83_python_hooks.asyncSetUp:327 setUp() 'test_AnyDHCPService_remove' 2020-11-22 03:19:22 INFO 83_python_hooks.asyncSetUp:328 ################# test_AnyDHCPService_remove (6/162) #################
Code was merged and adapted to Kelvin. Documentation was written for Kelvin. The recursion handling code was back ported from Kelvin to 4.4. A new pytest based test for Kelvin is in underway. The code in 4.4 was squashed, merged and build. Commits and packages in 4.4: [4.4] ad6147bf9 Bug #49557: add support for Python based hooks [4.4] 51f5b51ab Bug #49557: advisories ucs-school-lib (12.2.26) ucs-school-import (17.0.60) ucs-school-ox-support (3.0.1) ucs-test-ucsschool (6.0.225)
I tested the recursion with your suggested code, everything worked wonderfully in 4.4 and kelvin class MyHook(Hook): model = SchoolClass def post_create(self, obj): sc = SchoolClass(name="DEMOSCHOOL-foo", school="DEMOSCHOOL") sc.create(self.lo) # calls pre_create(); create_without_hooks(); post_create() -> recursion merge in 4.4 → OK changelogs → OK yamls → OK > A new pytest based test for Kelvin is in underway. Looking forward to that! When commited, please set the bug to resolve ;)
Kelvin code is finished, incl. a test, and was merged to feature/kelvin: [feature/kelvin] 2ec245366 Bug #49557: add support for Python based hooks
Wonderful! The kelvin-test passes on my VM. I will verify this after the Jenkins tests.
I improved the test to execute hooks for most of the ucsschool.lib classes. Excluded were those that cannot be initialized directly (base classes for other model classes) and commented out are those with bugs in the UDM REST API or the UDM REST API client. While doing this, I stumbled upon an error in the User class. [feature/kelvin] 0e9cf3164 Bug #49557: fix bug in User class [feature/kelvin] 32977bc3c Bug #49557: execute hook test for (almost) all ucsschool.lib classes The test currently covers only the {pre, post}_create hooks. Those for modify, move and delete should be added as well (when there's time).
Created attachment 10734 [details] test cases Thanks a lot for extending the test! The tests pass on my VM, attached you find the test output. Code → OK I appreciate the commented out code/ comments as they indicate that there are still parts to be fixed which are not directly connected to the changes you made in ucs-school-lib/import (→ UDM REST API)
The path for the Python hooks has been changed for 4.4 to /var/lib/ucs-school-lib/hooks. [4.4 950ac5d89] Bug #49557: change path for Python based hooks [4.4 31c9f6b0b] Bug #49557: advisory update The path for Kelvin is kept: /var/lib/ucs-school-import/kelvin-hooks
QA Code -> OK Test still pass -> OK The path for the Python hooks has been changed for 4.4 to /var/lib/ucs-school-lib/hooks. -> OK
Kelvin support and test are now done as well: [feature/kelvin 0b6d23f4e] Bug #49557: change hook path, improve test, bump app version, add changelog TODO: build Docker image
The Docker images was build and push to the registry. docker-upload.software-univention.de/ucsschool-kelvin-rest-api:1.4.2 (f88b30fadc14) The apps ini for version 1.4.2 was adapted to use that image.
QA -> All OK -> Verify Code -> looks good Passes in Jenkins (4.4, kelvin) -> yes build docker-container -> yes build docker-container locally -> OK app-ini -> test passes, in Test Appcenter -> OK documentation for 4.4 & kelvin -> OK
UCS@school Kelvin REST API 1.4.2 has been released. http://appcenter.software-univention.de/univention-repository/4.4/maintained/component/ucsschool-kelvin-rest-api_20210518142444/README_UPDATE_DE If this error occurs again, please clone this bug.
Set to verified for ucsschool errata release.
Errata updates for UCS@school 4.4 v9 have been released. https://docs.software-univention.de/changelog-ucsschool-4.4v9-de.html If this error occurs again, please clone this bug.