Bug 49557 - add support for Python based hooks to ucsschool.lib model classes
add support for Python based hooks to ucsschool.lib model classes
Product: UCS@school
Classification: Unclassified
Component: Ucsschool-lib
UCS@school 4.4
Other Linux
: P5 normal (vote)
: UCS@school 4.4 v9-errata
Assigned To: Daniel Tröder
Tobias Wenzel
Depends on:
Blocks: 49986
  Show dependency treegraph
Reported: 2019-05-27 16:29 CEST by Daniel Tröder
Modified: 2021-05-26 11:28 CEST (History)
4 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:
troeder: Patch_Available+

test cases (10.35 KB, text/plain)
2021-05-20 08:51 CEST, Tobias Wenzel

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2019-05-27 16:29:59 CEST
Add support for Python based hooks to the ucsschool.lib model classes (like the ImportUser class has).

Python based hooks execute much faster that those executed by subprocess.call(["run-parts", ...]) and allow access to the Python object being created/modified/moved/removed.
Comment 1 Lukas Zumvorde univentionstaff 2020-04-07 14:16:08 CEST
I have also had a case were this functionality would have saved a lot of effort.

One customer wanted to have an import hook to modify newly created classes. This hook was not supposed to be run when creating a class in the UMC.
Comment 2 Daniel Tröder univentionstaff 2020-04-07 15:35:44 CEST
PyHooks for all UCS@school objects would normally also be executed when the UMC is used.

If there is a use cases for distinguishing between callers, then the hook classes could have a Boolean switch for that. Something like:

class SomeHook(GroupHook):
   run_in_import: bool = True
   run_in_umc: bool = False

   def post_create():
Comment 3 Daniel Tröder univentionstaff 2020-06-15 16:05:18 CEST
An implementation can be found in dtroeder/ucsschool.lib.pyhooks.
Comment 4 Patrick Ziegler univentionstaff 2021-03-24 11:57:02 CET
Link to backlog: https://taiga.knut.univention.de/project/oschwieg-ucs-5/us/2378
Comment 5 Daniel Tröder univentionstaff 2021-05-11 17:46:08 CEST
New branch (as old one was not mergeable anymore): dtroeder/ucsschool.lib.pyhooks_4.4

fe49631f7 Bug #49557: sort dirs
8400c5eec Bug #49557: close fd of imported module
be8d7e9fc Bug #49557: make debugging of errors during module loading easier
292be08bb Bug #49557: assert may be optimized away
bb85c63e1 Bug #49557: improve decoupling by not importing PyHook classes directly 1
4592ca530 Bug #49557: improve decoupling by not importing PyHook classes directly 2
694b3283c Bug #49557: move hook execution code to utils, call_hooks always returns a bool
5714cbb9e Bug #49557: add LDAP connection argument to call_hooks()
c652e6e86 Bug #49557: adapt to changed call_hooks() arguments
9a6f088ce Bug #49557: some small fixes
f521b2e26 Bug #49557: mypy
39dc2349a Bug #49557: add support for Python based hooks, add hook example
139e4890e Bug #49557: make ExamUserPyHook a subclass of ImportPyHook, so ImportPyHookLoader.init_hook() won't raise a TypeError
a56ec98bc Bug #49557: make FormatPyHook a subclass of ImportPyHook, so ImportPyHookLoader.init_hook() won't raise a TypeError
a43e8f094 Bug #49557: avoid unnecessary initialization of import configuration system
9a36e5d0d Bug #49557: handle import errors
634084b28 Bug #49557: test for general ucsschool pyhooks
9cde9007b Bug #49557: unrelated linter fixes

Hooks will be loaded for a specific class (declared with the hook class' "model" attribute) and their subclasses.

The test in 90_ucsschool/83_python_hooks.py now creates a hook for every UCS@school object type and verifies that it ran.

* Documentation
* Migration to feature/kelvin branch
* Migration to 5.0 branch
Comment 6 Daniel Tröder univentionstaff 2021-05-12 11:30:48 CEST
Work on documentation has begun:

Repo "doc-common":

[master 34bbeef] Bug #49557: add link to UCS@school CLI import docs

Repo "ucsschool":

[dtroeder/ucsschool.lib.pyhooks_4.4 0962828ae] Bug #49557: add documentation on Python hooks (WIP)
Comment 7 Daniel Tröder univentionstaff 2021-05-14 15:42:32 CEST
Documentation is done:

[dtroeder/ucsschool.lib.pyhooks_4.4] d2ee56b64 Bug #49557: add documentation on Python hooks
[dtroeder/ucsschool.lib.pyhooks_4.4] 914eaf846 Bug #49557: fix TypeError

The Jenkins job to check for syntax errors does not run on branches, so that'll have to wait until merged to 4.4.
But please QA the text in the branch.

When looking for good log output for the documentation I stumbled upon this:
The TypeError happens only when get_logger() is used directly. As usually get_stream_handler() or get_file_handler() are used, the missing str() hadn't been discovered before.
Comment 9 Tobias Wenzel univentionstaff 2021-05-18 19:52:02 CEST
QA → "Reopen"

OK Code review 4.4
OK Doku 4.4
OK Tests pass
OK Example Hooks work
OK in own hook 
	OK self.lo -> reading/writing with self.get_udm_object(self.lo) works
	OK self.ucr -> reading ucr["ldap/base"]
	OK self.logger -> logging works as expected
OK are called at the places we expect
OK test new hooks
exam_user_not_ox_user-hook which is derived of ExamUserPyHook which is derived of ImportPyHook still works

→ Todo recursion handled: I saw some code in kelvin, but not in 4.4. Did you mean to catch something like this?

def post_create(self, obj: SchoolClass):

OK Doku feature/kelvin
OK Code review feature/kelvin
OK test functionality in kelvin

→ Test does not pass. We need to discuss this.

2020-11-22 03:19:22 INFO  83_python_hooks._test_move:612  ** OK 5/162 move() of model 'AnyDHCPService'.
test_AnyDHCPService_remove (__main__.TestPythonHooks) ... 2020-11-22 03:19:22 DEBUG 83_python_hooks.asyncSetUp:327  setUp() 'test_AnyDHCPService_remove'
2020-11-22 03:19:22 INFO  83_python_hooks.asyncSetUp:328  #################  test_AnyDHCPService_remove (6/162)  #################
Comment 10 Daniel Tröder univentionstaff 2021-05-19 09:26:43 CEST
Code was merged and adapted to Kelvin.
Documentation was written for Kelvin.
The recursion handling code was back ported from Kelvin to 4.4.
A new pytest based test for Kelvin is in underway.
The code in 4.4 was squashed, merged and build.

Commits and packages in 4.4:

[4.4] ad6147bf9 Bug #49557: add support for Python based hooks
[4.4] 51f5b51ab Bug #49557: advisories

ucs-school-lib (12.2.26)
ucs-school-import (17.0.60)
ucs-school-ox-support (3.0.1)
ucs-test-ucsschool (6.0.225)
Comment 11 Tobias Wenzel univentionstaff 2021-05-19 09:36:32 CEST
I tested the recursion with your suggested code, everything worked wonderfully in 4.4 and kelvin

class MyHook(Hook):
    model = SchoolClass
    def post_create(self, obj):
         sc = SchoolClass(name="DEMOSCHOOL-foo", school="DEMOSCHOOL")
         sc.create(self.lo)   #  calls pre_create(); create_without_hooks(); post_create() -> recursion

merge in 4.4 → OK
changelogs → OK
yamls → OK

> A new pytest based test for Kelvin is in underway.
Looking forward to that!

When commited, please set the bug to resolve ;)
Comment 12 Daniel Tröder univentionstaff 2021-05-19 10:04:07 CEST
Kelvin code is finished, incl. a test, and was merged to feature/kelvin:

[feature/kelvin] 2ec245366 Bug #49557: add support for Python based hooks
Comment 13 Tobias Wenzel univentionstaff 2021-05-19 10:05:43 CEST

The kelvin-test passes on my VM.

I will verify this after the Jenkins tests.
Comment 14 Daniel Tröder univentionstaff 2021-05-19 17:50:06 CEST
I improved the test to execute hooks for most of the ucsschool.lib classes.
Excluded were those that cannot be initialized directly (base classes for other model classes) and commented out are those with bugs in the UDM REST API or the UDM REST API client.

While doing this, I stumbled upon an error in the User class.

[feature/kelvin] 0e9cf3164 Bug #49557: fix bug in User class
[feature/kelvin] 32977bc3c Bug #49557: execute hook test for (almost) all ucsschool.lib classes

The test currently covers only the {pre, post}_create hooks. Those for modify, move and delete should be added as well (when there's time).
Comment 15 Tobias Wenzel univentionstaff 2021-05-20 08:51:20 CEST
Created attachment 10734 [details]
test cases

Thanks a lot for extending the test! The tests pass on my VM, attached you find the test output.
Code → OK

I appreciate the commented out code/ comments as they indicate that there are still parts to be fixed which are not directly connected to the changes you made in ucs-school-lib/import (→ UDM REST API)
Comment 16 Daniel Tröder univentionstaff 2021-05-20 09:53:06 CEST
The path for the Python hooks has been changed for 4.4 to /var/lib/ucs-school-lib/hooks.

[4.4 950ac5d89] Bug #49557: change path for Python based hooks
[4.4 31c9f6b0b] Bug #49557: advisory update

The path for Kelvin is kept: /var/lib/ucs-school-import/kelvin-hooks
Comment 17 Tobias Wenzel univentionstaff 2021-05-21 14:34:37 CEST

Code -> OK
Test still pass -> OK

The path for the Python hooks has been changed for 4.4 to /var/lib/ucs-school-lib/hooks.
-> OK
Comment 18 Daniel Tröder univentionstaff 2021-05-21 18:39:57 CEST
Kelvin support and test are now done as well:

[feature/kelvin 0b6d23f4e] Bug #49557: change hook path, improve test, bump app version, add changelog

TODO: build Docker image
Comment 19 Daniel Tröder univentionstaff 2021-05-22 00:33:27 CEST
The Docker images was build and push to the registry.

docker-upload.software-univention.de/ucsschool-kelvin-rest-api:1.4.2 (f88b30fadc14)

The apps ini for version 1.4.2 was adapted to use that image.
Comment 20 Tobias Wenzel univentionstaff 2021-05-25 11:30:36 CEST
QA -> All OK -> Verify

Code -> looks good
Passes in Jenkins (4.4, kelvin) -> yes
build docker-container -> yes
build docker-container locally -> OK
app-ini -> test passes, in Test Appcenter -> OK
documentation for 4.4 & kelvin -> OK
Comment 21 Tobias Wenzel univentionstaff 2021-05-26 09:25:54 CEST
UCS@school Kelvin REST API 1.4.2 has been released.


If this error occurs again, please clone this bug.
Comment 22 Tobias Wenzel univentionstaff 2021-05-26 09:33:50 CEST
Set to verified for ucsschool errata release.
Comment 23 Ole Schwiegert univentionstaff 2021-05-26 11:28:48 CEST
Errata updates for UCS@school 4.4 v9 have been released.


If this error occurs again, please clone this bug.