Univention Bugzilla – Bug 45888
Add Python 3 support for listener modules
Last modified: 2021-01-11 16:37:44 CET
Currently, listener modules have to be implented in Python 2. I received the feedback that it would great to support Python 3 with the Listener API. This would make the development of listener plugins for products depending on Python 3 a lot easier.
As Lisener Modules are executed inside the Lisener itself, it currently is not possible to embed two versions of the Python interpreter C code.
Earliest time for a switch to Python3 is UCS-5 in 2019-2020
See Bug #44950 on how to use Celery as an intermediate queue, which can then also be handled by Python3.
Please note that using Celery-workers will make the listener module asynchronous.
If that is OK, than I strongly suggest to use the new async API introduced in Bug #44786. It will take care of launching jobs etc.
It will also take care that even when running in parallel the order of transactions will be preserved per entryUUID. When running without parallelism that is no concern.
A customer told me, that if we'd make a listener asynchronous he'd need a tool to determine the replication state.
(In reply to Philipp Hahn from comment #2)
> See Bug #44950 on how to use Celery as an intermediate queue, which can then
> also be handled by Python3.
Unfortunately this ticket seems to be non-public. Is there any chance that the description will be made public?
Why not run two listener processes? One serving
* Py2 listener modules from /u/l/u-d-l/system and the other
* Py3 listener modules from /u/l/u-d-l3/system
This would allow us and others to gradually migrate Py2 code to Py3.
There would have to be some kind of locking on the transaction queue, to ensure that one listener doesn't get ahead of the other.
(In reply to Daniel Tröder from comment #5)
Sorry, I will undo this because the comments there contain customer information/names. You could strip down the relevant information and add them here.
> Why not run two listener processes?
This though crossed my mind too. Drawback: if we run them in parallel we would probably have two caches. Only the currrent python2 one would continue to do replication.py and eventually feeding another notifier (on a DC backup).
a smaller solution might be to make a classic python2 listener wrapper that runs a python3 (or whatever) handler script, passing all data to it.
From the chat:
Patch for a UDL running python 3 in git:arequate/feature/python3-listener
c4e0ec86c7 | add python 3 compatibility to the async listener code
0a00974c84 | "
a1922bc827 | execute UDL modules with Python 3
Listener modules are now executed with Python 3 only.
All scripts have been migrated to Python 3.
938d4071d437 | Bug #45888: migrate to Python 3
aba2f4bbd91b | fixup! Bug #45888: execute UDL modules with Python 3
a1922bc82723 | Bug #45888: execute UDL modules with Python 3
0a00974c843d | fixup! Bug #45888: add python 3 compatibility
c4e0ec86c77d | Bug #45888: add python 3 compatibility
*** Bug 52256 has been marked as a duplicate of this bug. ***
Plus these changes:
a74b7c7827a1 | Bug #52256: migate UDL listener to Python 3
fad52fdf3f1d | Bug #52256 ucs-test: Migrate listener to Python 3
I tried to test with the example code but the listener fails to import it:
09.01.21 19:21:33.358 LISTENER ( ERROR ) : import of filename=/usr/lib/univention-directory-listener/system/listener_module_template.py failed
09.01.21 19:21:33.358 LISTENER ( ERROR ) : import of filename=/usr/lib/univention-directory-listener/system/listener_module_template.py failed in module_import()
shell# cp /usr/share/doc/univention-directory-listener/examples/listener_module_template.py \
shell# cd /usr/lib/univention-directory-listener/system
>>> import listener_module_template
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/univention-directory-listener/system/listener_module_template.py", line 34, in <module>
File "/usr/lib/python3/dist-packages/univention/listener/handler.py", line 53, in __new__
kls.config = kls._get_configuration()
File "/usr/lib/python3/dist-packages/univention/listener/handler.py", line 347, in _get_configuration
File "/usr/lib/python3/dist-packages/univention/listener/handler_configuration.py", line 72, in __init__
for k, v in kwargs.items():
RuntimeError: dictionary changed size during iteration
a5e3dc939e2a | Bug #45888: fix dictionary iteration
* Code review
* Functional test
* UCS 5 Changelog