Univention Bugzilla – Bug 45888
Add Python 3 support for listener modules
Last modified: 2021-05-25 15:59:13 CEST
Currently, listener modules have to be implented in Python 2. I received the feedback that it would great to support Python 3 with the Listener API. This would make the development of listener plugins for products depending on Python 3 a lot easier.
As Lisener Modules are executed inside the Lisener itself, it currently is not possible to embed two versions of the Python interpreter C code. Earliest time for a switch to Python3 is UCS-5 in 2019-2020 See Bug #44950 on how to use Celery as an intermediate queue, which can then also be handled by Python3.
Please note that using Celery-workers will make the listener module asynchronous. If that is OK, than I strongly suggest to use the new async API introduced in Bug #44786. It will take care of launching jobs etc. It will also take care that even when running in parallel the order of transactions will be preserved per entryUUID. When running without parallelism that is no concern. A customer told me, that if we'd make a listener asynchronous he'd need a tool to determine the replication state.
(In reply to Philipp Hahn from comment #2) > See Bug #44950 on how to use Celery as an intermediate queue, which can then > also be handled by Python3. Unfortunately this ticket seems to be non-public. Is there any chance that the description will be made public?
done
Why not run two listener processes? One serving * Py2 listener modules from /u/l/u-d-l/system and the other * Py3 listener modules from /u/l/u-d-l3/system This would allow us and others to gradually migrate Py2 code to Py3. There would have to be some kind of locking on the transaction queue, to ensure that one listener doesn't get ahead of the other.
(In reply to Daniel Tröder from comment #5) > done Sorry, I will undo this because the comments there contain customer information/names. You could strip down the relevant information and add them here.
> Why not run two listener processes? This though crossed my mind too. Drawback: if we run them in parallel we would probably have two caches. Only the currrent python2 one would continue to do replication.py and eventually feeding another notifier (on a DC backup). a smaller solution might be to make a classic python2 listener wrapper that runs a python3 (or whatever) handler script, passing all data to it.
From the chat: https://stackoverflow.com/questions/3320742/dynamically-loading-two-libpython-versions
Patch for a UDL running python 3 in git:arequate/feature/python3-listener
c4e0ec86c7 | add python 3 compatibility to the async listener code 0a00974c84 | " a1922bc827 | execute UDL modules with Python 3
Listener modules are now executed with Python 3 only. All scripts have been migrated to Python 3. univention-directory-listener (14.0.3-1) 938d4071d437 | Bug #45888: migrate to Python 3 aba2f4bbd91b | fixup! Bug #45888: execute UDL modules with Python 3 a1922bc82723 | Bug #45888: execute UDL modules with Python 3 0a00974c843d | fixup! Bug #45888: add python 3 compatibility c4e0ec86c77d | Bug #45888: add python 3 compatibility
*** Bug 52256 has been marked as a duplicate of this bug. ***
Plus these changes: univention-directory-listener (14.0.0-4) a74b7c7827a1 | Bug #52256: migate UDL listener to Python 3 ucs-test (10.0.1-27) fad52fdf3f1d | Bug #52256 ucs-test: Migrate listener to Python 3
I tried to test with the example code but the listener fails to import it: ======================================================================= 09.01.21 19:21:33.358 LISTENER ( ERROR ) : import of filename=/usr/lib/univention-directory-listener/system/listener_module_template.py failed 09.01.21 19:21:33.358 LISTENER ( ERROR ) : import of filename=/usr/lib/univention-directory-listener/system/listener_module_template.py failed in module_import() ======================================================================= shell# cp /usr/share/doc/univention-directory-listener/examples/listener_module_template.py \ /usr/lib/univention-directory-listener/system shell# cd /usr/lib/univention-directory-listener/system shell# python3 >>> import listener_module_template Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/univention-directory-listener/system/listener_module_template.py", line 34, in <module> class ListenerModuleTemplate(ListenerModuleHandler): File "/usr/lib/python3/dist-packages/univention/listener/handler.py", line 53, in __new__ kls.config = kls._get_configuration() File "/usr/lib/python3/dist-packages/univention/listener/handler.py", line 347, in _get_configuration return cls._configuration_class(**kwargs) File "/usr/lib/python3/dist-packages/univention/listener/handler_configuration.py", line 72, in __init__ for k, v in kwargs.items(): RuntimeError: dictionary changed size during iteration
Fixed in: univention-directory-listener (14.0.4-1) a5e3dc939e2a | Bug #45888: fix dictionary iteration
Verified: * Code review * Functional test * UCS 5 Changelog
UCS 5.0 has been released: https://docs.software-univention.de/release-notes-5.0-0-en.html https://docs.software-univention.de/release-notes-5.0-0-de.html If this error occurs again, please use "Clone This Bug".