Bug 45888 - Add Python 3 support for listener modules
Add Python 3 support for listener modules
Product: UCS
Classification: Unclassified
Component: Listener (univention-directory-listener)
UCS 5.0
Other Linux
: P5 enhancement (vote)
: UCS 5.0
Assigned To: Florian Best
Arvid Requate
: 52256 (view as bug list)
Depends on:
  Show dependency treegraph
Reported: 2017-12-18 16:30 CET by Nico Gulden
Modified: 2021-01-11 16:37 CET (History)
5 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?: Yes
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:
best: Patch_Available+


Note You need to log in before you can comment on or make changes to this bug.
Description Nico Gulden univentionstaff 2017-12-18 16:30:45 CET
Currently, listener modules have to be implented in Python 2. I received the feedback that it would great to support Python 3 with the Listener API. This would make the development of listener plugins for products depending on Python 3 a lot easier.
Comment 2 Philipp Hahn univentionstaff 2017-12-19 09:15:56 CET
As Lisener Modules are executed inside the Lisener itself, it currently is not possible to embed two versions of the Python interpreter C code.
Earliest time for a switch to Python3 is UCS-5 in 2019-2020
See Bug #44950 on how to use Celery as an intermediate queue, which can then also be handled by Python3.
Comment 3 Daniel Tröder univentionstaff 2017-12-19 09:44:30 CET
Please note that using Celery-workers will make the listener module asynchronous.

If that is OK, than I strongly suggest to use the new async API introduced in Bug #44786. It will take care of launching jobs etc.

It will also take care that even when running in parallel the order of transactions will be preserved per entryUUID. When running without parallelism that is no concern.

A customer told me, that if we'd make a listener asynchronous he'd need a tool to determine the replication state.
Comment 4 Niko Wenselowski 2017-12-19 17:33:07 CET
(In reply to Philipp Hahn from comment #2)
> See Bug #44950 on how to use Celery as an intermediate queue, which can then
> also be handled by Python3.
Unfortunately this ticket seems to be non-public. Is there any chance that the description will be made public?
Comment 5 Daniel Tröder univentionstaff 2017-12-20 07:51:39 CET
Comment 6 Daniel Tröder univentionstaff 2017-12-20 17:29:06 CET
Why not run two listener processes? One serving
* Py2 listener modules from /u/l/u-d-l/system and the other
* Py3 listener modules from /u/l/u-d-l3/system

This would allow us and others to gradually migrate Py2 code to Py3.

There would have to be some kind of locking on the transaction queue, to ensure that one listener doesn't get ahead of the other.
Comment 7 Florian Best univentionstaff 2017-12-20 17:36:45 CET
(In reply to Daniel Tröder from comment #5)
> done

Sorry, I will undo this because the comments there contain customer information/names. You could strip down the relevant information and add them here.
Comment 8 Arvid Requate univentionstaff 2017-12-20 21:05:45 CET
> Why not run two listener processes?

This though crossed my mind too. Drawback: if we run them in parallel we would probably have two caches. Only the currrent python2 one would continue to do replication.py and eventually feeding another notifier (on a DC backup).

a smaller solution might be to make a classic python2 listener wrapper that runs a python3 (or whatever) handler script, passing all data to it.
Comment 10 Florian Best univentionstaff 2020-06-29 16:43:21 CEST
Patch for a UDL running python 3 in git:arequate/feature/python3-listener
Comment 11 Arvid Requate univentionstaff 2020-11-03 17:39:43 CET
c4e0ec86c7 | add python 3 compatibility to the async listener code
0a00974c84 | "
a1922bc827 | execute UDL modules with Python 3
Comment 12 Florian Best univentionstaff 2020-12-15 17:12:08 CET
Listener modules are now executed with Python 3 only.
All scripts have been migrated to Python 3.

univention-directory-listener (14.0.3-1)
938d4071d437 | Bug #45888: migrate to Python 3
aba2f4bbd91b | fixup! Bug #45888: execute UDL modules with Python 3
a1922bc82723 | Bug #45888: execute UDL modules with Python 3
0a00974c843d | fixup! Bug #45888: add python 3 compatibility
c4e0ec86c77d | Bug #45888: add python 3 compatibility
Comment 13 Florian Best univentionstaff 2020-12-15 17:16:49 CET
*** Bug 52256 has been marked as a duplicate of this bug. ***
Comment 14 Florian Best univentionstaff 2020-12-15 17:17:07 CET
Plus these changes:

univention-directory-listener (14.0.0-4)
a74b7c7827a1 | Bug #52256: migate UDL listener to Python 3

ucs-test (10.0.1-27)
fad52fdf3f1d | Bug #52256 ucs-test: Migrate listener to Python 3
Comment 15 Arvid Requate univentionstaff 2021-01-09 19:32:36 CET
I tried to test with the example code but the listener fails to import it:

09.01.21 19:21:33.358  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/listener_module_template.py failed
09.01.21 19:21:33.358  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/listener_module_template.py failed in module_import()

shell# cp /usr/share/doc/univention-directory-listener/examples/listener_module_template.py \
shell# cd /usr/lib/univention-directory-listener/system
shell# python3
>>> import listener_module_template
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/univention-directory-listener/system/listener_module_template.py", line 34, in <module>
    class ListenerModuleTemplate(ListenerModuleHandler):
  File "/usr/lib/python3/dist-packages/univention/listener/handler.py", line 53, in __new__
    kls.config = kls._get_configuration()
  File "/usr/lib/python3/dist-packages/univention/listener/handler.py", line 347, in _get_configuration
    return cls._configuration_class(**kwargs)
  File "/usr/lib/python3/dist-packages/univention/listener/handler_configuration.py", line 72, in __init__
    for k, v in kwargs.items():
RuntimeError: dictionary changed size during iteration
Comment 16 Florian Best univentionstaff 2021-01-11 10:48:43 CET
Fixed in:

univention-directory-listener (14.0.4-1)
a5e3dc939e2a | Bug #45888: fix dictionary iteration
Comment 17 Arvid Requate univentionstaff 2021-01-11 16:37:44 CET
* Code review
* Functional test
* UCS 5 Changelog