Univention Bugzilla – Bug 47315
Support Listener integration: Converter
Last modified: 2018-08-22 14:26:21 CEST
We need a tool to translate the files created by an App's listener to something that Apps can process. The object should be read with the permissions of the Docker App account.
If the code runs inside the listener, why not just convert new and old, instead of reading from LDAP?
(In reply to Daniel Tröder from comment #1) > If the code runs inside the listener, why not just convert new and old, > instead of reading from LDAP? The object is re-read intentionally to honor ACLs that may be given to the App's machine account. Fixed in univention-appcenter 7.0.2-7A~4.3.0.201807080001 One converter service is started for every App that uses the listener integration. It writes { 'id': entry_uuid, 'object': attrs, 'dn': dn, 'udm_object_type': udm_type, } into files in /var/lib/univention-appcenter/apps/$app/data/listener and runs the trigger. The converter logs into /var/log/univention/listener_modules/$app.log so that the whole listener integration logs to one place. Logging includes the output of the trigger script. The service management is done by systemd so the converter runs after a reboot.
* Timeout -> 5s (instead of 1) * univention-appcenter-listener-converter should have one mandatory parameter, the app id, please remove the "for app in apps:" stuff * move ucr_load(), reload_package_manager(), Apps().find(app)... to outside of while loop, only find_and_convert_files and trigger in the loop, if ucr etc changes, the daemon has to be restarted
(In reply to Felix Botner from comment #3) > if ucr etc changes, the daemon has to be restarted How will an ISVs app developer know that the daemon should be restarted? I think it should be 1. an automatic part of the update procedure. All app updates should trigger a daemon reload. 2. an automatic part of app init-scripts: "service docker-app-dudle restart" should also restart this daemon.
If I see this correctly, the current implementation uses systemd to stop the daemon with SIGTERM. As the signal is not handled, that will result in created and started trigger scripts but not deleted job files. The next time the daemon runs, it will trigger the same action again.
Please stop the listener/convert in the first step during the removal of an app. i removed my test app and during the remove the followoing json has been created { "command": "modify", "dn": "cn=Computers,cn=groups,dc=four,dc=three", "entry_uuid": "6adc6952-bbd2-1037-95b0-3d87321d6737", "object_type": "groups/group" }
(In reply to Felix Botner from comment #6) > Please stop the listener/convert in the first step during the removal of an > app. i removed my test app and during the remove the followoing json has > been created > > { > "command": "modify", > "dn": "cn=Computers,cn=groups,dc=four,dc=three", > "entry_uuid": "6adc6952-bbd2-1037-95b0-3d87321d6737", > "object_type": "groups/group" > } also get this traceback on removal 2018-07-24 00:57:02 ERROR Error in the Listener converter Traceback (most recent call last): File "/usr/share/univention-appcenter-listener-converter", line 191, in main find_and_convert_files(app) File "/usr/share/univention-appcenter-listener-converter", line 161, in find_and_convert_files if convert(app, dumped, filename): File "/usr/share/univention-appcenter-listener-converter", line 116, in convert objs = search_objects(udm_type, lo, pos, entryUUID=entry_uuid) File "/usr/lib/pymodules/python2.7/univention/appcenter/udm.py", line 124, in search_objects objs = module.lookup(None, lo, filter_str, base=base) File "/usr/lib/pymodules/python2.7/univention/admin/handlers/groups/group.py", line 1063, in lookup for dn, attrs in lo.search(unicode(filter), base, scope, [], unique, required, timeout, sizelimit): File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 710, in search raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) ldapError: Insufficient access
check in the univention-appcenter-listener-converter if the file still exists after the trigger has been executed, the trigger may remove the file and the converter can't handle this currently File "/usr/share/univention-appcenter-listener-converter", line 192, in main run_trigger(app) File "/usr/share/univention-appcenter-listener-converter", line 104, in run_trigger os.unlink(fname) OSError: [Errno 2] No such file or directory: '/var/lib/univention-appcenter/apps/listener-test-app/data/listener/2018-07-24-01-05-08-689325.json'
We should use the host's ldap/server/name setting as ldap server to look for objects in the converter (ucs@school!).
Fixed in univention-appcenter 7.0.2-15A~4.3.0.201807271137 Set default sleep duation to 5s Not updating the App in the loop Restart the listener during the update Now using ldap/server/name As for files that are "out of sync", i.e. created while the App was not installed or the service stopped while the trigger processed the files: I do not see the need to make the service robust in this sense. Instead, the trigger scripts have to be robust and be able to process files twice, etc. The trigger _needs_ to be able to handle a listener resync, for example. One easy way to make the trigger better would be to delete the processed file on its own.
(1) created a ucs-test 73_app_listener_integration, @Dirk can you have a look (2) listener is not restarted during upgrade (3) log from the listener module should go only into the listener.log not the listener_modules log ( Wrote /var/lib/univention-app... Modification of...) (4) i would prefer only one line of log in both log filesby defauls: listener log: listener-test-app: Creation of uid=aa1,dc=four,dc=three (id: 5b1cc10e-2a15-1038-8b0d-4dc3e8117431, file: 2018-08-01-22-29-38-212698.json) listener_modules: Modify/Create of uid=aa1,dc=four,dc=three (id: ..., file: 2018-08-01-22-29-38-236071.json) no Converting and Deleting unused in the default log level (5) please rename the listener_modules/listener-test-app.log to /var/log/univention/appcenter-listener-converter-APP_ID.log, and add a logrotate rule (6) univention-appcenter-listener-converter: check call_process trigger call for non-docker apps (success?) (7) what is this (python/appcenter/udm.py) lo = base_access(host=host, port=port, base=base, binddn=userdn, bindpw=...) lo = access(lo=lo) why not just univention.admin.uldap, as far as i can see this admin.uldap uses univention.uldap of lo is not given (8) Am I right assuming we do not need to try/except for LDAP exceptions in the univention-appcenter-listener-converter main loop because all the error handling is done in univention.admin.uldap?
(In reply to Felix Botner from comment #11) > (5) > please rename the listener_modules/listener-test-app.log to > /var/log/univention/appcenter-listener-converter-APP_ID.log, and add a > logrotate rule The filename is created by default from the listeners 'name' property. Either it must be "appcenter-listener-converter-APP_ID", or you can change the call in init: def __init__(self, *args, **kwargs): super(AppListener, self).__init__(self, *args, **kwargs) self.logger = get_logger( self.config.get_name(), '/var/log/univention/listener_modules/appcenter-listener-converter-{}'.format(APP_ID) )
Fixed in univention-appcenter 7.0.2-19A~4.3.0.201808171759 Listener is still not restarted during upgrade. Why would this be important? The only thing I can think of would be a change in ListenerUDMModules. I say this is rather unlikely. But even then, the Listener needs to be restarted and resynced. I will set the Bug to RESOLVED, but we can discuss this further. I have improved the log messages. (But currently, the log level cannot be changed) The returncode of the trigger is now evaluated even for Non-Docker Apps JSON file creation is not atomic by creating a JSON file in /tmp/ and then moving it The converter now exits in case of an error. But the systemd-configuration will restart the service after 5 seconds. This should handle changes in the credentials, LDAP_SERVER_DOWN, etc. where you want a fresh start. For other errors (JSON parsing, Container stopped), the program exits, too. But effectively, due to the auto-restart, this is not really different from catching them all and going on in the loop. You may easily test this with touch /var/lib/univention-appcenter/listener/listener-test-app/empty.json and tail -f /var/log/univention/listener_modules/listener-test-app.log
OK - univention-appcenter OK - yaml
<http://errata.software-univention.de/ucs/4.3/211.html>