Univention Bugzilla – Bug 48341
Admin Diary Backend Storage
Last modified: 2019-03-12 13:40:43 CET
This bug is about creating the Python library code and database config that is necessary to *store* Events in the Admin Diary Backend. The idea is that this will be installed on a some system (chosen by customer) in an UCS domain by installing a Debian package, sth. like "univention-admin-diary-storage". Project context: The Admin Diary concept has these players: * Event producer (Client/Source system) * Event consumer (Server/Backend Storage) * Diary GUI (UMC Module) For the Admin Diary the UMC Frontend Module needs to be able to retrieve Events from a database located on some system in the domain (or possibly on two, in case of UCS@school). Let's call this Admin Diary Backend here. We have two options here, how the UMC frontend may access the data in the Admin Diary Backend: Either the UMC frontend directly connect to the Backend Storage Database, or the UMC frontend connects to an Admin Diary backend service (e.g. another UMC module without any GUI). The second approach would have the advantage to abstract from the particular Storage Database component (MariaDB/PostgreSQL). Anyway, in the first step we should create a Python library for the Admin Diary Backend, which can be used to write Events to a Backend Database. These Events originate from arbitrary UCS systems in the domain, will be buffered locally on the source system and transferred to the remote database once the presence of an Admin Diary Backend service is advertised by some UCS system in the domain. We need to provision a table in some database. The design should be so, that customers can choose (by setting some UCR variables) to e.g. use an existing MariaDB Cluster hosted on some (possibly non-UCS) server.
Works reasonably well. You can switch between "mysql" and "postgresql". Currently, this is rather inconvenient: You need to set the value in the UCRV "admin/diary/dbms" BEFORE installation of univention-admin-diary-backend. You should also have univention-mysql or univention-postgresql installed before. Changing it afterwards is not really possible as the tables are created in postinst. You would need to purge the package and reinstall it. Anyway, for a first glimpse at the technology, it works. MySQL should be considerably slower due to multiple tables. (Not really important for assumed admin diary throughput)
Created attachment 9798 [details] 1.diff The patch makes a couple of small suggestions: * Rename "issued" to "timestamp" * Rename "diary_id" to "context_id" * Simplify xgettext translation support * Add german translation strings (plus minor message wording adjustments) * Fix a bug in 60univention-admin-diary-client.inst (my fault) * Make admindiary debugging level adjustable (well, basically..) and raise the default to WARNING * Add support to configure a different admin/diary/dbhost instead of localhost As discussed via phone, we need support for storing event annotations in the database.
I liked all of it. I changed the debug level patch a bit: get_logger() does not have a loglevel argument, instead use this: ucr set admin/diary/logging/backend=DEBUG Event annotations exist: from univention.admindiary.events import SERVER_PASSWORD_CHANGED from univention.admindiary.client import write_event, add_comment context = write_event(SERVER_PASSWORD_CHANGED) add_comment('I felt like doing it now', context)
01_base.27check_logfiles_univention.test Some potentially sensitive log files are world-readable: [2019-01-16 15:44:25.816040] -rw-r--r-- 1 root root 0 Jan 16 15:24 /var/log/univention/admindiary.log
easiest fix seems to be +. /usr/share/univention-lib/all.sh + +create_logfile /var/log/univention/admindiary.log "root:adm" 640 in the postinst
Is this enough when logrotate rotates the file?
(In reply to Arvid Requate from comment #6) > Is this enough when logrotate rotates the file? i am not sure, the test i started after the installation. How do we ensure that logrotate has been executed?
Fixed with ca91e9dab9 6f2fc94761 Log rotate has this in its config: /var/log/univention/admindiary.log { ... create 640 root adm ... }
* Seems missing dependency on postgresql or mysql. * It seems postgres things are done in postinst, they should be in the joinscript to redo them on failure (e.g. because postgres is not installed). (make sure the joinscript fails if the database configuration fails) * The package is not installed by default, should be recommends of the UMC packages? Remark: nach apt-get install --reinstall univention-admin-diary-backend: File: /etc/rsyslog.d/univention-admin-diary-backend.conf Script: /etc/univention/templates/scripts/univention-admin-diary-setup-dbms ALTER ROLE psql:/usr/share/univention-admin-diary/sql/postgresql.sql:13: FEHLER: Relation »events« existiert nicht psql:/usr/share/univention-admin-diary/sql/postgresql.sql:14: FEHLER: Relation »entries« existiert nicht psql:/usr/share/univention-admin-diary/sql/postgresql.sql:15: FEHLER: Relation »entries« existiert nicht psql:/usr/share/univention-admin-diary/sql/postgresql.sql:16: FEHLER: Relation »entries« existiert nicht psql:/usr/share/univention-admin-diary/sql/postgresql.sql:17: FEHLER: Relation »entries« existiert nicht psql:/usr/share/univention-admin-diary/sql/postgresql.sql:18: FEHLER: Relation »entries« existiert nicht psql:/usr/share/univention-admin-diary/sql/postgresql.sql:19: FEHLER: Relation »entries« existiert nicht
Backend storage should be stable now. (In reply to Florian Best from comment #9) > * Seems missing dependency on postgresql or mysql. Dependencies rearranged. postgresql or mysql are now a recommended. > * It seems postgres things are done in postinst, they should be in the > joinscript to redo them on failure (e.g. because postgres is not installed). > (make sure the joinscript fails if the database configuration fails) It is done in postinst and triggered by ucr set admin/diary/dbms=... I think this is okay. > * The package is not installed by default, should be recommends of the UMC > packages? No, it shall be its own App > > Remark: nach apt-get install --reinstall univention-admin-diary-backend: > > File: /etc/rsyslog.d/univention-admin-diary-backend.conf > Script: /etc/univention/templates/scripts/univention-admin-diary-setup-dbms > ALTER ROLE > psql:/usr/share/univention-admin-diary/sql/postgresql.sql:13: FEHLER: > Relation »events« existiert nicht Yes, bug in intermediate version. Fixed
(In reply to Dirk Wiesenthal from comment #10) > (In reply to Florian Best from comment #9) > > * It seems postgres things are done in postinst, they should be in the > > joinscript to redo them on failure (e.g. because postgres is not installed). > > (make sure the joinscript fails if the database configuration fails) > > It is done in postinst and triggered by ucr set admin/diary/dbms=... > I think this is okay. No it's not. The code to setup the database *must* be in the joinscript. In all project where a database is used, users have had problems in the past that could only be fixed by rerunning the setup code. When the instructions were in the postinst, they required the help of the support team, which in the end led to us having to move the code to the joinscript.
I agree with Daniel, the current implementation is a bit magic. Cool magic, but a bit intrasparent :-) We should explicitely call the db setup instead of triggering a UCR script via UCR variable.
Okay. Done in univention-admin-diary 1.0.0-35A~4.4.0.201902271137 ucr set admin/diary/dbms=postgresql Create admin/diary/dbms File: /etc/rsyslog.d/univention-admin-diary-backend.conf Script: /etc/univention/templates/scripts/univention-admin-diary-setup-dbms Okay. Now please run /usr/share/univention-admin-diary/create-database File: /etc/rsyslog.d/univention-admin-diary-client.conf QA: Please verify that the initial setting of the UCRV is done prior to the execution the join script. Otherwise the join script will always fail at the first attempt. ucr set admin/diary/dbms is in python-univention-admin-diary-backend whereas the joinscript is shipped with univention-admin-diary-backend The dependency resolver should do it right, but I have not yet tested it for new installations of the backend.
services/univention-admin-diary/python/admindiary/client.py: > 55 class RsyslogEmitter(object): > 56 »···def __init__(self): > 57 »···»···self.handler = SysLogHandler(address='/dev/log', facility='user') ... > 63 emitter = RsyslogEmitter() You MUST NOT execute such code on module import! 14:54:31 Traceback (most recent call last): 14:54:31 File "/usr/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 529, in import_object 14:54:31 __import__(self.modname) 14:54:31 File "/var/lib/jenkins/workspace/Mitarbeiter/phahn/UCS-API-Doc/py/univention/uvmm/uvmm_ldap.py", line 43, in <module> 14:54:31 import univention.admin.handlers.uvmm.info as uvmm_info 14:54:31 File "/var/lib/jenkins/workspace/Mitarbeiter/phahn/UCS-API-Doc/py/univention/admin/handlers/__init__.py", line 60, in <module> 14:54:31 from univention.admindiary.client import write_event 14:54:31 File "/var/lib/jenkins/workspace/Mitarbeiter/phahn/UCS-API-Doc/py/univention/admindiary/client.py", line 63, in <module> 14:54:31 emitter = RsyslogEmitter() 14:54:31 File "/var/lib/jenkins/workspace/Mitarbeiter/phahn/UCS-API-Doc/py/univention/admindiary/client.py", line 57, in __init__ 14:54:31 self.handler = SysLogHandler(address='/dev/log', facility='user') 14:54:31 File "/usr/lib/python2.7/logging/handlers.py", line 761, in __init__ 14:54:31 self._connect_unixsocket(address) 14:54:31 File "/usr/lib/python2.7/logging/handlers.py", line 789, in _connect_unixsocket 14:54:31 self.socket.connect(address) 14:54:31 File "/usr/lib/python2.7/socket.py", line 228, in meth 14:54:31 return getattr(self._sock,name)(*args) 14:54:31 error: [Errno 2] No such file or directory
(In reply to Philipp Hahn from comment #14) > services/univention-admin-diary/python/admindiary/client.py: > > 55 class RsyslogEmitter(object): > > 56 »···def __init__(self): > > 57 »···»···self.handler = SysLogHandler(address='/dev/log', facility='user') > ... > > 63 emitter = RsyslogEmitter() > > You MUST NOT execute such code on module import! At least try lazy evaluation: class RsyslogEmitter(object): def __init__(self): self.handler = None # type: Optional[logging.Handler] def emit(self, entry): record = logging.LogRecord('diary-rsyslogger', logging.INFO, None, None, 'ADMINDIARY: ' + str(entry), (), None, None) if self.handler is None: self.handler = SysLogHandler(address='/dev/log', facility='user') self.handler.emit(record)
(In reply to Philipp Hahn from comment #15) > (In reply to Philipp Hahn from comment #14) > > services/univention-admin-diary/python/admindiary/client.py: > > > 55 class RsyslogEmitter(object): > > > 56 »···def __init__(self): > > > 57 »···»···self.handler = SysLogHandler(address='/dev/log', facility='user') > > ... > > > 63 emitter = RsyslogEmitter() > > > > You MUST NOT execute such code on module import! > > At least try lazy evaluation: > > class RsyslogEmitter(object): > def __init__(self): > self.handler = None # type: Optional[logging.Handler] > > def emit(self, entry): > record = logging.LogRecord('diary-rsyslogger', logging.INFO, None, None, > 'ADMINDIARY: ' + str(entry), (), None, None) > if self.handler is None: > self.handler = SysLogHandler(address='/dev/log', facility='user') > > self.handler.emit(record) You are right, of course. Did as advised in univention-admin-diary 1.0.0-39A~4.4.0.201903040458
Alternative: import lazy_object_proxy emitter = lazy_object_proxy.Proxy(lambda: RsyslogEmitter()) # type: logging.Handler
(In reply to Daniel Tröder from comment #17) > Alternative: > > import lazy_object_proxy > > emitter = lazy_object_proxy.Proxy(lambda: RsyslogEmitter()) # type: > logging.Handler If possible also in __init__(), to execute as little as possible in the module scope. It is mainly nice to spare yourself and the reader the if xy is <guard value>: xy = ...
Ok, works, I've added the bug number to changelog-4.4-0.xml.
UCS 4.4 has been released: https://docs.software-univention.de/release-notes-4.4-0-en.html https://docs.software-univention.de/release-notes-4.4-0-de.html If this error occurs again, please use "Clone This Bug".