Bug 53726 - Do not restart UDL during package updates
Do not restart UDL during package updates
Status: NEW
Product: UCS
Classification: Unclassified
Component: Listener (univention-directory-listener)
UCS 4.4
Other Linux
: P5 normal (vote)
: ---
Assigned To: UCS maintainers
UCS maintainers
https://jenkins.knut.univention.de:81...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2021-08-31 13:09 CEST by Philipp Hahn
Modified: 2021-09-08 10:38 CEST (History)
2 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 7: Crash: Bug causes crash or data loss
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.240
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2021-08-31 13:09:12 CEST
<https://jenkins.knut.univention.de:8181/job/UCS-4.4/job/UCS-4.4-8/job/S4Connector/ws/cfg/update-master-s4connector/test/listener.log/*view*/>:
…
31.08.21 00:18:42.350  LISTENER    ( WARN    ) : received signal 15
31.08.21 00:18:48.055  LISTENER    ( WARN    ) : Notifier/LDAP server is master071c.autotest071c.local:7389
31.08.21 00:18:48.055  LDAP        ( PROCESS ) : connecting to ldap://master071c.autotest071c.local:7389
31.08.21 00:18:51.876  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/samba4-idmap.py failed
Traceback (most recent call last):
  File "/usr/lib/univention-directory-listener/system/samba4-idmap.py", line 46, in <module>
    from samba.provision import setup_idmapdb
  File "/usr/lib/python2.7/dist-packages/samba/provision/__init__.py", line 118, in <module>
    from samba.provision.sambadns import (
  File "/usr/lib/python2.7/dist-packages/samba/provision/sambadns.py", line 35, in <module>
    from samba.dcerpc import dnsp, misc, security
ImportError: /usr/lib/python2.7/dist-packages/samba/dcerpc/dnsp.x86_64-linux-gnu.so: symbol ndr_table_dnsp, version NDR_NBT_0.0.1 not defined in file libndr-nbt.so.0 with link time reference
31.08.21 00:18:51.877  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/samba4-idmap.py failed in module_import()
…
UNIVENTION_DEBUG_BEGIN  : uldap.__open host=master071c.autotest071c.local port=7389 base=dc=autotest071c,dc=local
UNIVENTION_DEBUG_END    : uldap.__open host=master071c.autotest071c.local port=7389 base=dc=autotest071c,dc=local
31.08.21 00:19:23.686  LISTENER    ( WARN    ) : received signal 15

<https://jenkins.knut.univention.de:8181/job/UCS-4.4/job/UCS-4.4-8/job/S4Connector/ws/cfg/update-master-s4connector/test/journalctl.log/*view*/>
…
Aug 31 00:18:42 master071c systemd[1]: Reloading.
Aug 31 00:18:42 master071c systemd[1]: apt-daily.timer: Adding 3h 21min 10.487255s random time.
Aug 31 00:18:42 master071c systemd[1]: Stopping LSB: Univention Directory Listener Daemon...
Aug 31 00:18:47 master071c univention-directory-listener[17246]: Stopping Univention Directory Listener Daemon: univention-directory-listener.

<https://jenkins.knut.univention.de:8181/job/UCS-4.4/job/UCS-4.4-8/job/S4Connector/ws/cfg/update-master-s4connector/test/dpkg.log/*view*/>:
…
2021-08-31 00:18:50 upgrade python-samba:amd64 2:4.10.1-1A~4.4.0.202004212102 2:4.10.1-1A~4.4.0.202006301635
2021-08-31 00:18:50 status half-configured python-samba:amd64 2:4.10.1-1A~4.4.0.202004212102
2021-08-31 00:18:50 status unpacked python-samba:amd64 2:4.10.1-1A~4.4.0.202004212102
2021-08-31 00:18:50 status half-installed python-samba:amd64 2:4.10.1-1A~4.4.0.202004212102
2021-08-31 00:18:51 status unpacked python-samba:amd64 2:4.10.1-1A~4.4.0.202006301635
…
2021-08-31 00:18:42 upgrade univention-admin-diary-client:all 1.0.0-50A~4.4.0.202002181208 1.0.0-51A~4.4.0.202004211541
2021-08-31 00:18:42 status half-configured univention-admin-diary-client:all 1.0.0-50A~4.4.0.202002181208
2021-08-31 00:18:42 status unpacked univention-admin-diary-client:all 1.0.0-50A~4.4.0.202002181208
2021-08-31 00:18:42 status half-installed univention-admin-diary-client:all 1.0.0-50A~4.4.0.202002181208
2021-08-31 00:18:47 status unpacked univention-admin-diary-client:all 1.0.0-51A~4.4.0.202004211541
…
2021-08-31 00:18:53 upgrade samba-libs:amd64 2:4.10.1-1A~4.4.0.202004212102 2:4.10.1-1A~4.4.0.202006301635
2021-08-31 00:18:53 status half-configured samba-libs:amd64 2:4.10.1-1A~4.4.0.202004212102
2021-08-31 00:18:53 status unpacked samba-libs:amd64 2:4.10.1-1A~4.4.0.202004212102
2021-08-31 00:18:53 status half-installed samba-libs:amd64 2:4.10.1-1A~4.4.0.202004212102
2021-08-31 00:18:54 status unpacked samba-libs:amd64 2:4.10.1-1A~4.4.0.202006301635

<file:/var/lib/dpkg/info/univention-admin-diary-client.postrm>
> if [ -x /etc/init.d/univention-directory-listener ]; then
>         invoke-rc.d univention-directory-listener crestart
> fi

UDL is restarted unconditionally during the upgrade.
During the restart `python-samba` is only unpacked, but not yet configured.

We should not restart UDL during updates.




# find /usr/lib/ -name \*.so\* -type f -exec grep -l ndr_table_dnsp {} +
/usr/lib/x86_64-linux-gnu/libndr-nbt.so.0.0.1
/usr/lib/x86_64-linux-gnu/samba/libndr-samba4.so.0
/usr/lib/python2.7/dist-packages/samba/dcerpc/dnsp.x86_64-linux-gnu.so

# find /usr/lib/x86_64-linux-gnu/libndr-nbt.so.0.0.1 /usr/lib/x86_64-linux-gnu/samba/libndr-samba4.so.0 /usr/lib/python2.7/dist-packages/samba/dcerpc/dnsp.x86_64-linux-gnu.so -exec objdump -T {} \; | grep ndr_table_dnsp
0000000000223ee0 g    DO .data.rel.ro   0000000000000048  NDR_NBT_0.0.1 ndr_table_dnsp
0000000000000000      DO *UND*  0000000000000000  NDR_NBT_0.0.1 ndr_table_dnsp
0000000000000000      DO *UND*  0000000000000000  NDR_NBT_0.0.1 ndr_table_dnsp

install ok installed	samba	2:4.10.18-1A~4.4.0.202104270925
install ok installed	samba-common	2:4.10.18-1A~4.4.0.202104270925
install ok installed	samba-common-bin	2:4.10.18-1A~4.4.0.202104270925
install ok installed	samba-dsdb-modules	2:4.10.18-1A~4.4.0.202104270925
install ok installed	samba-libs:amd64	2:4.10.18-1A~4.4.0.202104270925
install ok installed	samba-vfs-modules	2:4.10.18-1A~4.4.0.202104270925
install ok installed	python-samba	2:4.10.18-1A~4.4.0.202104270925
Comment 1 Florian Best univentionstaff 2021-08-31 13:16:37 CEST
(In reply to Philipp Hahn from comment #0)
> We should not restart UDL during updates.
We have to be able to reload listener modules during package upgrades because of e.g. internal python API changes.

Maybe a dpkg-trigger to reload the UDL after all packages have been upgraded would be a preferable solution.
Comment 2 Philipp Hahn univentionstaff 2021-08-31 13:49:34 CEST
(In reply to Florian Best from comment #1)
> (In reply to Philipp Hahn from comment #0)
> > We should not restart UDL during updates.
> We have to be able to reload listener modules during package upgrades
> because of e.g. internal python API changes.
> 
> Maybe a dpkg-trigger to reload the UDL after all packages have been upgraded
> would be a preferable solution.

This was also my first idea to restart UDL automatically via an dpkg trigger whenever /usr/lib/univention-directory-listener/system/ is updated. The problem is that the fan-out of UDL is too height: It depends on the UDL modules installed, which themselves depends on lots on other packages. They all must be ready-for-use (configured) for no Traceback to happen. So basically UDL should only be restarted WHEN they are all configured, which trivially is so at the end of the upgrade. Otherwise we would need to track the dependencies ourselves, which looks undoable.

PoC:

management/univention-directory-listener/debian/univention-directory-listener.triggers:
> interest-noawait /usr/lib/univention-directory-listener/system

TODO: `-nowait`?

management/univention-directory-listener/debian/univention-directory-listener.postinst:
> trigger () {
>   deb-systemd-invoke try-reload-or-restart univention-directory-listener.service
> }
>
> case "$1" in
> triggered)
>   trigger $2 # no quoting; split on $IFS
>   ;;
> esac

See /usr/share/doc/dpkg-dev/triggers.txt.gz
Comment 3 Philipp Hahn univentionstaff 2021-08-31 13:51:18 CEST
(In reply to Philipp Hahn from comment #2)
> PoC:

+ remove all calls to `systemctrl restart udl.service` from ALL .{pre,post}{inst,rm} files ...
Comment 4 Florian Best univentionstaff 2021-09-08 10:26:26 CEST
I am raising the user pain to "7: Bug causes data loss".
If this error occurs listener modules are not loaded, therefore all transactions are lost during the period until the next restart of the UDL.

This already happened in Bug #53754, Bug #50338, Bug #49832, Bug #50381, Bug #50514, Bug #49866 and will next time happen when we release Bug #53631 comment 13.