Univention Bugzilla – Bug 49575
appcenter/query updater/maintenance_information : local_verify_locations() : IOError FileNotFoundError
Last modified: 2023-11-09 09:26:19 CET
+++ This bug was initially created as a clone of Bug #45797 +++ Reported again with higher version: Version: 4.4-0 errata113 (Blumenthal) Internal server error during "appcenter/query". Request: appcenter/query Traceback (most recent call last): File "%PY2.7%/univention/management/console/base.py", line 260, in execute function.__func__(self, request, *args, **kwargs) File "%PY2.7%/univention/management/console/modules/decorators.py", line 318, in _response result = _multi_response(self, request) File "%PY2.7%/univention/management/console/modules/decorators.py", line 192, in _response return function(self, request) File "%PY2.7%/univention/management/console/modules/decorators.py", line 440, in _response return list(function(self, iterator, *nones)) File "%PY2.7%/univention/management/console/modules/decorators.py", line 286, in _fake_func yield function(self, *args) File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 204, in query self.update_applications() File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 232, in update_applications update.call() File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 220, in call return obj.call_with_namespace(namespace) File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 226, in call_with_namespace result = self.main(namespace) File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 84, in main if self._download_apps(app_cache): File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 177, in _download_apps if self._download_files(app_cache, filenames): File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 156, in _download_files new_etag = self._download_file(server, filename, cache_dir, etag, ucs_version) File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 58, in _func return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 211, in _download_file response = urlopen(request) File "/usr/lib/python2.7/dist-packages/univention/appcenter/utils.py", line 324, in urlopen return urllib2.urlopen(request, timeout=60) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 429, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 447, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib/python2.7/dist-packages/univention/appcenter/utils.py", line 311, in https_open return self.do_open(HTTPSConnection, req) File "/usr/lib/python2.7/urllib2.py", line 1195, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/usr/lib/python2.7/httplib.py", line 1042, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1082, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 882, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 844, in send self.connect() File "/usr/lib/python2.7/dist-packages/univention/appcenter/utils.py", line 305, in connect ca_certs="/etc/ssl/certs/ca-certificates.crt") File "/usr/lib/python2.7/ssl.py", line 943, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 552, in __init__ self._context.load_verify_locations(ca_certs) IOError: [Errno 2] No such file or directory Role: domaincontroller_master
reported again: Version: 4.4-4 errata642 (Blumenthal) Error: Internal server error during "appcenter/query". Request: appcenter/query Role: domaincontroller_master
reported again: Version: 4.4-5 errata703 (Blumenthal) Error: Internal server error during "appcenter/query". Request: appcenter/query Role: domaincontroller_master
reported again: Version: 4.4-8 errata962 (Blumenthal) Remark: Trying to get the application to run for the first time and ended up with this error message. Error: Internal server error during "appcenter/query". Request: appcenter/query Traceback (most recent call last): File "%PY2.7%/univention/management/console/base.py", line 359, in __error_handling six.reraise(etype, exc, etraceback) File "%PY2.7%/univention/management/console/base.py", line 262, in execute function.__func__(self, request, *args, **kwargs) File "%PY2.7%/univention/management/console/modules/decorators.py", line 321, in _response result = _multi_response(self, request) File "%PY2.7%/univention/management/console/modules/decorators.py", line 181, in _response return function(self, request) File "%PY2.7%/univention/management/console/modules/decorators.py", line 443, in _response return list(function(self, iterator, *nones)) File "%PY2.7%/univention/management/console/modules/decorators.py", line 289, in _fake_func yield function(self, *args) File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 246, in query self.update_applications() File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 264, in update_applications update.call() File "%PY2.7%/univention/appcenter/actions/__init__.py", line 220, in call return obj.call_with_namespace(namespace) File "%PY2.7%/univention/appcenter/actions/__init__.py", line 226, in call_with_namespace result = self.main(namespace) File "%PY2.7%/univention/appcenter/actions/update.py", line 87, in main if self._download_apps(app_cache): File "%PY2.7%/univention/appcenter/actions/update.py", line 180, in _download_apps if self._download_files(app_cache, filenames): File "%PY2.7%/univention/appcenter/actions/update.py", line 159, in _download_files new_etag = self._download_file(server, filename, cache_dir, etag, ucs_version) File "%PY2.7%/univention/appcenter/actions/__init__.py", line 58, in _func return func(*args, **kwargs) File "%PY2.7%/univention/appcenter/actions/update.py", line 214, in _download_file response = urlopen(request) File "%PY2.7%/univention/appcenter/utils.py", line 325, in urlopen return urllib_request.urlopen(request, timeout=60) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 429, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 447, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "%PY2.7%/univention/appcenter/utils.py", line 312, in https_open return self.do_open(HTTPSConnection, req) File "/usr/lib/python2.7/urllib2.py", line 1195, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/usr/lib/python2.7/httplib.py", line 1058, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1098, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 1054, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 892, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 854, in send self.connect() File "%PY2.7%/univention/appcenter/utils.py", line 306, in connect self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, cert_reqs=ssl.CERT_REQUIRED, ca_certs="/etc/ssl/certs/ca-certificates.crt") File "/usr/lib/python2.7/ssl.py", line 943, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 552, in __init__ self._context.load_verify_locations(ca_certs) IOError: [Errno 2] No such file or directory Role: domaincontroller_master
*** Bug 53254 has been marked as a duplicate of this bug. ***
*** Bug 54357 has been marked as a duplicate of this bug. ***
For yet some unknown reason ca-certificates is not available. Observed with UCS - 4.4-0 - 4.4-4 - 4.4-5 - 4.4-8 - 4.4-9 - 5.0-0 Observed for UMC commands - appcenter/query - updater/maintenance_information # dpkg -s python3-univention-appcenter python3-univention-updater univention-updater Package: python3-univention-appcenter Depends: … Package: python3-univention-updater Depends: … Package: univention-updater Depends: …, ca-certificates, …, python3-univention-updater (= 15.0.10-1), … As univention/appcenter/utils.py directly references /etc/ssl/certs/ca-certificates.crt it must declare a "Depends: ca-certificates": management/univention-appcenter/python/appcenter/utils.py:373: ssl_context.load_verify_locations("/etc/ssl/certs/ca-certificates.crt") management/univention-appcenter/umc/python/appcenter/util.py:160: ssl_context.load_verify_locations("/etc/ssl/certs/ca-certificates.crt") python3-univention-updater should also declare the dependency.
There is a race-condition as `update-ca-certificates` is called from `ucs:test/utils/utils.sh:sa_bug53751` while `/usr/share/univention-updater/univention-updater-check` is called from `/etc/cron.d/univention-updater-check` during the initial boot process: > /etc/cron.d/univention-updater-check:14:# @reboot root [ -x /usr/share/univention-updater/univention-updater-check ] && /usr/sbin/jitter 30 /usr/share/univention-updater/univention-updater-check 2>/dev/null >/dev/null This should be fixed by <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=920348>, which is only in Debian-12-Bookworm: It no longer does the > rm -f /etc/ssl/certs/ca-certificates.crt but only does the atomic `mv`. Until then it might be worked-around by running this instead: /etc/ssl/certs# ls -li ca-certificates.crt ca-certificates.tmp 785176 -rw-r--r-- 1 root root 214668 Oct 5 14:23 ca-certificates.crt 784898 -rw-r--r-- 1 root root 214668 Oct 5 14:55 ca-certificates.tmp /etc/ssl/certs# update-ca-certificates --certbundle ca-certificates.tmp Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. /etc/ssl/certs# ls -li ca-certificates.crt ca-certificates.tmp 785176 -rw-r--r-- 1 root root 214668 Oct 5 14:23 ca-certificates.crt 784908 -rw-r--r-- 1 root root 214668 Oct 5 14:55 ca-certificates.tmp /etc/ssl/certs# mv ca-certificates.tmp ca-certificates.crt [4.4-9] 45c61919c6 test(utils): Workaround update-ca-certificates test/utils/utils.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) [5.0-5] 5c7f81e7ed test(utils): Workaround update-ca-certificates test/utils/utils.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) From /var/log/univention/appcenter.log.1.gz > 1596 actions.upgrade-search 23-09-27 07:41:37 [ DEBUG]: Calling upgrade-search ... > 1596 actions.update 23-09-27 07:41:42 [ INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.2/index.json.gz.gpg"... > 1596 actions.update 23-09-27 07:41:42 [ ERROR]: [Errno 2] No such file or directory Traceback (most recent call last): > File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 226, in call_with_namespace > result = self.main(namespace) ... > File "/usr/lib/python2.7/ssl.py", line 552, in __init__ > self._context.load_verify_locations(ca_certs) >IOError: [Errno 2] No such file or directory > 1596 actions.upgrade-search 23-09-27 07:41:42 [ ERROR]: [Errno 2] No such file or directory >Traceback (most recent call last): ... > self._context.load_verify_locations(ca_certs) >IOError: [Errno 2] No such file or directory From `journalctl -b 1`: > -- Logs begin at Wed 2023-09-27 07:41:07 CEST, end at Wed 2023-09-27 09:12:47 CEST. -- ... >Sep 27 07:41:20 unassigned-hostname sshd[1335]: Accepted keyboard-interactive/pam for root from 10.205.2.91 port 33414 ssh2 ... >Sep 27 07:41:40 unassigned-hostname sshd[1624]: Accepted keyboard-interactive/pam for root from 10.205.2.91 port 55746 ssh2
This did not happen again for a few week, so closing as fixed.
(In reply to Philipp Hahn from comment #8) > This did not happen again for a few week, so closing as fixed. But how did it happen in customer environments? We have several traceback reports. Your fix is only in our utils.sh Jenkins configuration.
(In reply to Philipp Hahn from comment #6) > For yet some unknown reason ca-certificates is not available. Also see Bug #51203: univention-ssl must depend on ca-certificates