Bug 49575 - appcenter/query updater/maintenance_information : local_verify_locations() : IOError FileNotFoundError
appcenter/query updater/maintenance_information : local_verify_locations() : ...
Status: RESOLVED FIXED
Product: UCS
Classification: Unclassified
Component: General
UCS 4.4
Other Linux
: P5 normal (vote)
: ---
Assigned To: Philipp Hahn
UCS maintainers
:
: 53254 54357 (view as bug list)
Depends on: 45797
Blocks:
  Show dependency treegraph
 
Reported: 2019-05-29 14:06 CEST by Johannes Keiser
Modified: 2023-11-09 09:26 CET (History)
7 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.171
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2022062821000348, 2021031321000347, 2021032921000335, 2021042221000981, 2019052921000381, 2020063021000034, 2020081121000183
Bug group (optional): Error handling
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Johannes Keiser univentionstaff 2019-05-29 14:06:37 CEST
+++ This bug was initially created as a clone of Bug #45797 +++

Reported again with higher version: Version: 4.4-0 errata113 (Blumenthal)

Internal server error during "appcenter/query".
Request: appcenter/query

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 260, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 318, in _response
    result = _multi_response(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 192, in _response
    return function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 440, in _response
    return list(function(self, iterator, *nones))
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 286, in _fake_func
    yield function(self, *args)
  File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 204, in query
    self.update_applications()
  File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 232, in update_applications
    update.call()
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 220, in call
    return obj.call_with_namespace(namespace)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 226, in call_with_namespace
    result = self.main(namespace)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 84, in main
    if self._download_apps(app_cache):
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 177, in _download_apps
    if self._download_files(app_cache, filenames):
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 156, in _download_files
    new_etag = self._download_file(server, filename, cache_dir, etag, ucs_version)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 58, in _func
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/update.py", line 211, in _download_file
    response = urlopen(request)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/utils.py", line 324, in urlopen
    return urllib2.urlopen(request, timeout=60)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/utils.py", line 311, in https_open
    return self.do_open(HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1195, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/utils.py", line 305, in connect
    ca_certs="/etc/ssl/certs/ca-certificates.crt")
  File "/usr/lib/python2.7/ssl.py", line 943, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 552, in __init__
    self._context.load_verify_locations(ca_certs)
IOError: [Errno 2] No such file or directory

Role: domaincontroller_master
Comment 1 Christian Castens univentionstaff 2020-08-20 15:13:14 CEST
reported again:

Version: 4.4-4 errata642 (Blumenthal)

Error:
Internal server error during "appcenter/query".
Request: appcenter/query

Role: domaincontroller_master
Comment 2 Christian Castens univentionstaff 2020-08-24 10:09:01 CEST
reported again:

Version: 4.4-5 errata703 (Blumenthal)

Error:
Internal server error during "appcenter/query".
Request: appcenter/query

Role: domaincontroller_master
Comment 3 Maximilian Janßen univentionstaff 2021-11-26 11:23:49 CET
reported again:


Version: 4.4-8 errata962 (Blumenthal)

Remark: Trying to get the application to run for the first time and ended up with this error message. 

Error: 
Internal server error during "appcenter/query".
Request: appcenter/query

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 359, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "%PY2.7%/univention/management/console/base.py", line 262, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 321, in _response
    result = _multi_response(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 246, in query
    self.update_applications()
  File "%PY2.7%/univention/management/console/modules/appcenter/__init__.py", line 264, in update_applications
    update.call()
  File "%PY2.7%/univention/appcenter/actions/__init__.py", line 220, in call
    return obj.call_with_namespace(namespace)
  File "%PY2.7%/univention/appcenter/actions/__init__.py", line 226, in call_with_namespace
    result = self.main(namespace)
  File "%PY2.7%/univention/appcenter/actions/update.py", line 87, in main
    if self._download_apps(app_cache):
  File "%PY2.7%/univention/appcenter/actions/update.py", line 180, in _download_apps
    if self._download_files(app_cache, filenames):
  File "%PY2.7%/univention/appcenter/actions/update.py", line 159, in _download_files
    new_etag = self._download_file(server, filename, cache_dir, etag, ucs_version)
  File "%PY2.7%/univention/appcenter/actions/__init__.py", line 58, in _func
    return func(*args, **kwargs)
  File "%PY2.7%/univention/appcenter/actions/update.py", line 214, in _download_file
    response = urlopen(request)
  File "%PY2.7%/univention/appcenter/utils.py", line 325, in urlopen
    return urllib_request.urlopen(request, timeout=60)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "%PY2.7%/univention/appcenter/utils.py", line 312, in https_open
    return self.do_open(HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1195, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 1058, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1098, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1054, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 892, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 854, in send
    self.connect()
  File "%PY2.7%/univention/appcenter/utils.py", line 306, in connect
    self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, cert_reqs=ssl.CERT_REQUIRED, ca_certs="/etc/ssl/certs/ca-certificates.crt")
  File "/usr/lib/python2.7/ssl.py", line 943, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 552, in __init__
    self._context.load_verify_locations(ca_certs)
IOError: [Errno 2] No such file or directory

Role: domaincontroller_master
Comment 4 Philipp Hahn univentionstaff 2023-09-20 12:35:06 CEST
*** Bug 53254 has been marked as a duplicate of this bug. ***
Comment 5 Philipp Hahn univentionstaff 2023-09-20 12:35:27 CEST
*** Bug 54357 has been marked as a duplicate of this bug. ***
Comment 6 Philipp Hahn univentionstaff 2023-09-20 12:59:34 CEST
For yet some unknown reason ca-certificates is not available.

Observed with UCS
- 4.4-0
- 4.4-4
- 4.4-5
- 4.4-8
- 4.4-9
- 5.0-0

Observed for UMC commands
- appcenter/query
- updater/maintenance_information


# dpkg -s python3-univention-appcenter python3-univention-updater univention-updater
Package: python3-univention-appcenter
Depends: …

Package: python3-univention-updater
Depends: …

Package: univention-updater
Depends: …, ca-certificates, …, python3-univention-updater (= 15.0.10-1), …



As univention/appcenter/utils.py directly references /etc/ssl/certs/ca-certificates.crt it must declare a "Depends: ca-certificates":
management/univention-appcenter/python/appcenter/utils.py:373:        ssl_context.load_verify_locations("/etc/ssl/certs/ca-certificates.crt")
management/univention-appcenter/umc/python/appcenter/util.py:160:        ssl_context.load_verify_locations("/etc/ssl/certs/ca-certificates.crt")

python3-univention-updater should also declare the dependency.
Comment 7 Philipp Hahn univentionstaff 2023-10-05 15:11:09 CEST
There is a race-condition as `update-ca-certificates` is called from `ucs:test/utils/utils.sh:sa_bug53751` while `/usr/share/univention-updater/univention-updater-check` is called from `/etc/cron.d/univention-updater-check` during the initial boot process:
> /etc/cron.d/univention-updater-check:14:# @reboot   root   [ -x /usr/share/univention-updater/univention-updater-check ] && /usr/sbin/jitter 30 /usr/share/univention-updater/univention-updater-check 2>/dev/null >/dev/null


This should be fixed by <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=920348>, which is only in Debian-12-Bookworm: It no longer does the
> rm -f /etc/ssl/certs/ca-certificates.crt
but only does the atomic `mv`.


Until then it might be worked-around by running this instead:

/etc/ssl/certs# ls -li ca-certificates.crt ca-certificates.tmp
  785176 -rw-r--r-- 1 root root 214668 Oct  5 14:23 ca-certificates.crt
  784898 -rw-r--r-- 1 root root 214668 Oct  5 14:55 ca-certificates.tmp
/etc/ssl/certs# update-ca-certificates --certbundle ca-certificates.tmp
  Updating certificates in /etc/ssl/certs...
  0 added, 0 removed; done.
  Running hooks in /etc/ca-certificates/update.d...
  done.
/etc/ssl/certs# ls -li ca-certificates.crt ca-certificates.tmp
  785176 -rw-r--r-- 1 root root 214668 Oct  5 14:23 ca-certificates.crt
  784908 -rw-r--r-- 1 root root 214668 Oct  5 14:55 ca-certificates.tmp
/etc/ssl/certs# mv ca-certificates.tmp ca-certificates.crt

[4.4-9] 45c61919c6 test(utils): Workaround update-ca-certificates
 test/utils/utils.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

[5.0-5] 5c7f81e7ed test(utils): Workaround update-ca-certificates
 test/utils/utils.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)



From /var/log/univention/appcenter.log.1.gz
>  1596 actions.upgrade-search           23-09-27 07:41:37 [   DEBUG]: Calling upgrade-search
...
>  1596 actions.update                   23-09-27 07:41:42 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.2/index.json.gz.gpg"...
>  1596 actions.update                   23-09-27 07:41:42 [   ERROR]: [Errno 2] No such file or directory
Traceback (most recent call last):
>  File "/usr/lib/python2.7/dist-packages/univention/appcenter/actions/__init__.py", line 226, in call_with_namespace
>    result = self.main(namespace)
...
>  File "/usr/lib/python2.7/ssl.py", line 552, in __init__
>    self._context.load_verify_locations(ca_certs)
>IOError: [Errno 2] No such file or directory
>  1596 actions.upgrade-search           23-09-27 07:41:42 [   ERROR]: [Errno 2] No such file or directory
>Traceback (most recent call last):
...
>    self._context.load_verify_locations(ca_certs)
>IOError: [Errno 2] No such file or directory


From `journalctl -b 1`:
> -- Logs begin at Wed 2023-09-27 07:41:07 CEST, end at Wed 2023-09-27 09:12:47 CEST. --
...
>Sep 27 07:41:20 unassigned-hostname sshd[1335]: Accepted keyboard-interactive/pam for root from 10.205.2.91 port 33414 ssh2
...
>Sep 27 07:41:40 unassigned-hostname sshd[1624]: Accepted keyboard-interactive/pam for root from 10.205.2.91 port 55746 ssh2
Comment 8 Philipp Hahn univentionstaff 2023-10-26 13:25:49 CEST
This did not happen again for a few week, so closing as fixed.
Comment 9 Florian Best univentionstaff 2023-10-26 13:30:44 CEST
(In reply to Philipp Hahn from comment #8)
> This did not happen again for a few week, so closing as fixed.

But how did it happen in customer environments? We have several traceback reports.
Your fix is only in our utils.sh Jenkins configuration.
Comment 10 Philipp Hahn univentionstaff 2023-11-09 09:26:19 CET
(In reply to Philipp Hahn from comment #6)
> For yet some unknown reason ca-certificates is not available.

Also see Bug #51203: univention-ssl must depend on ca-certificates