Created attachment 11313 [details] Example test files Using UCS 5.2.0 with local mirrored repos returns an 403 error in the file /etc/apt/sources.list.d/15_ucs-online-version.list # An error occurred during the repository check. The error message: # Traceback (most recent call last): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 611, in access # res = UCSHttpServer.opener.open(req, timeout=self.timeout) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 525, in open # response = meth(req, response) # ^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response # response = self.parent.error( # ^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 563, in error # return self._call_chain(*args) # ^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain # result = func(*args) # ^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default # raise HTTPError(req.full_url, code, msg, hdrs, fp) # urllib.error.HTTPError: HTTP Error 403: # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1212, in _get_releases # _code, _size, data = self.server.access(None, 'ucs-releases.json', get=True) we found that is no exception handling for 403 in tools.py (/usr/lib/python3/dist-packages/univention/updater/tools.py) we extend our tests and found that curl works with no errors but wget need the option --auth-no-challenge to work correctly with the repository server. Some tests with python3 and the libs urllib and http.client shows a different result. urllib returns 403 http.client no error It looks like urllib does not function out of the box with our repository server jfrog artifactory. attached the python3 test programs AI generated (urllib, http.client) that produced the error.
Can you please attach the whole traceback?
It would be good to know the actual HTTP transfer which is done. Both clients send HTTP basic auch credentials, right? Are you sure the password (or username) doesn't contain special UTF-8 characters? Your AI generated example script "test-httpclient.py" uses wrong (or at least not good defined) HTTP semantics: > encoded_credentials = base64.b64encode(credentials.encode("utf-8")).decode("utf-8") Test it again via: > encoded_credentials = base64.b64encode(credentials.encode("ISO8859-1")).decode("ASCII")
yes, username and password doesn't contain special UTF-8 Character. and the http.client ist correctly working, the problem is with urllib. Which traceback you need ?
The whole content of /etc/apt/sources.list.d/15_ucs-online-version.list contains the whole changed tracebacks. In your initial description only the first one is fully present.
#Warning: This file is auto-generated and might be overwritten by # univention-config-registry. # Please edit the following file(s) instead: #Warnung: Diese Datei wurde automatisch generiert und kann durch # univention-config-registry ueberschrieben werden. # Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en): # # /etc/univention/templates/files/etc/apt/sources.list.d/15_ucs-online-version.list # # An error occurred during the repository check. The error message: # Traceback (most recent call last): # File "/usr/lib/python3.11/urllib/request.py", line 1348, in do_open # h.request(req.get_method(), req.selector, req.data, headers, # File "/usr/lib/python3.11/http/client.py", line 1282, in request # self._send_request(method, url, body, headers, encode_chunked) # File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request # self.endheaders(body, encode_chunked=encode_chunked) # File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders # self._send_output(message_body, encode_chunked=encode_chunked) # File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output # self.send(msg) # File "/usr/lib/python3.11/http/client.py", line 975, in send # self.connect() # File "/usr/lib/python3.11/http/client.py", line 1447, in connect # super().connect() # File "/usr/lib/python3.11/http/client.py", line 941, in connect # self.sock = self._create_connection( # ^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/socket.py", line 851, in create_connection # raise exceptions[0] # File "/usr/lib/python3.11/socket.py", line 836, in create_connection # sock.connect(sa) # OSError: [Errno 101] Network is unreachable # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 611, in access # res = UCSHttpServer.opener.open(req, timeout=self.timeout) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 519, in open # response = self._open(req, data) # ^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 536, in _open # result = self._call_chain(self.handle_open, protocol, protocol + # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain # result = func(*args) # ^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 1391, in https_open # return self.do_open(http.client.HTTPSConnection, req, # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 1351, in do_open # raise URLError(err) # urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable> # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "<stdin>", line 12, in <module> # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1143, in __init__ # self.ucr_reinit() # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1196, in ucr_reinit # self._get_releases() # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1212, in _get_releases # _code, _size, data = self.server.access(None, 'ucs-releases.json', get=True) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 679, in access # raise ConfigurationError(uri, reason) # univention.updater.errors.ConfigurationError: Configuration error: Network is unreachable # # After fixing this issue, you should rewrite this file with the following command: # univention-config-registry commit /etc/apt/sources.list.d/15_ucs-online-version.list #
> OSError: [Errno 101] Network is unreachable (comment 5) is a different error than before, which was (comment 0): > urllib.error.HTTPError: HTTP Error 403:
Sorry, this last traceback was from another test system, this ist the actual output, this ist the origínal : #Warning: This file is auto-generated and might be overwritten by # univention-config-registry. # Please edit the following file(s) instead: #Warnung: Diese Datei wurde automatisch generiert und kann durch # univention-config-registry ueberschrieben werden. # Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en): # # /etc/univention/templates/files/etc/apt/sources.list.d/15_ucs-online-version.list # # An error occurred during the repository check. The error message: # Traceback (most recent call last): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 611, in access # res = UCSHttpServer.opener.open(req, timeout=self.timeout) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 525, in open # response = meth(req, response) # ^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response # response = self.parent.error( # ^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 563, in error # return self._call_chain(*args) # ^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain # result = func(*args) # ^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default # raise HTTPError(req.full_url, code, msg, hdrs, fp) # urllib.error.HTTPError: HTTP Error 403: # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1212, in _get_releases # _code, _size, data = self.server.access(None, 'ucs-releases.json', get=True) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 655, in access # raise DownloadError(uri, res.code) # univention.updater.errors.DownloadError: Error downloading https://XXXXXXX.de/artifactory/api/deb/XXXXXX-ucs-updates-software-univention-test/snapshots/test/ucs-releases.json: 403 # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "<stdin>", line 12, in <module> # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1143, in __init__ # self.ucr_reinit() # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1196, in ucr_reinit # self._get_releases() # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1218, in _get_releases # raise ConfigurationError(uri, 'non-existing prefix "%s": %s' % (self.repourl.path, uri)) # univention.updater.errors.ConfigurationError: Configuration error: non-existing prefix "/artifactory/api/deb/XXXXX-ucs-updates-software-univention-test/snapshots/test/": https://XXXXXX.de/artifactory/api/deb/XXXXX-ucs-updates-software-univention-test/snapshots/test/ucs-releases.json # # After fixing this issue, you should rewrite this file with the following command: # univention-config-registry commit /etc/apt/sources.list.d/15_ucs-online-version.list #
MWE to replicate the jFrog Auth behaviour derived from https://jfrog.com/help/r/jfrog-platform-administration-documentation/basic-authentication (pip install Flask-BasicAuth) ``` from flask import Flask, jsonify from flask_basicauth import BasicAuth app = Flask(__name__) app.config['BASIC_AUTH_USERNAME'] = 'john' app.config['BASIC_AUTH_PASSWORD'] = 'matrix' basic_auth = BasicAuth(app) @app.route("/") def hello(): return "Hello World!", 200 @app.route("/authfail") def fail(): return "Auth failed", 403 @app.route("/basicauth", methods=['GET']) @basic_auth.required def auth_foo(): return "BasicAuth successful!" if __name__ == '__main__': app.run( debug=True, ) ``` With this, I can use the provided test scripts, but without SSL.
Problem Analysis • urllib waits for 401 Unauthorized response before sending credentials • JFrog Artifactory returns 403 Forbidden instead of 401 for missing auth • This breaks the authentication flow in tools.py • wget --auth-no-challenge works because it sends auth headers preemptively • http.client works because it sends headers immediately
Can we create a bug report at JFrog, so that they can at least give a statement about this behavior?
Last traceback with the recent patch by carlos: ``` #Warning: This file is auto-generated and might be overwritten by # univention-config-registry. # Please edit the following file(s) instead: #Warnung: Diese Datei wurde automatisch generiert und kann durch # univention-config-registry ueberschrieben werden. # Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en): # # /etc/univention/templates/files/etc/apt/sources.list.d/15\_ucs-online-version.list # # An error occurred during the repository check. The error message: # Traceback (most recent call last): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 618, in access # res = UCSHttpServer.opener.open(req, timeout=self.timeout) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 525, in open # response = meth(req, response) # ^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 634, in http\_response # response = self.parent.error( # ^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 563, in error # return self.\_call\_chain(\*args) # ^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 496, in \_call\_chain # result = func(\*args) # ^^^^^^^^^^^ # File "/usr/lib/python3.11/urllib/request.py", line 643, in http\_error\_default # raise HTTPError(req.full\_url, code, msg, hdrs, fp) # urllib.error.HTTPError: HTTP Error 403: # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "\<stdin>", line 12, in \<module> # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1156, in **init** # self.ucr\_reinit() # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1209, in ucr\_reinit # self.\_get\_releases() # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1225, in \_get\_releases # \_code, \_size, data = self.server.access(None, 'ucs-releases.json', get=True) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 662, in access # raise ConfigurationError(uri, 'access forbidden') # univention.updater.errors.ConfigurationError: Configuration error: access forbidden # # After fixing this issue, you should rewrite this file with the following command: # univention-config-registry commit /etc/apt/sources.list.d/15\_ucs-online-version.list # ```
There's an untested patch in the referenced merge request. But it's important to point out that JFrog is wrong in returning a 403 instead of the expected 401. Ideally, the bug should be opened there instead of us making a workaround for it.
I've updated the patch to get the credentials from UCR and not just the base URL. The patch is not super clean and could be improved, but hopefully it's enough temporarily.
(In reply to Florian Best from comment #10) > Can we create a bug report at JFrog, so that they can at least give a > statement about this behavior? This behavior is intended https://jfrog.com/help/r/jfrog-platform-administration-documentation/hide-existence-of-unauthorized-resources ``` When a user tries to access a resource for which they are not authorized, the default behavior is to indicate that the resource exists but is protected. For example, an anonymous request will result in a request for authentication (401), and a request by an unauthorized authenticated user will simply be denied (403). ```
Created attachment 11323 [details] Patch which should solve it (by Iván/Carlos) This seems to be solved by circumventing the "wrong" 403 error code. This adds "preemptive authentication", which first tries it with credentials, but then also proceeds if they are url-encoded.
univention-updater.yaml 055dcd93466f | feat: Add preemptive authentication for JFrog Artifactory compatibility univention-updater (17.2.5) 055dcd93466f | feat: Add preemptive authentication for JFrog Artifactory compatibility Successful build Package: univention-updater Version: 17.2.5 Branch: 5.2-0 Scope: errata5.2-2
The 403 returned code is handled properly, and also preemptive authentication is added if the server has a user and a password for the corresponding URL.
QA: OK: test OK: patch works in customer env
<https://errata.software-univention.de/#/?erratum=5.2x150>