Univention Bugzilla – Bug 55044
UCRV repository/online/component/$comp/server=https://… disables username,password,prefix
Last modified: 2023-04-13 16:03:33 CEST
univention/updater/repo_url.py:74:__init__() has this code block: > server = ucrv('server', '') > url = urlsplit(server) > if url.scheme: > self.scheme = url.scheme > self.username = url.username > self.password = url.password > self.hostname = url.hostname > port = url.port > prefix = url.path > else: > self.username = ucrv('username', defaults.username) > self.password = ucrv('password', defaults.password) > if server: > self.hostname = server > port = ucrv('port', 80) > self.scheme = 'https' if port == 443 else 'http' > prefix = ucrv('prefix', None) > else: > self.hostname = defaults.hostname > port = ucrv('port', defaults.port) > self.scheme = defaults.scheme > prefix = ucrv('prefix', defaults.path) As soon as `repository/online/component/…/server includes a schema, the username, password, port (and prefix - watch out for Bug #47267) are taken from the URL and no longer from the old UCRVs: they are silently ignored. This is hard to debug and caused one major support case, where the customer was no longer able to update his many servers. This is especially annoying as the schema MUST be specified if https://services.software-univention.de/ should be used; otherwise the customer credentials are sent UNENCRYPTED over the internet! /etc/apt/mirror.list only showed the following unhelpful traceback, as it neither named the failing component nor did it include any useful URL; even adding manual logging did not help and `pdb3` had to be used to find the location above: # An error occurred during the repository check. The error message: # Traceback (most recent call last): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 683, in access # res = UCSHttpServer.opener.open(req, timeout=self.timeout) # File "/usr/lib/python3.7/urllib/request.py", line 531, in open # response = meth(req, response) # File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response # 'http', request, response, code, msg, hdrs) # File "/usr/lib/python3.7/urllib/request.py", line 569, in error # return self._call_chain(*args) # File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain # result = func(*args) # File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default # raise HTTPError(req.full_url, code, msg, hdrs, fp) # urllib.error.HTTPError: HTTP Error 401: Unauthorized # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "<stdin>", line 28, in <module> # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1592, in print_component_repositories # result += comp.repositories(start, end, clean=clean, for_mirror_list=for_mirror_list, failed=failed) # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1138, in repositories # for server, struct in self.versions(start, end, for_mirror_list): # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1116, in versions # server = self.server(for_mirror_list=for_mirror_list) # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1093, in server # assert testserver.access(None, '') # File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 721, in access # raise ConfigurationError(uri, 'credentials not accepted') # univention.updater.errors.ConfigurationError: Configuration error: credentials not accepted The bug was resolved by specifying "service.univention.de" as `server` instead of "http://service.univention.de". Long-term all customer should switch to "https://$USERNAME@$PASSWORD@service.software-univention.de/apt/$CUSTOMER/" to use encrypted https://. 1. The UCRV repository/online/component/…{,/{server,username,password,port,prefix}} should be better documented in the (extended) manual 2. There should be a UMC diagnostics module, which should warn the user, that {username,password,port,prefix} will be UNUSED if `server` contains a scheme 3. The updater should also warn 4. The updater should provider better debugging options
(In reply to Philipp Hahn from comment #0) > > 1. The UCRV > repository/online/component/…{,/{server,username,password,port,prefix}} > should be better documented in the (extended) manual > 2. There should be a UMC diagnostics module, which should warn the user, > that {username,password,port,prefix} will be UNUSED if `server` contains a > scheme > 3. The updater should also warn > 4. The updater should provider better debugging options I think 1, 3 & 4 should be the focus here. 2 could be splitted in a separate feature request.
(In reply to Ingo Steuwer from comment #1) > (In reply to Philipp Hahn from comment #0) > > > > 1. The UCRV > > repository/online/component/…{,/{server,username,password,port,prefix}} > > should be better documented in the (extended) manual > > 2. There should be a UMC diagnostics module, which should warn the user, > > that {username,password,port,prefix} will be UNUSED if `server` contains a > > scheme > > 3. The updater should also warn > > 4. The updater should provider better debugging options > > I think 1, 3 & 4 should be the focus here. 2 could be splitted in a separate > feature request. The problem is that the Updater does not have a good feedback channel if things go wrong or are wrongly configured: You set some UCR variables and if you're luck you well see error messages in /etc/apt/sources.list.d/*.list, but you have to open those files in a console. You might also get errors when you manually start an update via UMC (or CLI), but they will go to /dev/null if they are done by cron.
MR: https://git.knut.univention.de/univention/ucs/-/merge_requests/531 Commit: ee2791fb820062e987476d1ab02bb851175161da
Implementation of this issue consists of several parts: * The Repository Setting module got now additional logic to support the merging of the following UCR variables which got an deprecated state: repository/online/(prefix, port) repository/online/component/*/(prefix, port, username, password, unmaintained) * When entering the Repository Settings module and these variables exist, they will be merged into the corresponding 'server' variables. The inputs fields for the deprecated variables where deleted as their contents is reflected in the 'server' variable only. This is the case for the general repository settings as well as for the settings for the additional repositories. * When pressing the Apply Changes button the merged 'server' UCR variable is updated and the deprecated UCR variables are deleted. This is true for the general repository settings as well as for the settings for the additional repositories. If the scheme in the 'server' variable is different to http or https an error message is raised, the 'server' variable is not updated and the deprecated variables are not deleted. * The UCR UMC module does hide the deprecated variables. * A new diagnostic routine was added to check and optionally to reestablish the correctness of the repository configuration. The following checks are performed: 1. It is checked, if there are deprecated variables still defined. In this case by pressing the "ADJUST ALL COMPONENTS" button the merge process which is also done in the repository setting module is executed by the diagnostic routine including the deletion of the deprecated variables. 2. It is checked if there are repository/online/server or /repository/online/component/*/server UCR variables having a scheme other than http or https. This can only be corrected manually using either the repository settings module or the UCR module to directly modify the variables. This second check can be disabled by defining an UCR variable 'diagnostic/check/65_check_repository_config/ignore' to any non empty value. * In UCR a new variable type 'url_http' was added in order to support validation of http/https URL strings. The definition of port and server variables got updated accordingly.
Package: univention-config-registry Version: 15.0.4-4A~5.0.0.202211281358 Branch: ucs_5.0-0 Scope: errata5.0-2 Package: univention-updater Version: 15.0.7-22A~5.0.0.202211281353 Branch: ucs_5.0-0 Scope: errata5.0-2 Package: univention-appcenter Version: 9.0.3-5A~5.0.0.202211281403 Branch: ucs_5.0-0 Scope: errata5.0-2 Package: univention-management-console-module-diagnostic Version: 6.0.2-7A~5.0.0.202211281411 Branch: ucs_5.0-0 Scope: errata5.0-2 Package: univention-management-console-module-ucr Version: 9.0.0-8A~5.0.0.202211281417 Branch: ucs_5.0-0 Scope: errata5.0-2
Verified: * Code review * The repository settings diagnostic module warns about deprecated configurations. * The repository settings diagnostic module warns about non http(s) server variables. * The repository settings diagnostic module merges deprecated variables into the server one clicking "ADJUST ALL COMPONENTS". * The UCR diagnostic module warns about "repository/online/component/*/server" variables that are not an absolute URL or doesn't use http(s). * The UMC repository settings component merges and removes deprecated repostiry variables in the server variable on "Apply changes". * The UMC repository settings component does not allow setting non http(s) or incomplete URLs. * The UMC UCR components hiddes deprecated repository variables. * Test for repository variable merge. * Advisories
During REVIEW the were twe questions: 1. OX configures multiple repositories with password "the old way". Check what happens when the are converted to "the new way", e.g. if an OX update still works. 2. Do we need to update any documentation?
Additional note: Since this change, the diagnostic check fails on all test machines. Because we ourselves set the default repository server to updates.software-univention.de and similar. Without scheme. I'd ask you to fix this in the seed jobs, where the default is determined. https://git.knut.univention.de/univention/dist/jenkins/-/tree/master/seed-jobs And our activate-errata-test-scope.sh script may need alterations too. https://git.knut.univention.de/univention/dist/jenkins-data/-/tree/master/
[univention/dist/jenkins-data>master] 31d0734 fix(errata): bash scripts/activate-errata-test-scope.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) [univention/dist/jenkins>master] 461c043 fix(seed): Use URL for repositories conf/jobdsl/seed-ucsschool.groovy | 2 +- ...ns.plugins.extensible_choice_parameter.GlobalTextareaChoiceListProvider.xml | 12 ++++++------ seed-jobs/create_ucs_branch_jobs.groovy | 8 ++++---- seed-jobs/create_ucs_branch_jobs_4.3.groovy | 8 ++++---- seed-jobs/create_ucs_branch_jobs_4.4.groovy | 8 ++++---- seed-jobs/create_ucs_branch_jobs_5.0.groovy | 4 ++-- seed-jobs/create_ucsschool_branch_jobs.groovy | 4 ++-- seed-jobs/utilities/EC2Tools.groovy | 14 +++++++------- 8 files changed, 30 insertions(+), 30 deletions(-) Copied to jenkins1 and jenkins2
Updates: repository/online/component/*/unmaintained is not made deprecated any more: https://git.knut.univention.de/univention/ucs/-/merge_requests/581 Package: univention-appcenter Version: 9.0.3-7A~5.0.0.202212071136 Branch: ucs_5.0-0 Scope: errata5.0-2 Package: univention-management-console-module-diagnostic Version: 6.0.2-8A~5.0.0.202212071141 Branch: ucs_5.0-0 Scope: errata5.0-2
Deprecated repository/online/component UCR variables username, password and prefix also removed from OX appsuite. MR: https://git.knut.univention.de/univention/open-xchange/appsuite/-/merge_requests/6 Commit: 723a7565 Bug #55044: Remove deprecated UCRVs Summary of all commits in the ucs repo: df109cecb0 Bug #55044: Rework of Repository Setting UMC Module and marking deprecated of repository/online/component/... UCR vars 9295b47774 Bug #55044: Adding validation fpr http(s) URLs in UCR 6127851ded Bug #55044: Added scheme validation in repository seetings and diagnostics routine aa3ccf8567 Bug #55044: Added YAML files dad3e8aeb6 Bug #55044: Updated YAML files 403ce5a99c Bug #55044: Still use unmaintained UCRV for components 8409999414 Bug #55044: Still use unmaintained UCRV for components 3eb645d1ed Bug #55044: overwrite repository/online/server from kvm template df1ac470ae Bug #55044: Correct scheme in repository url
(In reply to Juan Carlos from comment #7) > Verified: > > * Code review > * The repository settings diagnostic module warns about deprecated > configurations. > * The repository settings diagnostic module warns about non http(s) server > variables. > * The repository settings diagnostic module merges deprecated variables into > the server one clicking "ADJUST ALL COMPONENTS". > * The UCR diagnostic module warns about > "repository/online/component/*/server" variables that are not an absolute > URL or doesn't use http(s). > * The UMC repository settings component merges and removes deprecated > repostiry variables in the server variable on "Apply changes". > * The UMC repository settings component does not allow setting non http(s) > or incomplete URLs. > * The UMC UCR components hiddes deprecated repository variables. > * Test for repository variable merge. > * Advisories Also verified: * repository/online/component/*/unmaintained Not deprecated anymore * Updated kvm template * Updated OX Appsuite repository settings.
<https://errata.software-univention.de/#/?erratum=5.0x524> <https://errata.software-univention.de/#/?erratum=5.0x525> <https://errata.software-univention.de/#/?erratum=5.0x526> <https://errata.software-univention.de/#/?erratum=5.0x527> <https://errata.software-univention.de/#/?erratum=5.0x528>
*** Bug 44906 has been marked as a duplicate of this bug. ***
Customer effected Ticket#2023022621000321 ## Check failed: 65_check_repository_config - Check repository configuration for correctness ## Some UCR variables for the repository configuration are deprecated. As these variables should be no longer used, this check has been added to check for the existence of these variables. Use the {appcenter:components} to correct these values by once saving the General repository settings as well as saving the settings for all Additional repositories or press the Button ADJUST ALL COMPONENTS to correct these settings and delete the obsolete variables. Furthermore, it is checked if the scheme of the server variable is either http or https The variable 'repository/online/component/backend7106/username' is deprecated and should no longer be used. The variable 'repository/online/component/frontend7106/username' is deprecated and should no longer be used. The variable 'repository/online/component/mobileapi1143/username' is deprecated and should no longer be used. The variable 'repository/online/component/mobileapirestricted211/username' is deprecated and should no longer be used. The variable 'repository/online/component/oxdocconv7106/username' is deprecated and should no longer be used.
A partner ran in this issue again, and has some remarks: using the ucr Variables: # Component <example>: Configuration error: credentials not accepted --------------- But setting the url with username and password directly repository/online/component/example/parts: maintained repository/online/component/example/server: https://xxxxx:1234567@service.software-univention.de/apt/xxxxx/ repository/online/component/example: yes gets the warning: W: Value 'maintained' incompatible for 'repository/online/component/meinbilo/parts', but setting anyway which is interpreted as an error by our Ansible module univention.ucs_modules.univention_config_registry as an error, which then which then aborts the playbook ----------- If one omits the variable .../parts, then comes from UCR no more warning, with which also the Ansible role then wants. But: if you have the credentials in the URL, then APT complains with every update: N: Usage of apt_auth.conf(5) should be preferred over embedding login information directly in the sources.list(5) entry for 'https://service.software-univention.de/univention-repository/apt/172906/5.0[..]' I do not know if it is relevant, but I find the ansible part quite interesting.
(In reply to Christina Scheinig from comment #17) > repository/online/component/example/parts: maintained That UCRV is deprecated since UCS 3.1-1 Bug #30261 comment 5 and has been replaced by UCRV repository/online/component/$comp/maintained=<bool> > But: if you have the credentials in the URL, then APT complains with every > update: > > N: Usage of apt_auth.conf(5) should be preferred over embedding login > information directly in the sources.list(5) entry for > 'https://service.software-univention.de/univention-repository/apt/172906/5. > 0[..]' > > I do not know if it is relevant, but I find the ansible part quite > interesting. This message is only a "Notice", no an "Error" or a "Warning". The issue is known and tracked via Bug #55528