Univention Bugzilla – Bug 31792
Check architecture packages
Last modified: 2016-10-05 20:19:52 CEST
A test case is missing which checks all installed apps if the defined packages from the ini file available in the other architecture. Background of this test case is, that we only run auto tests on amd64.
test created and committed
> test created and committed name is : 43_installed_apps_check_packages_available_for_i386
Suggestions for 43_installed_apps_check_packages_available_for_i386: * Replace raise Exception(...) with sys.exit(1) → exception seems at first glance like some error with the script itself. * Currently, the script output is: > Packages with amd64 architecture: univention-samba4 ,univention-nagios-server > > Error! > Package: univention-nagios-server , version: 7.0.6-2.263.201403171419 > Package: univention-samba4 , version: 3.0.39-17.572.201403131852 > not found in repositorys IMHO, this is a bit unclear and could be more informative: - Which app is concerned? - Which architecture exists and which is missing? - At which URL did you find the package that you checked? - The output could be a bit different, e.g.: > Installed apps: [...] > Error: following packages could not be found for architecture i386 > http://updates.software-univention.de/3.2/maintained/3.2-0/amd64/univention-nagios-server_7.0.5-2.255.201310251145_amd64.deb > ...
fixed it, should be better now
Needs to take Bug#34320 into account: If SupportedArchitectures is present in the ini file and it does not list the server's architecture, this test should not fail as the App Center will prevent installation.
adapted the test . Test now ignores packages with blacklisted architecture
Reopened w.r.t. to discussed changes.
changed structural elements of test 20/43 for better overview also test now only uses python elements (no more bash)
pyflakes says: > 43_installed_apps_check_packages_available_for_i386:42: undefined name 'idefault_package_master' > 43_installed_apps_check_packages_available_for_i386:96: local variable 'app_repository_server' is assigned to but never used On my machine, the script raises a traceback at line 42. Seems to be a typo in the variable name. BTW, I could not find the test being executed in jenkins? Minor stylistic remark: * line 70: > for ipackage in other_arch_packages_with_repository.iteritems(): Could be replace with (as iterating only over the values is necessary): > for in other_arch_packages_with_repository.itervalues(): * line 132: > for ipackage in not_found_packages.iteritems() Could be replace with (as iterating only the keys is necessary): > for ipackage in not_found_packages:
-fixed typo -removed unused variable -adapted test to new Packages.gz structure
(In reply to Drees Dormann from comment #10) > -fixed typo > -removed unused variable > -adapted test to new Packages.gz structure At least the test fails in jenkins: http://jenkins.knut.univention.de:8080/view/Autotest/job/UCS%203.2-2%20Autotest%20MultiEnv/61/SambaVersion=s3,Systemrolle=master/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/ 29.06.14 19:40:13.813 MODULE ( WARN ) : drbd_20140606.ini not found in archive! 29.06.14 19:40:13.813 MODULE ( WARN ) : drbd_20140606.png not found in archive! Traceback (most recent call last): File "43_installed_apps_check_packages_available_for_i386", line 164, in <module> check_packages_available(single_architecture_packages) File "43_installed_apps_check_packages_available_for_i386", line 71, in check_packages_available url = urllib2.urlopen(repository_index_url) File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib/python2.6/urllib2.py", line 1184, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib/python2.6/urllib2.py", line 1130, in do_open h = http_class(host, timeout=req.timeout) # will parse host:port File "/usr/lib/python2.6/httplib.py", line 661, in __init__ self._set_hostport(host, port) File "/usr/lib/python2.6/httplib.py", line 686, in _set_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) httplib.InvalidURL: nonnumeric port: 'xxXXxXx@testing.univention.de' Please file a bug against the App Center if it is a bug in the App Center code.
(In reply to Stefan Gohmann from comment #11) > httplib.InvalidURL: nonnumeric port: 'xxXXxXx@testing.univention.de' This happens when the newest version of an App is provided though a password protected repository, like it currently happens with staged errata update of "univention-samba4". APT returns the URI with the credentials included inline as "http://$username:$password@$hostname/$path". This is passed verbatim to urllib2.urlopen(), while urllib2 does not handle it: it requires an explicit use of an *AuthHandler. urllib.urlopen() on the other hand still supports the inline basic auth information.
Switched to urllib and did some minor fixes, should work as exected now.
In UCS 4 test test case failed: http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-0/job/Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=master/lastCompletedBuild/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/ *** BEGIN *** ['/usr/bin/python', '43_installed_apps_check_packages_available_for_i386'] *** *** 20_appcenter/43_installed_apps_check_packages_available_for_i386 *** Checks if App packages are available as i386 *** *** START TIME: 2014-12-02 19:39:53 *** 02.12.14 19:39:54.623 MODULE ( PROCESS ) : Downloading "https://appcenter.software-univention.de/meta-inf/4.0/index.json.gz"... 02.12.14 19:39:54.768 MODULE ( PROCESS ) : 0 file(s) are new Traceback (most recent call last): File "43_installed_apps_check_packages_available_for_i386", line 156, in <module> single_architecture_packages = get_installed_default_packages() File "43_installed_apps_check_packages_available_for_i386", line 50, in get_installed_default_packages packages[package] = pack.installed.version, pack.installed.uris[0], application IndexError: list index out of range *** END TIME: 2014-12-02 19:39:54 *** *** TEST DURATION (H:MM:SS.ms): 0:00:01.381278 *** *** END *** 1 ***
(In reply to Stefan Gohmann from comment #14) > In UCS 4 test test case failed: → I created Bug 37166 for this
As discussed: 'Architecture' -> 'SupportedArchitectures' MultiArch: ucr set repository/online/architectures='i386 amd64' dpkg --add-architecture i386 apt-get update (package_manager.update()) # begin test http://apt.alioth.debian.org/python-apt-doc/library/apt_pkg.html#apt_pkg.Group # owncloud7 # needs some tests for existance, of course c = apt.apt_pkg.Cache() g = apt.apt_pkg.Group(c, 'php5-libsmbclient') g.find_package('amd64').version_list[0].arch g.find_package('i386').version_list[0].arch # compare with apt.apt_pkg.Group(c, 'owncloud') # end test finally: ucr set repository/online/architectures='amd64' dpkg --remove-architecture i386 apt-get update
the test has been rewritten to match the scheme, exposure is now set to dangerous Now also dependencys and recommends (if installed) will be tested as well.
The test fails on all system roles: <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s4,Systemrolle=master/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s3,Systemrolle=master/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s3,Systemrolle=slave/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s4,Systemrolle=backup/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s3,Systemrolle=backup/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s4,Systemrolle=member/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s4,Systemrolle=slave/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/lastCompletedBuild/SambaVersion=s3,Systemrolle=member/testReport/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/> Traceback (most recent call last): File "43_installed_apps_check_packages_available_for_i386", line 149, in <module> test.check_packages_available() File "43_installed_apps_check_packages_available_for_i386", line 107, in check_packages_available group = apt.apt_pkg.Group(self.pkg_cache, app_package) TypeError: argument 1 must be apt_pkg.Cache, not None
checking of architecture was error prone since ucr varible "repository/online/architecture" is not set by default. Checking has been adapted and also the variable self.pkg_cache will be set in either case, so test should work now.
(In reply to Drees Dormann from comment #19) > Checking has been adapted and also the variable self.pkg_cache will be set > in either case, so test should work now. No, it doesn't: http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=member/23/testReport/junit/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/ Fehlermeldung Test failed Standard Ausgabe (STDOUT) Warning: Could not determin system architecture, not changes made Gathering packages of installed apps Checking if packages are available Standard Fehler (STDERR) 27.02.15 19:18:09.707 MODULE ( PROCESS ) : Downloading "https://appcenter.software-univention.de/meta-inf/4.0/index.json.gz"... 27.02.15 19:18:09.875 MODULE ( PROCESS ) : 0 file(s) are new Traceback (most recent call last): File "43_installed_apps_check_packages_available_for_i386", line 149, in <module> test.check_packages_available() File "43_installed_apps_check_packages_available_for_i386", line 107, in check_packages_available group = apt.apt_pkg.Group(self.pkg_cache, app_package) TypeError: argument 1 must be apt_pkg.Cache, not None
the executed test was still the old revision, ucs-test has been rebuilt. This should be fixed with new revision
It still fails: http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=slave/26/testReport/junit/20_appcenter/43_installed_apps_check_packages_available_for_i386/test/ Fehlermeldung Test failed Standard Ausgabe (STDOUT) Gathering packages of installed apps Adding amd64 architecture File: /etc/apt/mirror.list File: /etc/apt/sources.list.d/15_ucs-online-version.list File: /etc/apt/sources.list.d/20_ucs-online-component.list Standard Fehler (STDERR) 02.03.15 19:56:49.706 MODULE ( PROCESS ) : Downloading "https://appcenter.software-univention.de/meta-inf/4.0/index.json.gz"... 02.03.15 19:56:49.877 MODULE ( PROCESS ) : 0 file(s) are new Traceback (most recent call last): File "43_installed_apps_check_packages_available_for_i386", line 152, in <module> test.set_multi_architecture() File "43_installed_apps_check_packages_available_for_i386", line 133, in set_multi_architecture if not self.pkg_cache.is_multi_arch: AttributeError: 'NoneType' object has no attribute 'is_multi_arch'
found the error, is fixed now
20_appcenter.43_installed_apps_check_packages_available_for_i386.test is still failing on all roles: > Fehlermeldung > > Test failed > Standard Ausgabe (STDOUT) > > Gathering packages of installed apps > Adding amd64 architecture > File: /etc/apt/mirror.list > File: /etc/apt/sources.list.d/15_ucs-online-version.list > File: /etc/apt/sources.list.d/20_ucs-online-component.list > �Reading package lists... 0%��Reading package lists... 100%��Reading package lists... Done� > �Building dependency tree... 0%��Building dependency tree... 0%��Building dependency tree... 50%��Building dependency tree... 50%��Building dependency tree � > �Reading state information... 0%��Reading state information... 0%��Reading state information... Done� > Error adding amd64 architecture > > Standard Fehler (STDERR) > > 04.03.15 20:09:51.002 MODULE ( PROCESS ) : Downloading "https://appcenter.software-univention.de/meta-inf/4.0/index.json.gz"... > 04.03.15 20:09:51.171 MODULE ( PROCESS ) : 0 file(s) are new # apt-cache policy ucs-test-appcenter ucs-test-appcenter: Installiert: 5.0.122-6.976.201503031337 Installationskandidat: 5.0.122-6.976.201503031337 Versionstabelle: *** 5.0.122-6.976.201503031337 0 500 http://omar.knut.univention.de/build2/ ucs_4.0-0-errata4.0-1/all/ Packages 100 /var/lib/dpkg/status 5.0.109-1.962.201502090611 0 500 http://univention-repository.knut.univention.de/4.0/unmaintained/ 4.0-1/all/ Packages 5.0.52-1.868.201411181026 0 500 http://univention-repository.knut.univention.de/4.0/unmaintained/ 4.0-0/all/ Packages # ./parse-debian-changelog.py | grep -B1 31792 2015-03-03 09:05:31+01:00 5.0.122-4 * 20_appcenter/43_installed_apps_check_packages_available_for_i386· bugfix (Bug #31792) -- 2015-02-25 11:34:33+01:00 5.0.120-2 * 20_appcenter/43_installed_apps_check_packages_available_for_i386 corrected error during determination of host architecture (Bug #31792) # sed -i 's, > /dev/null,,' /usr/share/ucs-test/20_appcenter/43_installed_apps_check_packages_available_for_i386 # /usr/share/ucs-test/20_appcenter/43_installed_apps_check_packages_available_for_i386 -vvvf ... > Error adding i386 architecture # grep repository/online/architectures /var/log/univention/config-registry.replog 2015-03-05 11:48:26: set repository/online/architectures='i386 amd64' old:[Previously undefined] On the first run _get_other_arch() returns something other then on all following runs. The UCRV should be unset.
Created attachment 6747 [details] Use dpkg --print-architecture Use subprocess Don't >/dev/null
adapted the test according to proposal
I have changed the ##tag and the ##role (role because it checks DefaultPackagesMaster and these are not installed on every role) REOPENED because it does not work. I does not find packages that are amd64 only. I tested it manually with owncloud8's php5-libsmbclient. This is not a dependency so it would not have catched automatically. But it was not catched even after I tweaked some things.