Bug 46993 - Installation of server role package fails due to DNS issues
Installation of server role package fails due to DNS issues
Status: CLOSED FIXED
Product: UCS Test
Classification: Unclassified
Component: Framework
unspecified
Other Linux
: P3 major (vote)
: UCS 4.3
Assigned To: Philipp Hahn
Felix Botner
:
Depends on:
Blocks: 43885 ucs424jenkins
  Show dependency treegraph
 
Reported: 2018-05-10 10:03 CEST by Stefan Gohmann
Modified: 2023-03-25 06:48 CET (History)
4 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
DNS analysis (144.99 KB, application/vnd.oasis.opendocument.spreadsheet)
2018-05-24 17:05 CEST, Philipp Hahn
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2018-05-10 10:03:36 CEST
It looks like several EC2 setups fails in Jenkins due to DNS settings.

For example:
http://jenkins.knut.univention.de:8080/job/UCS-4.3/job/UCS-4.3-0/job/AutotestJoin/SambaVersion=smbtorture,Systemrolle=master/156/

In this case the test fails because univention-license-import is not installed:

[master091] 2018-05-09T23:47:54.723010	+ '[' '!' -e ./ValidTest.license ']'
[master091] 2018-05-09T23:47:54.723010	+ univention-license-import ./ValidTest.license
[master091] 2018-05-09T23:47:54.727146	utils.sh: Zeile 830: univention-license-import: Kommando nicht gefunden.
*** Failed 127: . utils.sh; import_license

The error happens earlier:

[master091] 2018-05-09T23:41:18.180868	=== 10_basis/50uuid (2018-05-09 23:41:17) ===
[master091] 2018-05-09T23:41:18.180868	__NAME__:10_basis/50uuid Generating UUID
[master091] 2018-05-09T23:41:18.331795	Setting uuid/system
[master091] 2018-05-09T23:41:18.331795	File: /usr/share/univention-management-console/meta.json
[master091] 2018-05-09T23:41:18.331795	File: /etc/apt/apt.conf.d/55user_agent
[master091] 2018-05-09T23:41:27.678539	File: /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:41:27.678539	File: /etc/apt/sources.list.d/20_ucs-online-component.list
[master091] 2018-05-09T23:41:36.790497	File: /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:41:36.792241	# An error occurred during the repository check. The error message:
[master091] 2018-05-09T23:41:36.805644	Get:1 file:/var/cache/univention-system-setup/packages ./ InRelease
[master091] 2018-05-09T23:41:36.806156	Ign:1 file:/var/cache/univention-system-setup/packages ./ InRelease
[master091] 2018-05-09T23:41:36.806156	Get:2 file:/var/cache/univention-system-setup/packages ./ Release
[master091] 2018-05-09T23:41:36.806584	Ign:2 file:/var/cache/univention-system-setup/packages ./ Release
[master091] 2018-05-09T23:41:36.807057	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.807057	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.807057	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.807266	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.823626	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.823626	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.823626	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.823626	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.823626	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.823626	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:41:36.835589	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages [1,739 kB]
[master091] 2018-05-09T23:41:36.893470	Ign:4 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ InRelease
[master091] 2018-05-09T23:41:36.922377	Ign:5 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ InRelease
[master091] 2018-05-09T23:41:36.957202	Hit:6 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ Release
[master091] 2018-05-09T23:41:36.985975	Hit:7 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ Release
[master091] 2018-05-09T23:41:37.019517	Ign:8 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/all/ InRelease
[master091] 2018-05-09T23:41:37.048291	Ign:10 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/amd64/ InRelease
[master091] 2018-05-09T23:41:37.076823	Hit:11 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/all/ Release
[master091] 2018-05-09T23:41:37.105524	Hit:13 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/amd64/ Release
[master091] 2018-05-09T23:41:37.250144	Reading package lists...
[master091] 2018-05-09T23:41:37.305233	E: Problem executing scripts APT::Update::Pre-Invoke 'grep -Fqs '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list && ucr commit /etc/apt/sources.list.d/15_ucs-online-version.list ; ! grep -F '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list'
[master091] 2018-05-09T23:41:37.305233	E: Sub-process returned an error code
[master091] 2018-05-09T23:41:37.305766	+ nscd -i hosts
[master091] 2018-05-09T23:41:37.306157	/usr/lib/univention-system-setup/scripts/setup-join.sh: line 109: nscd: command not found
[master091] 2018-05-09T23:41:37.306157	+ cat /etc/resolv.conf
[master091] 2018-05-09T23:41:37.307127	# Warning: This file is auto-generated and might be overwritten by
[master091] 2018-05-09T23:41:37.307127	#          univention-config-registry.
[master091] 2018-05-09T23:41:37.307127	#          Please edit the following file(s) instead:
[master091] 2018-05-09T23:41:37.307127	# Warnung: Diese Datei wurde automatisch generiert und kann durch
[master091] 2018-05-09T23:41:37.307127	#          univention-config-registry ueberschrieben werden.
[master091] 2018-05-09T23:41:37.307127	#          Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en):
[master091] 2018-05-09T23:41:37.307127	# 
[master091] 2018-05-09T23:41:37.307127	# 	/etc/univention/templates/files/etc/resolv.conf
[master091] 2018-05-09T23:41:37.307127	# 
[master091] 2018-05-09T23:41:37.307127	domain	unassigned-domain
[master091] 2018-05-09T23:41:37.307127	nameserver  208.67.222.222
[master091] 2018-05-09T23:41:37.307127	options timeout:2
[master091] 2018-05-09T23:41:37.307257	+ cat /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:41:37.308521	#Warning: This file is auto-generated and might be overwritten by
[master091] 2018-05-09T23:41:37.308521	#         univention-config-registry.
[master091] 2018-05-09T23:41:37.308521	#         Please edit the following file(s) instead:
[master091] 2018-05-09T23:41:37.308521	#Warnung: Diese Datei wurde automatisch generiert und kann durch
[master091] 2018-05-09T23:41:37.308521	#         univention-config-registry ueberschrieben werden.
[master091] 2018-05-09T23:41:37.308521	#         Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en):
[master091] 2018-05-09T23:41:37.308521	#
[master091] 2018-05-09T23:41:37.308521	#	/etc/univention/templates/files/etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:41:37.308521	#
[master091] 2018-05-09T23:41:37.308521	# An error occurred during the repository check. The error message:
[master091] 2018-05-09T23:41:37.308521	#   Traceback (most recent call last):
[master091] 2018-05-09T23:41:37.308521	#     File "<stdin>", line 16, in <module>
[master091] 2018-05-09T23:41:37.308521	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1589, in print_version_repositories
[master091] 2018-05-09T23:41:37.308521	#       for server, ver in self._iterate_version_repositories(start, end, self.parts, archs, dists):
[master091] 2018-05-09T23:41:37.308521	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1445, in _iterate_version_repositories
[master091] 2018-05-09T23:41:37.308521	#       for ver in self._iterate_versions(struct, start, end, parts, archs, server):
[master091] 2018-05-09T23:41:37.308521	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1393, in _iterate_versions
[master091] 2018-05-09T23:41:37.308521	#       code, size, content = server.access(ver)
[master091] 2018-05-09T23:41:37.308521	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 590, in access
[master091] 2018-05-09T23:41:37.308521	#       raise ConfigurationError(uri, reason)
[master091] 2018-05-09T23:41:37.308521	#   ConfigurationError: Configuration error: Temporary failure in name resolution
[master091] 2018-05-09T23:41:37.308521	#
[master091] 2018-05-09T23:41:37.308521	# After fixing this issue, you should rewrite this file with the following command:
[master091] 2018-05-09T23:41:37.308521	#   univention-config-registry commit /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:41:37.308521	#
[master091] 2018-05-09T23:41:37.308658	+ ifconfig
[master091] 2018-05-09T23:41:37.309512	docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
[master091] 2018-05-09T23:41:37.309512	        inet 172.17.42.1  netmask 255.255.0.0  broadcast 0.0.0.0
[master091] 2018-05-09T23:41:37.309512	        ether 02:42:d7:64:3b:6a  txqueuelen 0  (Ethernet)
[master091] 2018-05-09T23:41:37.309512	        RX packets 0  bytes 0 (0.0 B)
[master091] 2018-05-09T23:41:37.309512	        RX errors 0  dropped 0  overruns 0  frame 0
[master091] 2018-05-09T23:41:37.309512	        TX packets 0  bytes 0 (0.0 B)
[master091] 2018-05-09T23:41:37.309512	        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[master091] 2018-05-09T23:41:37.309512	eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
[master091] 2018-05-09T23:41:37.309512	        inet 10.210.144.39  netmask 255.255.0.0  broadcast 10.210.255.255
[master091] 2018-05-09T23:41:37.309512	        inet6 fe80::13ff:feb9:36e0  prefixlen 64  scopeid 0x20<link>
[master091] 2018-05-09T23:41:37.309512	        ether 02:00:13:b9:36:e0  txqueuelen 1000  (Ethernet)
[master091] 2018-05-09T23:41:37.309512	        RX packets 129563  bytes 186397281 (177.7 MiB)
[master091] 2018-05-09T23:41:37.309512	        RX errors 0  dropped 0  overruns 0  frame 0
[master091] 2018-05-09T23:41:37.309512	        TX packets 66813  bytes 5173419 (4.9 MiB)
[master091] 2018-05-09T23:41:37.309512	        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[master091] 2018-05-09T23:41:37.309512	lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
[master091] 2018-05-09T23:41:37.309512	        inet 127.0.0.1  netmask 255.0.0.0
[master091] 2018-05-09T23:41:37.309512	        inet6 ::1  prefixlen 128  scopeid 0x10<host>
[master091] 2018-05-09T23:41:37.309512	        loop  txqueuelen 1  (Local Loopback)
[master091] 2018-05-09T23:41:37.309512	        RX packets 1462  bytes 2509379 (2.3 MiB)
[master091] 2018-05-09T23:41:37.309512	        RX errors 0  dropped 0  overruns 0  frame 0
[master091] 2018-05-09T23:41:37.309512	        TX packets 1462  bytes 2509379 (2.3 MiB)
[master091] 2018-05-09T23:41:37.309512	        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[master091] 2018-05-09T23:41:37.309647	++ ucr get repository/online/server
[master091] 2018-05-09T23:41:37.335098	+ ping -c 4 https://updates.software-univention.de/
[master091] 2018-05-09T23:41:37.376647	ping: https://updates.software-univention.de/: Name or service not known
[master091] 2018-05-09T23:41:37.377034	++ ucr get repository/online/server
[master091] 2018-05-09T23:41:37.402514	+ nslookup https://updates.software-univention.de/
[master091] 2018-05-09T23:41:37.433217	Server:		208.67.222.222
[master091] 2018-05-09T23:41:37.433217	Address:	208.67.222.222#53
[master091] 2018-05-09T23:41:37.433217	** server can't find https://updates.software-univention.de/: NXDOMAIN
[master091] 2018-05-09T23:41:37.433654	+ sleep 60
[master091] 2018-05-09T23:42:37.435069	+ ucr commit /etc/apt/sources.list.d/05univention-system-setup.list /etc/apt/sources.list.d/15_ucs-online-version.list /etc/apt/sources.list.d/20_ucs-online-component.list
[master091] 2018-05-09T23:42:47.833112	File: /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:47.833112	File: /etc/apt/sources.list.d/20_ucs-online-component.list
[master091] 2018-05-09T23:42:47.833717	+ apt-get update
[master091] 2018-05-09T23:42:52.228966	File: /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:52.230512	# An error occurred during the repository check. The error message:
[master091] 2018-05-09T23:42:52.244524	Get:1 file:/var/cache/univention-system-setup/packages ./ InRelease
[master091] 2018-05-09T23:42:52.244784	Ign:1 file:/var/cache/univention-system-setup/packages ./ InRelease
[master091] 2018-05-09T23:42:52.245230	Get:2 file:/var/cache/univention-system-setup/packages ./ Release
[master091] 2018-05-09T23:42:52.245230	Ign:2 file:/var/cache/univention-system-setup/packages ./ Release
[master091] 2018-05-09T23:42:52.245525	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.245525	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.245796	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.245796	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.246008	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.246213	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.246443	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.261415	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.261652	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.261871	Ign:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:42:52.274377	Get:3 file:/var/cache/univention-system-setup/packages ./ Packages [1,739 kB]
[master091] 2018-05-09T23:42:52.329291	Ign:4 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ InRelease
[master091] 2018-05-09T23:42:52.357191	Ign:5 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ InRelease
[master091] 2018-05-09T23:42:52.391421	Hit:6 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ Release
[master091] 2018-05-09T23:42:52.419396	Hit:7 http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ Release
[master091] 2018-05-09T23:42:52.452049	Ign:8 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/all/ InRelease
[master091] 2018-05-09T23:42:52.479557	Ign:10 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/amd64/ InRelease
[master091] 2018-05-09T23:42:52.508286	Hit:11 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/all/ Release
[master091] 2018-05-09T23:42:52.535817	Hit:12 https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/amd64/ Release
[master091] 2018-05-09T23:42:52.689789	Reading package lists...
[master091] 2018-05-09T23:42:52.745107	E: 
[master091] 2018-05-09T23:42:52.745551	Problem executing scripts APT::Update::Pre-Invoke 'grep -Fqs '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list && ucr commit /etc/apt/sources.list.d/15_ucs-online-version.list ; ! grep -F '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list'
[master091] 2018-05-09T23:42:52.745551	E: Sub-process returned an error code
[master091] 2018-05-09T23:42:52.745819	+ cat /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:52.746870	#Warning: This file is auto-generated and might be overwritten by
[master091] 2018-05-09T23:42:52.746870	#         univention-config-registry.
[master091] 2018-05-09T23:42:52.746870	#         Please edit the following file(s) instead:
[master091] 2018-05-09T23:42:52.746870	#Warnung: Diese Datei wurde automatisch generiert und kann durch
[master091] 2018-05-09T23:42:52.746870	#         univention-config-registry ueberschrieben werden.
[master091] 2018-05-09T23:42:52.746870	#         Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en):
[master091] 2018-05-09T23:42:52.746870	#
[master091] 2018-05-09T23:42:52.746870	#	/etc/univention/templates/files/etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:52.746870	#
[master091] 2018-05-09T23:42:52.746870	# An error occurred during the repository check. The error message:
[master091] 2018-05-09T23:42:52.746870	#   Traceback (most recent call last):
[master091] 2018-05-09T23:42:52.746870	#     File "<stdin>", line 16, in <module>
[master091] 2018-05-09T23:42:52.746870	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1589, in print_version_repositories
[master091] 2018-05-09T23:42:52.746870	#       for server, ver in self._iterate_version_repositories(start, end, self.parts, archs, dists):
[master091] 2018-05-09T23:42:52.746870	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1445, in _iterate_version_repositories
[master091] 2018-05-09T23:42:52.746870	#       for ver in self._iterate_versions(struct, start, end, parts, archs, server):
[master091] 2018-05-09T23:42:52.746870	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1389, in _iterate_versions
[master091] 2018-05-09T23:42:52.746870	#       assert server.access(ver)  # patchlevel
[master091] 2018-05-09T23:42:52.746870	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 590, in access
[master091] 2018-05-09T23:42:52.746870	#       raise ConfigurationError(uri, reason)
[master091] 2018-05-09T23:42:52.746870	#   ConfigurationError: Configuration error: Temporary failure in name resolution
[master091] 2018-05-09T23:42:52.746870	#
[master091] 2018-05-09T23:42:52.746870	# After fixing this issue, you should rewrite this file with the following command:
[master091] 2018-05-09T23:42:52.746870	#   univention-config-registry commit /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:52.746870	#
[master091] 2018-05-09T23:42:52.747156	+ cat /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:52.748288	#Warning: This file is auto-generated and might be overwritten by
[master091] 2018-05-09T23:42:52.748288	#         univention-config-registry.
[master091] 2018-05-09T23:42:52.748288	#         Please edit the following file(s) instead:
[master091] 2018-05-09T23:42:52.748288	#Warnung: Diese Datei wurde automatisch generiert und kann durch
[master091] 2018-05-09T23:42:52.748288	#         univention-config-registry ueberschrieben werden.
[master091] 2018-05-09T23:42:52.748288	#         Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en):
[master091] 2018-05-09T23:42:52.748288	#
[master091] 2018-05-09T23:42:52.748288	#	/etc/univention/templates/files/etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:52.748288	#
[master091] 2018-05-09T23:42:52.748288	# An error occurred during the repository check. The error message:
[master091] 2018-05-09T23:42:52.748288	#   Traceback (most recent call last):
[master091] 2018-05-09T23:42:52.748288	#     File "<stdin>", line 16, in <module>
[master091] 2018-05-09T23:42:52.748288	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1589, in print_version_repositories
[master091] 2018-05-09T23:42:52.748288	#       for server, ver in self._iterate_version_repositories(start, end, self.parts, archs, dists):
[master091] 2018-05-09T23:42:52.748288	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1445, in _iterate_version_repositories
[master091] 2018-05-09T23:42:52.748288	#       for ver in self._iterate_versions(struct, start, end, parts, archs, server):
[master091] 2018-05-09T23:42:52.748288	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 1389, in _iterate_versions
[master091] 2018-05-09T23:42:52.748288	#       assert server.access(ver)  # patchlevel
[master091] 2018-05-09T23:42:52.748288	#     File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 590, in access
[master091] 2018-05-09T23:42:52.748288	#       raise ConfigurationError(uri, reason)
[master091] 2018-05-09T23:42:52.748288	#   ConfigurationError: Configuration error: Temporary failure in name resolution
[master091] 2018-05-09T23:42:52.748288	#
[master091] 2018-05-09T23:42:52.748288	# After fixing this issue, you should rewrite this file with the following command:
[master091] 2018-05-09T23:42:52.748288	#   univention-config-registry commit /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:42:52.748288	#
[master091] 2018-05-09T23:42:52.748622	+ sed -i 's|deb cdrom:\[UCS |deb \[trusted=yes\] cdrom:\[UCS |' /etc/apt/sources.list
[master091] 2018-05-09T23:42:52.748913	+ /usr/lib/univention-system-setup/scripts/05_role/10role
[master091] 2018-05-09T23:42:53.379753	=== 05_role/10role (2018-05-09 23:42:52) ===
[master091] 2018-05-09T23:42:53.379753	__NAME__:05_role/10role Configuring server role
[master091] 2018-05-09T23:42:53.379753	__STEPS__:300
[master091] 2018-05-09T23:42:53.380061	__MSG__:Preparing package lists...
[master091] 2018-05-09T23:42:53.489863	__STEP__:0
[master091] 2018-05-09T23:43:06.808967	File: /etc/apt/sources.list.d/15_ucs-online-version.list
[master091] 2018-05-09T23:43:06.810533	# An error occurred during the repository check. The error message:
[master091] 2018-05-09T23:43:06.834525	__MSG__:0% [Working]
[master091] 2018-05-09T23:43:06.834525	__MSG__:Get:1 file:/var/cache/univention-system-setup/packages ./ InRelease
[master091] 2018-05-09T23:43:06.834525	__MSG__:Ign file:/var/cache/univention-system-setup/packages ./ InRelease
[master091] 2018-05-09T23:43:06.834525	__MSG__:8% [Working]
[master091] 2018-05-09T23:43:06.834525	__MSG__:8% [Working]
[master091] 2018-05-09T23:43:06.834525	__MSG__:Get:2 file:/var/cache/univention-system-setup/packages ./ Release
[master091] 2018-05-09T23:43:06.834933	__MSG__:Ign file:/var/cache/univention-system-setup/packages ./ Release
[master091] 2018-05-09T23:43:06.835153	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.835308	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.835658	__MSG__:Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.835658	__MSG__:14% [3 Packages 0 B]
[master091] 2018-05-09T23:43:06.835870	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.841805	__MSG__:Method gave a blank filename
[master091] 2018-05-09T23:43:06.842072	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.842072	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.842318	__MSG__:Get:4 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.842557	__MSG__:14% [4 Packages 0 B]
[master091] 2018-05-09T23:43:06.842859	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.843166	__MSG__:Method gave a blank filename
[master091] 2018-05-09T23:43:06.843166	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.843394	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.843703	__MSG__:Get:5 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.843911	__MSG__:14% [5 Packages 0 B]
[master091] 2018-05-09T23:43:06.844082	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.844239	__MSG__:Method gave a blank filename
[master091] 2018-05-09T23:43:06.844441	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.844714	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.844880	__MSG__:Get:6 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.845065	__MSG__:14% [6 Packages 0 B]
[master091] 2018-05-09T23:43:06.845272	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.845481	__MSG__:Method gave a blank filename
[master091] 2018-05-09T23:43:06.845707	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.845966	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.846123	__MSG__:Get:7 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.846359	__MSG__:14% [7 Packages 0 B]
[master091] 2018-05-09T23:43:06.846567	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.846792	__MSG__:Method gave a blank filename
[master091] 2018-05-09T23:43:06.846991	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.847215	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.847428	__MSG__:Get:8 file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.847650	__MSG__:14% [8 Packages 0 B]
[master091] 2018-05-09T23:43:06.847834	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master091] 2018-05-09T23:43:06.848040	__MSG__:Method gave a blank filename
[master091] 2018-05-09T23:43:06.848277	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.848503	__MSG__:14% [Working]
[master091] 2018-05-09T23:43:06.848689	__MSG__:14% [Connecting to updates-test.software-univention.de]
[master091] 2018-05-09T23:43:06.866757	__MSG__:14% [Connecting to updates-test.software-univention.de (176.9.114.147)]
[master091] 2018-05-09T23:43:06.867703	__MSG__:Get:9 file:/var/cache/univention-system-setup/packages ./ Packages [1,739 kB]
[master091] 2018-05-09T23:43:06.868012	__MSG__:0% [9 Packages 0 B/1,739 kB 0%]
[master091] 2018-05-09T23:43:06.868183	__MSG__:99% [Connecting to updates-test.software-univention.de (176.9.114.147)]
[master091] 2018-05-09T23:43:06.888891	__MSG__:99% [Waiting for headers]
[master091] 2018-05-09T23:43:06.918271	__MSG__:Ign http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ InRelease
[master091] 2018-05-09T23:43:06.918498	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:06.918673	__MSG__:99% [Waiting for headers]
[master091] 2018-05-09T23:43:06.946925	__MSG__:Ign http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ InRelease
[master091] 2018-05-09T23:43:06.947251	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:06.947560	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:06.947796	__MSG__:99% [Waiting for headers]
[master091] 2018-05-09T23:43:06.988997	__MSG__:Hit http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ Release
[master091] 2018-05-09T23:43:06.989240	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:06.989477	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:06.990310	__MSG__:99%
[master091] 2018-05-09T23:43:07.018312	__MSG__:Hit http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ Release
[master091] 2018-05-09T23:43:07.018442	__MSG__:99%
[master091] 2018-05-09T23:43:07.047344	__MSG__:Ign https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/all/ InRelease
[master091] 2018-05-09T23:43:07.047573	__MSG__:99%
[master091] 2018-05-09T23:43:07.072298	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:07.072507	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:07.072811	__MSG__:99%
[master091] 2018-05-09T23:43:07.076272	__MSG__:Ign https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/amd64/ InRelease
[master091] 2018-05-09T23:43:07.076541	__MSG__:99%
[master091] 2018-05-09T23:43:07.076658	__MSG__:99%
[master091] 2018-05-09T23:43:07.106201	__MSG__:Hit https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/all/ Release
[master091] 2018-05-09T23:43:07.106546	__MSG__:99%
[master091] 2018-05-09T23:43:07.106706	__MSG__:99%
[master091] 2018-05-09T23:43:07.135486	__MSG__:Hit https://updates.software-univention.de/4.3/maintained/component 4.3-0-errata/amd64/ Release
[master091] 2018-05-09T23:43:07.135601	__MSG__:99%
[master091] 2018-05-09T23:43:07.151395	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:07.151536	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:07.151760	__MSG__:99%
[master091] 2018-05-09T23:43:07.229424	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:07.229597	__MSG__:99% [Working]
[master091] 2018-05-09T23:43:07.229597	__MSG__:99%
[master091] 2018-05-09T23:43:07.305772	__MSG__:100% [Working]
[master091] 2018-05-09T23:43:07.305967	__MSG__:Fetched 0 B in 0s (0 B/s)
[master091] 2018-05-09T23:43:07.312118	__ERR__:Fetching failed
[master091] 2018-05-09T23:43:07.313179	__ERR__:
[master091] 2018-05-09T23:43:07.313179	__ERR__:Further information regarding this error:
[master091] 2018-05-09T23:43:07.313179	__ERR__:Error: Problem executing scripts APT::Update::Pre-Invoke 'grep -Fqs '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list && ucr commit /etc/apt/sources.list.d/15_ucs-online-version.list ; ! grep -F '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list'.
[master091] 2018-05-09T23:43:07.313415	__ERR__:Error: Sub-process returned an error code.
Comment 1 Stefan Gohmann univentionstaff 2018-05-10 10:04:38 CEST
It looks like it happens only in UCS 4.3.
Comment 4 Stefan Gohmann univentionstaff 2018-05-10 20:02:34 CEST
A first try:

[4.3-0 e6697c76a6] utils.sh: Re-create /etc/resolv.conf during the EC2 setup (Bug #46993)
Comment 5 Stefan Gohmann univentionstaff 2018-05-11 06:16:32 CEST
At least the latest tests are successful.

Debug output:
------------------------------------------------------------------------------
[master091] 2018-05-10T23:35:32.744784	+ cat /etc/resolv.conf
[master091] 2018-05-10T23:35:32.745638	options timeout:2
[master091] 2018-05-10T23:35:32.745638	nameserver 208.67.222.222
[master091] 2018-05-10T23:35:32.745638	nameserver 10.210.0.2
[master091] 2018-05-10T23:35:32.745903	+ ucr commit /etc/resolv.conf
[master091] 2018-05-10T23:35:32.837162	File: /etc/resolv.conf
[master091] 2018-05-10T23:35:32.838430	+ cat /etc/resolv.conf
[master091] 2018-05-10T23:35:32.839553	# Warning: This file is auto-generated and might be overwritten by
[master091] 2018-05-10T23:35:32.839553	#          univention-config-registry.
[master091] 2018-05-10T23:35:32.839553	#          Please edit the following file(s) instead:
[master091] 2018-05-10T23:35:32.839553	# Warnung: Diese Datei wurde automatisch generiert und kann durch
[master091] 2018-05-10T23:35:32.839553	#          univention-config-registry ueberschrieben werden.
[master091] 2018-05-10T23:35:32.839553	#          Bitte bearbeiten Sie an Stelle dessen die folgende(n) Datei(en):
[master091] 2018-05-10T23:35:32.839553	# 
[master091] 2018-05-10T23:35:32.839553	# 	/etc/univention/templates/files/etc/resolv.conf
[master091] 2018-05-10T23:35:32.839553	# 
[master091] 2018-05-10T23:35:32.839553	domain	unassigned-domain
[master091] 2018-05-10T23:35:32.839553	nameserver  208.67.222.222
[master091] 2018-05-10T23:35:32.839553	options timeout:2
------------------------------------------------------------------------------
Maybe 10.210.0.2 is not a full configured nameserver?
Comment 6 Stefan Gohmann univentionstaff 2018-05-11 08:45:57 CEST
It looks really strange:

http://jenkins.knut.univention.de:8080/job/UCS-4.3/job/UCS-4.3-0/job/AutotestJoin/SambaVersion=samba-pdc,Systemrolle=slave/ws/test/

[master098] 2018-05-10T23:47:07.434565	=== 05_role/10role (2018-05-10 23:47:06) ===
[master098] 2018-05-10T23:47:07.434565	__NAME__:05_role/10role Configuring server role
[master098] 2018-05-10T23:47:07.434565	__STEPS__:300
[master098] 2018-05-10T23:47:07.434565	__MSG__:Preparing package lists...
[master098] 2018-05-10T23:47:07.566822	__STEP__:0
[master098] 2018-05-10T23:47:07.702677	__MSG__:0% [Working]
[master098] 2018-05-10T23:47:07.702677	__MSG__:Get:1 file:/var/cache/univention-system-setup/packages ./ InRelease
[master098] 2018-05-10T23:47:07.702677	__MSG__:Ign file:/var/cache/univention-system-setup/packages ./ InRelease
[master098] 2018-05-10T23:47:07.702677	__MSG__:1% [Working]
[master098] 2018-05-10T23:47:07.702677	__MSG__:1% [Working]
[master098] 2018-05-10T23:47:07.702677	__MSG__:Get:2 file:/var/cache/univention-system-setup/packages ./ Release
[master098] 2018-05-10T23:47:07.702677	__MSG__:Ign file:/var/cache/univention-system-setup/packages ./ Release
[master098] 2018-05-10T23:47:07.702677	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.703879	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.704032	__MSG__:Get:3 file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.734176	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.735045	__MSG__:Method gave a blank filename
[master098] 2018-05-10T23:47:07.735045	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.735045	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.735045	__MSG__:Get:4 file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.735045	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.735535	__MSG__:Method gave a blank filename
[master098] 2018-05-10T23:47:07.735535	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.735535	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.735967	__MSG__:Get:5 file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.736210	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.736210	__MSG__:Method gave a blank filename
[master098] 2018-05-10T23:47:07.736644	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.736644	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.736940	__MSG__:Get:6 file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.737208	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.737455	__MSG__:Method gave a blank filename
[master098] 2018-05-10T23:47:07.737723	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.737981	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.737981	__MSG__:Get:7 file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.738307	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.738525	__MSG__:Method gave a blank filename
[master098] 2018-05-10T23:47:07.738759	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.739339	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.739596	__MSG__:Get:8 file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.739848	__MSG__:Err file:/var/cache/univention-system-setup/packages ./ Packages
[master098] 2018-05-10T23:47:07.740189	__MSG__:Method gave a blank filename
[master098] 2018-05-10T23:47:07.740488	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.740683	__MSG__:2% [Working]
[master098] 2018-05-10T23:47:07.741023	__MSG__:2% [Connecting to updates-test.software-univention.de]
[master098] 2018-05-10T23:47:07.741306	__MSG__:2% [Connecting to updates-test.software-univention.de (176.9.114.147)]
[master098] 2018-05-10T23:47:07.741561	__MSG__:Get:9 file:/var/cache/univention-system-setup/packages ./ Packages [1,739 kB]
[master098] 2018-05-10T23:47:07.741717	__MSG__:0% [9 Packages 0 B/1,739 kB 0%]
[master098] 2018-05-10T23:47:07.741936	__MSG__:99% [Connecting to updates-test.software-univention.de (176.9.114.147)]
[master098] 2018-05-10T23:47:07.751072	__MSG__:99% [Waiting for headers]
[master098] 2018-05-10T23:47:07.780996	__MSG__:Ign http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ InRelease
[master098] 2018-05-10T23:47:07.781278	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:07.781490	__MSG__:99% [Waiting for headers]
[master098] 2018-05-10T23:47:07.810444	__MSG__:Ign http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ InRelease
[master098] 2018-05-10T23:47:07.810706	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:07.810946	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:07.811174	__MSG__:99% [Waiting for headers]
[master098] 2018-05-10T23:47:07.850580	__MSG__:Hit http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/all/ Release
[master098] 2018-05-10T23:47:07.850931	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:07.850931	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:07.851187	__MSG__:99% [Waiting for headers]
[master098] 2018-05-10T23:47:07.851886	__MSG__:99%
[master098] 2018-05-10T23:47:07.880229	__MSG__:Hit http://updates-test.software-univention.de/4.3/maintained/component 4.3-0-errata-test/amd64/ Release
[master098] 2018-05-10T23:47:07.880484	__MSG__:99%
[master098] 2018-05-10T23:47:07.929239	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:07.929540	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:07.929762	__MSG__:99%
[master098] 2018-05-10T23:47:08.006837	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:08.507737	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:09.008700	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:09.509862	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:10.011202	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:10.512002	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:11.013194	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:11.514017	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:12.015010	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:12.516029	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:13.017472	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:13.518004	__MSG__:99% [Working]
[master098] 2018-05-10T23:47:14.019421	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.215979	__MSG__:Err https://updates.software-univention.de/4.0/maintained 4.0-0/all/ InRelease
[master098] 2018-05-10T23:47:14.216335	__MSG__:Could not resolve host: updates.software-univention.de
[master098] 2018-05-10T23:47:14.216611	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.216611	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.482662	__MSG__:Ign https://updates.software-univention.de/4.0/maintained 4.0-0/amd64/ InRelease
[master098] 2018-05-10T23:47:14.483007	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.483174	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.511330	__MSG__:Ign https://updates.software-univention.de/4.0/maintained 4.0-1/all/ InRelease
[master098] 2018-05-10T23:47:14.511330	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.511914	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.540252	__MSG__:Ign https://updates.software-univention.de/4.0/maintained 4.0-1/amd64/ InRelease
[master098] 2018-05-10T23:47:14.540643	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.540864	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.569472	__MSG__:Ign https://updates.software-univention.de/4.0/maintained 4.0-2/all/ InRelease
[master098] 2018-05-10T23:47:14.569826	__MSG__:99% [Working]                                                        275 kB/s 0s
[master098] 2018-05-10T23:47:14.570108	__MSG__:99% [Working]                                                        275 kB/s 0s
Comment 7 Stefan Gohmann univentionstaff 2018-05-11 08:48:11 CEST
[4.3-0 c23234ed9e] Make static /etc/hosts entries for our update servers. Workaround for Bug #46993
Comment 8 Philipp Hahn univentionstaff 2018-05-16 21:36:48 CEST
Again UCS-4.2-4/job/AutotestUpgrade failed as the repository server was not reachable:

>:25:03.147744	=== 10_basis/50uuid (2018-05-15 17:25:02) ===
>:25:03.147744	__NAME__:10_basis/50uuid Generating UUID
>:25:03.323033	Setting uuid/system
>:25:03.323033	File: /etc/apt/apt.conf.d/55user_agent

50uuid has finished now and we're back at /usr/lib/univention-system-setup/scripts/setup-join.sh:107

>:25:11.844764	File: /etc/apt/sources.list.d/15_ucs-online-version.list
>:25:11.844764	File: /etc/apt/sources.list.d/20_ucs-online-component.list

these are from "ucr commit"

>:25:18.227568	File: /etc/apt/sources.list.d/15_ucs-online-version.list

where does that 2nd commit on the file again come from?

>:25:18.237818	# An error occurred during the repository check. The error message:
>:25:18.240256	E: Problem executing scripts APT::Update::Pre-Invoke 'grep -Fqs '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list && ucr commit /etc/apt/sources.list.d/15_ucs-online-version.list ; ! grep -F '# An error occurred during the repository check.' /etc/apt/sources.list.d/15_ucs-online-version.list'
>:25:18.240256	E: Sub-process returned an error code
>:25:19.401187	=== 05_role/10role (2018-05-15 17:25:19) ===

Actually this seems to be a problem since 2012 - see Bug #28089
Comment 9 Stefan Gohmann univentionstaff 2018-05-17 06:27:00 CEST
(In reply to Philipp Hahn from comment #8)
> Actually this seems to be a problem since 2012 - see Bug #28089

It didn't happened so often in the last five years as it happens now. So, I think it isn't the same problem.
Comment 10 Philipp Hahn univentionstaff 2018-05-17 13:24:26 CEST
Again many test for UCS-4.2-4 failed last night: We still have no reliable results for that release!

I have the ideas that Amazons' Denial-of-Service prevention is kicking in to protect the AmazonProvidedDNS: <https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html#vpc-dns-limits>

To test this I committed the following changes to UCS-4.2-4 only¹:
[4.2-4] 6752a26c68 Bug #46993 test: Fix utils.sh issues
[4.2-4] 4b8aed25cc Bug #46993 test: Use NAT gateway
  to reduce the load on our VPN/NAT instance and to rule out any issues with that VM
[4.2-4] d080d7df35 Bug #46993 test: Debug DNS resolver issues
  add logging of DNS queries via "iptables -j LOG". This only shows the IP & UDP header, not the DNS query and answer itself. For that we either need tcpdump or ulogd2 or dumpcap.

Doing a simple "ucr commit /etc/apt/sources.list.d/*" shows many DNS queries being generated. As we start many tests with many combinations with many VMs in parallel, this might be considered a DoS.
This also explains why it does not happen if we run the test manually: We don't start that many VMs in parallel.

It gets worse with each UCS release as each one adds yet another repository statement:
  # grep -c ^deb /etc/apt/sources.list.d/15_ucs-online-version.list 
  24

The Python updater code is still using "urllib2", which has no session support AFAIK.
python-requests has support for that: <http://docs.python-requests.org/en/latest/user/advanced/#session-objects>
So each repository statement seems to trigger yet another DNS query.

If that proofs to be true we really should work on <https://trello.com/c/CvfA661E/104-nicht-alle-repos-bis-ucs-40-einbinden> or change the updater to fetch the list of all available UCS releases from <https://trello.com/c/HUCLW2Sm/81-ucs-maintenance-in-umc-anzeigen-zentrale-stelle-f%C3%BCr-meta-daten>.

I will run the test once more for UCS-4.2-4.
Depending on the results¹ the changes can/should be cherry-picked to UCS-4.3-x after that.
Comment 11 Philipp Hahn univentionstaff 2018-05-24 17:05:28 CEST
Created attachment 9538 [details]
DNS analysis

Looking at the DNS queries logged with the iptables rules for UCS-4.2-4 I observe the following behavior: Up to 340 queries are performed in a 10s interval against the *public* DNS server 208.67.222.222, which is one of the OpenDNS¹ resolvers. As this IP is external to the VPC, every request passes through our NAT/OpenVPN instance, where the package gets masqueraded using one of our public IP addresses. So the OpenDNS resolver get all those requests using the same address, which probably gets classified as an DoS attack.

A the AmazonProvidedDNS server is published through DHCP, we really should only use that one, as it
- sees each individual VM with a distinguished IP
- is internal to the VPC

¹: <https://de.wikipedia.org/wiki/OpenDNS>
Comment 12 Philipp Hahn univentionstaff 2018-05-26 09:09:06 CEST
[4.3-0] 99f6e57619 Bug #46993 test: Drop OpenDNS resolver
 Due to Bug #44462 the DHCP provided DNS is only stored in /etc/resolv.conf;
 touching nameserver[123] will trigger UCR to overwrite that file;
 the DHCP provided DNS will be gone then.
 We should permanently remove OpenDNS from our images and use DHCP only.
[4.3-0] 58a40d9395 Bug #46993 test: Fix utils.sh issues
[4.3-0] 310cf77e81 Bug #46993 test: Use NAT gateway
[4.3-0] d64bd25a29 Bug #46993 test: Debug DNS resolver issues
 It everythings works this patch needs to be reverted.
[4.3-0] 80d1555ffb Bug #46993 test: Revert DNS setup
[4.3-0] 6a760f9d4f Bug #46993 test: Revert DNS setup
 These 2 revert the previous static configuration

1st run looks okay: <http://jenkins.knut.univention.de:8080/job/UCS-4.3/job/UCS-4.3-0/job/AutotestJoin/>
Comment 13 Philipp Hahn univentionstaff 2018-05-28 10:53:59 CEST
*** Bug 47000 has been marked as a duplicate of this bug. ***
Comment 14 Philipp Hahn univentionstaff 2018-05-28 11:00:09 CEST
All tests seems to be fine after dropping OpenDNS and only using AmazonProvidedDNS.

I reverted the debugging code and backported the code to drop OpenDNS from UCS-4.3-0 to UCS-4.2-4.

[4.2-4] aa443da7cd Revert "Bug #46993 test: Debug DNS resolver issues"
 test/utils/utils.sh | 11 +++--------
[4.3-0] c4950c047b Revert "Bug #46993 test: Debug DNS resolver issues"
 test/utils/utils.sh | 6 ------
Comment 15 Felix Botner univentionstaff 2018-05-30 13:00:39 CEST
OK