Univention Bugzilla – Bug 48424
apt-mirror does not mirror dists/ correctly
Last modified: 2019-02-13 13:27:25 CET
1. apt-mirror defaults to arch=i386 2. URLs are downloaded multiple times
r18401 | Bug #48424: apt-mirror r18402 | Bug #48424: apt-mirror 2 repo_admin.py --cherrypick -p apt-mirror -r 4.3 --releasedest 4.3 --dest errata4.3-3 Package: apt-mirror Version: 0.5.4-1A~4.3.0.201901041140 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] 0d67b2e127 Bug #48424: apt-mirror 0.5.4-1A~4.3.0.201901041140 doc/errata/staging/apt-mirror.yaml | 13 +++++++++++++ 1 file changed, 13 insertions(+)
FYI: (In reply to Philipp Hahn from comment #0) > 1. apt-mirror defaults to arch=i386 If /"etc/apt/mirror.list" contains a line like "deb http://univention-repository.knut.univention.de/4.3/maintained ucs433 main" it would mirror 'i386' by default even on an 'amd64' architecture. This was already fixed with Debian-Stretch (apt-mirror 0.5.4), which was imported for UCS-4.3. During my patch session I worked on the older 0.4.x version, which still had that problem. I only later noticed that it was fixed with 0.5.4 and removed the patch again. > 2. URLs are downloaded multiple times This was the real problem: apt-mirror logs its list of download files/URLs to /var/lib/univention-repository/var/{ALL,NEW}, but that is *NOT* the actual list of files which will be downloaded! It is the list of *missing* files: When multiple Package files reference the same file, it will be logged in that file multiple times using *relative* paths. But after that "apt-mirror" pre-pends the base-URL and and again calls "remove_double_slashes()" for which the old version already worked fine. So again nothing was changed here and you can verify that no URL gets downloaded multiple times by running the following command: cat /var/lib/univention-repository/var/archive-urls.? | sort | uniq -d But you still get the normalized paths in /var/lib/univention-repository/var/{ALL,MD5,SHA1,SHA256} which makes it so much easier to check for duplicate files. 3. As we still only provide gzip compressed Packages files, I disabled the downloading of the bzip2 and xz compressed versions with aea9e7fd33819db2690519a1a41c7d6ed352c83a as this just clutters the log files with errors.
I've updated my test machine with univention-updater (13.0.1-69) and apt-mirror (0.5.4-1A~4.3.0.201901041140). Afterwards, I've executed 'univention-repository-update net' and started the PXE installation of a test system. See screenshots for more information. Test environment: - Repo Server: 10.201.23.1 - PXE client: lattjo stefan_UCS-4.3-0-23.2-Test
Created attachment 9803 [details] Installer-Error.png
Created attachment 9804 [details] Installer-Syslog.png
(In reply to Stefan Gohmann from comment #3) > I've updated my test machine with univention-updater (13.0.1-69) and > apt-mirror (0.5.4-1A~4.3.0.201901041140). Afterwards, I've executed > 'univention-repository-update net' and started the PXE installation of a > test system. The Debian component "main/debian-mirror" was not mirrored. That is added to "/etc/apt/mirror.list" now, too. code @ "git:phahn/46600-pxe" deb @ "deb [trusted=yes] http://10.200.18.180/debian/ phahn46600-pxe main" PS: You can use my "toolshed/REPO/repo-check-Release-valid" to make sure the Release files are valid, e.g. correct hash sums, signatures, .... I use it before each UCS Release: /usr/local/bin/repo-check-Release-valid /var/lib/univention-repository/mirror PPS: Your VM was successfully installed. For verification I added the following hunk to your VM and used "Esc" to boot into the VM: <domain ...> <os> ... <bootmenu enable='yes' timeout='3000'/> </os> ... </domain>
OK, it works now. So, I reopen this issue since it needs to be merged and build as erratum. I think it would be very good if you merge only the necessary changes and not all but I'm unsure if it is possible. Decide it for yourself. I won't be able to do a code review, if you need a code review of the additional changes in univention-updater, please create another bug.
(In reply to Stefan Gohmann from comment #7) > OK, it works now. > > So, I reopen this issue since it needs to be merged and build as erratum. See Bug #46600 comment 4. This Bug is only for "apt-mirror" and the changes are the 2 patched to "apt-mirror". This bug is a pre-requisite of Bug #46600 and prevents downloading files multiple times, but is independent from said bug. doc/errata/staging/apt-mirror.yaml is there and ready. No additional work is needed for UCS-4.4 as the identical package from 4.3 will be used (when copied to 4.4-0 within the release process)
Tests: OK - The PXE installation now also works after a repository update. ucs-test: OK YAML: OK
<http://errata.software-univention.de/ucs/4.3/420.html>