Bug 48424 - apt-mirror does not mirror dists/ correctly
apt-mirror does not mirror dists/ correctly
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Update - Repository administration
UCS 4.3
Other Linux
: P5 normal (vote)
: UCS 4.3-3-errata
Assigned To: Philipp Hahn
Stefan Gohmann
:
Depends on:
Blocks: 46600
  Show dependency treegraph
 
Reported: 2019-01-04 09:47 CET by Philipp Hahn
Modified: 2019-02-13 13:27 CET (History)
1 user (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 6: Setup Problem: Issue for the setup process
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.171
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:
hahn: Patch_Available+


Attachments
Installer-Error.png (31.42 KB, image/png)
2019-01-15 08:41 CET, Stefan Gohmann
Details
Installer-Syslog.png (45.21 KB, image/png)
2019-01-15 08:42 CET, Stefan Gohmann
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2019-01-04 09:47:05 CET
1. apt-mirror defaults to arch=i386
2. URLs are downloaded multiple times
Comment 1 Philipp Hahn univentionstaff 2019-01-04 11:44:35 CET
r18401 | Bug #48424: apt-mirror
r18402 | Bug #48424: apt-mirror 2

repo_admin.py --cherrypick -p apt-mirror -r 4.3 --releasedest 4.3 --dest errata4.3-3

Package: apt-mirror
Version: 0.5.4-1A~4.3.0.201901041140
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] 0d67b2e127 Bug #48424: apt-mirror 0.5.4-1A~4.3.0.201901041140
 doc/errata/staging/apt-mirror.yaml | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Comment 2 Philipp Hahn univentionstaff 2019-01-09 12:52:01 CET
FYI:

(In reply to Philipp Hahn from comment #0)
> 1. apt-mirror defaults to arch=i386

If /"etc/apt/mirror.list" contains a line like "deb http://univention-repository.knut.univention.de/4.3/maintained ucs433 main" it would mirror 'i386' by default even on an 'amd64' architecture.
This was already fixed with Debian-Stretch (apt-mirror 0.5.4), which was imported for UCS-4.3. During my patch session I worked on the older 0.4.x version, which still had that problem. I only later noticed that it was fixed with 0.5.4 and removed the patch again.

> 2. URLs are downloaded multiple times

This was the real problem: apt-mirror logs its list of download files/URLs to /var/lib/univention-repository/var/{ALL,NEW}, but that is *NOT* the actual list of files which will be downloaded! It is the list of *missing* files: When multiple Package files reference the same file, it will be logged in that file multiple times using *relative* paths.

But after that "apt-mirror" pre-pends the base-URL and and again calls "remove_double_slashes()" for which the old version already worked fine.
So again nothing was changed here and you can verify that no URL gets downloaded multiple times by running the following command:
 cat /var/lib/univention-repository/var/archive-urls.? | sort | uniq -d

But you still get the normalized paths in /var/lib/univention-repository/var/{ALL,MD5,SHA1,SHA256} which makes it so much easier to check for duplicate files.

3. As we still only provide gzip compressed Packages files, I disabled the downloading of the bzip2 and xz compressed versions with aea9e7fd33819db2690519a1a41c7d6ed352c83a as this just clutters the log files with errors.
Comment 3 Stefan Gohmann univentionstaff 2019-01-15 08:41:09 CET
I've updated my test machine with univention-updater (13.0.1-69) and apt-mirror (0.5.4-1A~4.3.0.201901041140). Afterwards, I've executed 'univention-repository-update net' and started the PXE installation of a test system.

See screenshots for more information.

Test environment:
 - Repo Server: 10.201.23.1
 - PXE client: lattjo stefan_UCS-4.3-0-23.2-Test
Comment 4 Stefan Gohmann univentionstaff 2019-01-15 08:41:47 CET
Created attachment 9803 [details]
Installer-Error.png
Comment 5 Stefan Gohmann univentionstaff 2019-01-15 08:42:04 CET
Created attachment 9804 [details]
Installer-Syslog.png
Comment 6 Philipp Hahn univentionstaff 2019-01-23 16:20:15 CET
(In reply to Stefan Gohmann from comment #3)
> I've updated my test machine with univention-updater (13.0.1-69) and
> apt-mirror (0.5.4-1A~4.3.0.201901041140). Afterwards, I've executed
> 'univention-repository-update net' and started the PXE installation of a
> test system.

The Debian component "main/debian-mirror" was not mirrored. That is added to "/etc/apt/mirror.list" now, too.

code @ "git:phahn/46600-pxe"
deb @ "deb [trusted=yes] http://10.200.18.180/debian/ phahn46600-pxe main"

PS: You can use my "toolshed/REPO/repo-check-Release-valid" to make sure the Release files are valid, e.g. correct hash sums, signatures, .... I use it before each UCS Release:
  /usr/local/bin/repo-check-Release-valid /var/lib/univention-repository/mirror

PPS: Your VM was successfully installed. For verification I added the following hunk to your VM and used "Esc" to boot into the VM:
  <domain ...>
    <os>
  ...
      <bootmenu enable='yes' timeout='3000'/>
    </os>
  ...
  </domain>
Comment 7 Stefan Gohmann univentionstaff 2019-01-29 07:38:56 CET
OK, it works now.

So, I reopen this issue since it needs to be merged and build as erratum.

I think it would be very good if you merge only the necessary changes and not all but I'm unsure if it is possible. Decide it for yourself.
I won't be able to do a code review, if you need a code review of the additional changes in univention-updater, please create another bug.
Comment 8 Philipp Hahn univentionstaff 2019-02-07 09:27:21 CET
(In reply to Stefan Gohmann from comment #7)
> OK, it works now.
> 
> So, I reopen this issue since it needs to be merged and build as erratum.

See Bug #46600 comment 4.
This Bug is only for "apt-mirror" and the changes are the 2 patched to "apt-mirror". This bug is a pre-requisite of Bug #46600 and prevents downloading files multiple times, but is independent from said bug.

doc/errata/staging/apt-mirror.yaml is there and ready.
No additional work is needed for UCS-4.4 as the identical package from 4.3 will be used (when copied to 4.4-0 within the release process)
Comment 9 Stefan Gohmann univentionstaff 2019-02-08 20:03:59 CET
Tests: OK - The PXE installation now also works after a repository update.

ucs-test: OK

YAML: OK
Comment 10 Erik Damrose univentionstaff 2019-02-13 13:27:25 CET
<http://errata.software-univention.de/ucs/4.3/420.html>