Bug 36269 - Adapt univention-repository-create to UCS-4.0 DVD layout
Adapt univention-repository-create to UCS-4.0 DVD layout
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Update - Repository administration
UCS 3.2
Other Linux
: P5 normal (vote)
: UCS 3.2-4
Assigned To: Sönke Schwardt-Krummrich
Philipp Hahn
:
: 36293 (view as bug list)
Depends on: 35927
Blocks: 35537 38248 38509 38615
  Show dependency treegraph
 
Reported: 2014-10-22 14:49 CEST by Philipp Hahn
Modified: 2015-05-28 09:32 CEST (History)
2 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2014-10-22 14:49:19 CEST
+++ This bug was initially created as a clone of Bug #35927 +++
A UCS-4.0 DVD has a different layout:
isoinfo -R -i /var/univention/buildsystem2/isotests/ucs_4.0-0-latest-amd64.iso -l
Directory listing of /
dr-xr-xr-x   1    0    0          159744 Oct 22 2014 [     23 02]  all 
dr-xr-xr-x   1    0    0          266240 Oct 22 2014 [    101 02]  amd64 
dr-xr-xr-x   1    0    0            2048 Oct 22 2014 [    231 02]  boot 
lr-xr-xr-x   1    0    0               0 Oct 22 2014 [    468 00]  debian -> .
dr-xr-xr-x   1    0    0            2048 Oct 22 2014 [    248 02]  dists 
dr-xr-xr-x   1    0    0            2048 Oct 22 2014 [    268 02]  .disk 

univention-repository-create must be modified to handle those instead of the previous ones:
/.univention_install
/boot
/profiles
/packages
/dists
Comment 1 Sönke Schwardt-Krummrich univentionstaff 2014-10-24 17:58:51 CEST
*** Bug 36293 has been marked as a duplicate of this bug. ***
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2014-10-24 18:42:25 CEST
univention-repository-create in UCS 4 is now able to handle UCS4-DVDs, too. The support for UCS 3 DVDs has not been removed. The automatic PXE configuration is currently unchanged and therefore broken due to missing required installer files on the DVD. This has to be updated/fixed later on.

Changelog entry for UCS 4 is not required as it will be ported also to UCS 3.2-3.
Comment 3 Sönke Schwardt-Krummrich univentionstaff 2014-11-07 18:04:59 CET
(In reply to Sönke Schwardt-Krummrich from comment #2)
> univention-repository-create in UCS 4 is now able to handle UCS4-DVDs, too.

The support for UCS4-DVDs has also been ported to UCS 3.2-4.

> The support for UCS 3 DVDs has not been removed. The automatic PXE
> configuration is currently unchanged and therefore broken due to missing
> required installer files on the DVD. This has to be updated/fixed later on.

On UCS 3:
univention-repository-create behaves in the old way with UCS 3 DVDs. When
importing UCS4 DVDs the following directories are no longer copied:
"boot", "profiles", "packages"
But the new directories are copied to local repo:
"amd64", "all", "dists"
Kernel and d-i initrd are copied from DVD netboot/{linux,initrd.gz} to
/var/lib/univention-client-boot/installer/<major>.<minor>-<patchlevel>/
and symlinked to /var/lib/univention-client-boot/{linux,initrd.gz}

On UCS 4:
The UCS 4 variant of univention-repository-create behaves the same as the
version of UCS 3.2-4 with one exception:
When importing UCS 3 DVDs the symlinks in /var/lib/univention-client-boot/
are no longer updated as the would refer to the old UCS 3 installer.

Therefore a UCS 4 repository server can only be used to perform a PXE
boot with UCS 4 installer or higher. The old UCS 3 installer is no longer
supported (see also Bug #35537).

> Changelog entry for UCS 4 is not required as it will be ported also to UCS
> 3.2-3.

Changes have been comitted to univention-update in UCS 3.2-4 and UCS 4.0-0.
Changelog entry for UCS 3.2-4 and UCS 4.0-0 have been added.

UCS 3.2-4: univention-updater (9.0.44-1) unstable; urgency=low
UCS 4.0-0: univention-updater (10.0.38-1) unstable; urgency=low

A UCS 3.2-3 repository server create with the UCS 3.2-4 version of
univention-repository-create was able to update a UCS 3.2-2 system via 3.2-3
to 4.0-0.
Comment 4 Philipp Hahn univentionstaff 2014-11-10 12:07:29 CET
OK: r55529 (4.0-0), r55528 (3.2-4)
+               if os.path.isdir( boot_dest ):
+                       shutil.rmtree( boot_dest )
...
+                       if os.path.isdir( profiles_dest ):
+                               shutil.rmtree( profiles_dest )
what happens if this is a file, device node, pipe, socket, broken symlink?

+               os.makedirs(pxedir, mode=0511)
Why this strange mode?

+       if options.major_version >= 4:
Perhaps print a message in the else-case?

OK: univention-updater_9.0.42-5.1273.201409010818
OK: univention-updater_10.0.38-1.1318.201411071756
OK: amd64 i386
FAIL: /var/lib/univention-client-boot
  please use relative paths for initrd.gz and linux as TFTP may use chroot.
  <https://docs.python.org/2.6/library/os.path.html#os.path.relpath>()

FAIL: /var/lib/univention-repository/mirror/4.0/maintained/4.0-0/dists/stable is not a symbolic link
  <https://docs.python.org/2.6/library/shutil.html#shutil.copytree>(symlinks=True)

FAIL: /var/lib/univention-repository/mirror/4.0/maintained/4.0-0/all/p{up,down}.sh are missing

OK: wget -qO- http://localhost/univention-repository/4.0/maintained/4.0-0/dists/ucs400/Release

FAIL: Setting up a UCS-4.0-0 repository on a UCS-3.2 leads to the situation, where the repository-server itself switches over to it's own local repository. As it does not contain UCS-3.2, /etc/apt/sources.list.d/15_ucs-online-version.list becomes empty and no further software installation is possible until the UCRV "repository/online/server" is reset to "updates.software-univention.de" and "repository/online/prefix" is unset.

FAIL:
> An UCS repository must always start with minor version 0, for example
> with UCS 4.0. Please synchronize the repository from 4.0 to 4.0
> by using the tool univention-repository-update.
360 »···»···if not options.minor_version == '0':
str('0') != int(0) ?


FYI: univention-repository-create fails to mount the DVD because of Bug #21995 c3.
Comment 5 Philipp Hahn univentionstaff 2014-11-10 12:16:35 CET
FAIL: # (cd /var/lib/univention-client-boot&&stat -c '%N' *)
„initrd.gz“ -> „/var/lib/univention-client-boot/installer/4.0-0/initrd.gz“
„installer“
„linux“ -> „/var/lib/univention-client-boot/installer/4.0-0/linux“
„linux.bin“ -> „/var/lib/univention-repository/boot/linux-3.10.0.bin“
„linux-server“ -> „/var/lib/univention-repository/boot/linux-3.10.0“

This is after first setting up a 4.0 repo and then adding a 3.2-3 repo DVD.
Which one should univention-tftp pick?
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2014-11-10 14:35:29 CET
(In reply to Philipp Hahn from comment #4)
> OK: r55529 (4.0-0), r55528 (3.2-4)
> +               if os.path.isdir( boot_dest ):
> +                       shutil.rmtree( boot_dest )
> ...
> +                       if os.path.isdir( profiles_dest ):
> +                               shutil.rmtree( profiles_dest )
> what happens if this is a file, device node, pipe, socket, broken symlink?

I left this code path untouched. If boot_dest is no directory, the repo structure does not comply with our defaults → do not delete as it could cause undesired data loss.
 
> +               os.makedirs(pxedir, mode=0511)
> Why this strange mode?

No reason. Removed.

> +       if options.major_version >= 4:
> Perhaps print a message in the else-case?

univention-repository-create of UCS 3.2-4 has an else-case. Added an else case with a warning, that PXE reinstallation feature is not possible with DVDs with UCS 3 or earlier → FIXED

> OK: univention-updater_9.0.42-5.1273.201409010818
> OK: univention-updater_10.0.38-1.1318.201411071756
> OK: amd64 i386
> FAIL: /var/lib/univention-client-boot
>   please use relative paths for initrd.gz and linux as TFTP may use chroot.
>   <https://docs.python.org/2.6/library/os.path.html#os.path.relpath>()

→ FIXED

> FAIL:
> /var/lib/univention-repository/mirror/4.0/maintained/4.0-0/dists/stable is
> not a symbolic link
>  
> <https://docs.python.org/2.6/library/shutil.html#shutil.
> copytree>(symlinks=True)
 
→ FIXED

> FAIL:
> /var/lib/univention-repository/mirror/4.0/maintained/4.0-0/all/p{up,down}.sh
> are missing

→ these files are missing on the UCS 4 DVD
Normally univention-repository-create is only called to create the repo. All further imports should be done via univention-repository-update.
No real problem here.

> OK: wget -qO-
> http://localhost/univention-repository/4.0/maintained/4.0-0/dists/ucs400/
> Release
> 
> FAIL: Setting up a UCS-4.0-0 repository on a UCS-3.2 leads to the situation,
> where the repository-server itself switches over to it's own local
> repository. As it does not contain UCS-3.2,
> /etc/apt/sources.list.d/15_ucs-online-version.list becomes empty and no
> further software installation is possible until the UCRV
> "repository/online/server" is reset to "updates.software-univention.de" and
> "repository/online/prefix" is unset.

→ this problem is not new; I think it can be fixed by adding a UCS 3.2-X-DVD to the repository.

> FAIL:
> > An UCS repository must always start with minor version 0, for example
> > with UCS 4.0. Please synchronize the repository from 4.0 to 4.0
> > by using the tool univention-repository-update.
> 360 »···»···if not options.minor_version == '0':
> str('0') != int(0) ?

→ FIXED

> FYI: univention-repository-create fails to mount the DVD because of Bug
> #21995 c3.

(In reply to Philipp Hahn from comment #5)
> FAIL: # (cd /var/lib/univention-client-boot&&stat -c '%N' *)
> „initrd.gz“ -> „/var/lib/univention-client-boot/installer/4.0-0/initrd.gz“
> „installer“
> „linux“ -> „/var/lib/univention-client-boot/installer/4.0-0/linux“
> „linux.bin“ -> „/var/lib/univention-repository/boot/linux-3.10.0.bin“
> „linux-server“ -> „/var/lib/univention-repository/boot/linux-3.10.0“
> 
> This is after first setting up a 4.0 repo and then adding a 3.2-3 repo DVD.
> Which one should univention-tftp pick?

Under UCS 3 the PXE configuration uses linux{-server,.bin} and under UCS 4 the PXE configuration uses linux and initrd.gz. This way the PXE config will also work when the repo/PXE server is updated to UCS 4.
Comment 7 Sönke Schwardt-Krummrich univentionstaff 2014-11-10 14:37:21 CET
univention-updater (9.0.44-2) unstable; urgency=low
univention-updater (10.0.40-1) unstable; urgency=low
Comment 8 Philipp Hahn univentionstaff 2014-11-10 16:31:55 CET
OK: r55597 (3.2-4)
OK: r55595 (4.0-0)
OK: univention-updater_9.0.44-2.1321.201411101437
OK: 4.0-0 update message is gone
OK: univention-repository-create 3.4-3
OK: univention-repository-create 4.0-0
...
Comment 9 Philipp Hahn univentionstaff 2014-11-11 09:48:09 CET
FAIL: "univention-repositroy-create" on a UCS-3.2-4 system with a UCS-4.0-0 DVD sets the UCRV "repository/mirror/version/start=4.0-0", but leaves "repository/mirror/version/end" unset. The updater then falls back to using the current (3.2-4) version, which leads "/etc/apt/mirror.list" not containing the main UCS repositories: Only the "3.0 errataXXX" repositories and the "3.2-4-errata" component are listed:
# grep ^deb /etc/apt/mirror.list
> deb http://apt.knut.univention.de/3.0/maintained/ errata1/all/
...
> deb http://apt.knut.univention.de/3.0/maintained/ errata140/amd64/
> deb http://apt.knut.univention.de/3.2/maintained/component/ 3.2-3-errata/all/

Manually setting
  ucr set repository/mirror/version/end=4.0-0 repository/mirror/server=apt.knut.univention.de
at least gets 4.0-0 listed, so "univention-repository-update net" can be run afterwards:
# grep ^deb /etc/apt/mirror.list
> deb http://apt.knut.univention.de/4.0/maintained/ 4.0-0/all/
> deb http://apt.knut.univention.de/4.0/maintained/ 4.0-0/amd64/
...
This is essential to get the preup.sh and postup.sh files mirrored!


FAIL: Why are the 3.0-0 errata still mirrored even when starting with 4.0? They're not configure in UCR.
# ucr search --brief --non-empty ^repository/
repository/app_center/installed: DH
repository/app_center/server: appcenter.software-univention.de
repository/mirror/basepath: /var/lib/univention-repository
repository/mirror/recreate_packages: yes
repository/mirror/server: apt.knut.univention.de
repository/mirror/threads: 10
repository/mirror/version/end: 4.0-0
repository/mirror/version/start: 4.0-0
repository/mirror: yes
repository/online/component/3.2-2-errata/description: Errata updates for UCS 3.2-2
repository/online/component/3.2-2-errata/localmirror: true
repository/online/component/3.2-2-errata/version: 3.2
repository/online/component/3.2-2-errata: false
repository/online/component/3.2-3-errata/description: Errata updates for UCS 3.2-3
repository/online/component/3.2-3-errata/version: 3.2
repository/online/component/3.2-3-errata: enabled
repository/online/port: 80
repository/online/prefix: univention-repository
repository/online/server: test35.phahn.qa
repository/online/unmaintained: no
repository/online: yes

...
Comment 10 Sönke Schwardt-Krummrich univentionstaff 2014-11-11 10:40:01 CET
(In reply to Philipp Hahn from comment #9)
> FAIL: "univention-repositroy-create" on a UCS-3.2-4 system with a UCS-4.0-0
> DVD sets the UCRV "repository/mirror/version/start=4.0-0", but leaves
> "repository/mirror/version/end" unset. …

repository/mirror/version/end is now set to DVD version if
- the UCR variable is unset
- the UCR version is smaller than the DVD version

univention-updater (10.0.42-2) unstable; urgency=low
univention-updater (9.0.44-4) unstable; urgency=low
Comment 11 Philipp Hahn univentionstaff 2014-11-11 12:37:12 CET
OK: r55632 3.2-4
OK: r55633 4.0-0
OK: 3.2-4 -> 4.0-0
OK: 4.0-0 -> 3.2.-4
OK: univention-upgrade
OK: univention-repository-upgrade && ls /var/lib/univention-repository/mirror/?.?/maintained/?.?-?/all/p*up.sh
Comment 12 Stefan Gohmann univentionstaff 2014-11-11 15:04:07 CET
UCS 3.2-4 has been released:
 http://docs.univention.de/release-notes-3.2-4-en.html
 http://docs.univention.de/release-notes-3.2-4-de.html

If this error occurs again, please use "Clone This Bug".