Univention Bugzilla – Bug 49900
Import Debian Buster 10 for UCS-<next>
Last modified: 2024-03-15 10:19:35 CET
Change default architecture to amd64 (Bug #31508) Create release in repo-DB. Test-import Debian Buster 10. Forward-port UCS specific patches. (other Bugs) Adapt UCR templates (Bug #46120)
PYTHONPATH=~/REPOS/repo-ng/src python -m univention.repong.debmirror -s buster -r 4.4-1 --into ucs_5.0-0 --dry-run --save ucs50.save -vvv ... INFO:__main__.process:add 5281 NEW packages INFO:__main__.process:COPY 16007 newer packages w/o patches INFO:__main__.process:MERGE 63 packages with patches INFO:__main__.process:REBUILD 84 packages due build-stamp INFO:__main__.process:CHECK 4 packages with patches INFO:__main__.process:BINNMU 482 packages INFO:__main__.process:KEEP 0 UCS packages INFO:__main__.process:ignore 0 packages with OBSOLETE patches
(In reply to Philipp Hahn from comment #1) > PYTHONPATH=~/REPOS/repo-ng/src python -m univention.repong.debmirror -s > buster -r 4.4-1 --into ucs_5.0-0 --dry-run --save ucs50.save -vvv > ... DONE: > INFO:__main__.process:add 5281 NEW packages > INFO:__main__.process:COPY 16007 newer packages w/o patches TODO: > INFO:__main__.process:MERGE 63 packages with patches > INFO:__main__.process:REBUILD 84 packages due build-stamp > INFO:__main__.process:CHECK 4 packages with patches > INFO:__main__.process:BINNMU 482 packages For now the following packages are taken directly from Debian 10 Buster to update our pbuilder.tgz, which is required for building all other packages: apt 1.8.2 amd64 [1.4.9A~4.3.3.201901230932] base-files 10.3 amd64 [9.9+deb9u9A~4.4.1.201907051639] base-passwd 3.5.46 amd64 [3.5.43A~4.3.0.201711271843] debootstrap 1.0.114 all [1.0.89A~4.4.0.201812141133] e2fslibs 1.44.5-1 amd64 [1.43.4-2A~4.3.0.201801041304] e2fsprogs 1.44.5-1 amd64 [1.43.4-2A~4.3.0.201801041304] libapt-pkg5.0 1.8.2 amd64 [1.4.9A~4.3.3.201901230932] libcomerr2 1.44.5-1 amd64 [1.43.4-2A~4.3.0.201801041304] libpam-modules 1.3.1-5 amd64 [1.1.8-3.6A~4.3.0.201711222052] libpam-modules-bin 1.3.1-5 amd64 [1.1.8-3.6A~4.3.0.201711222052] libpam-runtime 1.3.1-5 all [1.1.8-3.6A~4.3.0.201711222052] libpam0g 1.3.1-5 amd64 [1.1.8-3.6A~4.3.0.201711222052] libss2 1.44.5-1 amd64 [1.43.4-2A~4.3.0.201801041304] libsystemd0 241-5 amd64 [232-25+deb9u11A~4.3.3.201904091044] libudev1 241-5 amd64 [232-25+deb9u11A~4.3.3.201904091044] lsb-base 10.2019051400 all [9.20161125A~4.3.0.201711222238] systemd 241-5 amd64 [232-25+deb9u11A~4.3.3.201904091044] systemd-sysv 241-5 amd64 [232-25+deb9u11A~4.3.3.201904091044] sysvinit-utils 2.93-8 amd64 [2.88dsf-59.9A~4.3.0.201711221818] ...
TODO: - as UCS-5.0 must be self-contained, all "unchanged" packages must be copied from UCS-4.x to UCS-5.0
[4.2] eaf019c3 Bug #49900 debmirror: Fix --into [4.2] 5a053d1b Bug #49900 debmirror: Save to SQLite [4.2] 374837f9 Bug #49900 debmirror: UCS-5 import Debian Buster
Created attachment 10243 [details] qa.py For me, it looks like, the following packages were not imported: $ python qa.py | sort | uniq bgoffice-dict-downloader 0.10 cicero 0.7.2-4 cltl 1.0.30 crafty-bitmaps 1.0-1 crafty-books-medium 1.0.debian1-2 crafty-books-medtosmall 1.0.debian1-2 crafty-books-small 1.0.debian1-2 diaspora-installer 0.7.6.1+debian1 festvox-rablpc16k 2.4-1 fheroes2-pkg 0+svn20150122r3274-2-2 fonts-ibm-plex 1.2.3-2 hts-voice-nitech-jp-atr503-m001 1.05-4 hyperspec 1.32 libdvd-pkg 1.4.2-1-1 linuxbrew-wrapper 20180923-1 mathematica-fonts 21 muttdown 0.3.3-1 nvpy 1.0.0+git20171203.c91062c-1 openrocket 15.03.5 publicfile-installer 0.14 python-vertica 0.8.2-1 repo 1.13.2-1 ruby-graphiql-rails 1.4.10-1 sabnzbdplus 2.3.6+dfsg-1 sqlcl-package 0.1.1 sqldeveloper-package 0.5.4 translate-shell 0.9.6.9-1 uqm-russian 1.0.2-5 vusb-analyzer 1.1-7 winetricks 0.0+20181203-3 zfs-auto-snapshot 1.2.4-2
(In reply to Florian Best from comment #5) > For me, it looks like, the following packages were not imported: > > $ python qa.py | sort | uniq > crafty-bitmaps 1.0-1 > crafty-books-medium 1.0.debian1-2 > crafty-books-medtosmall 1.0.debian1-2 > crafty-books-small 1.0.debian1-2 > fheroes2-pkg 0+svn20150122r3274-2-2 > uqm-russian 1.0.2-5 Those are all "Section: contrib/games", which are not imported for UCS > bgoffice-dict-downloader 0.10 > cicero 0.7.2-4 > cltl 1.0.30 > diaspora-installer 0.7.6.1+debian1 > festvox-rablpc16k 2.4-1 > fonts-ibm-plex 1.2.3-2 > hts-voice-nitech-jp-atr503-m001 1.05-4 > hyperspec 1.32 > libdvd-pkg 1.4.2-1-1 > linuxbrew-wrapper 20180923-1 > mathematica-fonts 21 > muttdown 0.3.3-1 > nvpy 1.0.0+git20171203.c91062c-1 > openrocket 15.03.5 > publicfile-installer 0.14 > python-vertica 0.8.2-1 > repo 1.13.2-1 > ruby-graphiql-rails 1.4.10-1 > sabnzbdplus 2.3.6+dfsg-1 > sqlcl-package 0.1.1 > sqldeveloper-package 0.5.4 > translate-shell 0.9.6.9-1 > vusb-analyzer 1.1-7 > winetricks 0.0+20181203-3 > zfs-auto-snapshot 1.2.4-2 They all were already included in the apt/ucs_5.0-0/*/{Sources,Packages} files, but for some unknown reason were not added to the repo database; manually fixed by importing them via repo_admin.py -F -r 5.0 -p "${pkg}_${ver}.dsc" More packages were not found due to some oversights in your script. I imported the following packaes: repo_admin.py -F -r 5.0 -p b43-fwcutter_019-4.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-amber_4.3.8+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-fors_5.3.32+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-giraf_2.16.3+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-kmos_2.1.0+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-muse_2.6+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-naco_4.4.6+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-uves_5.9.1+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-vimos_3.2.3+dfsg-2.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-visir_4.3.7+dfsg-1.dsc repo_admin.py -F -r 5.0 -p cpl-plugin-xshoo_3.2.0+dfsg-1.dsc repo_admin.py -F -r 5.0 -p dynare_4.5.7-1.dsc repo_admin.py -F -r 5.0 -p festvox-mbrola_1.95-2.dsc repo_admin.py -F -r 5.0 -p google-android-installers_1472023576+nmu3.dsc repo_admin.py -F -r 5.0 -p hashcat-meta_20180828+nmu1.dsc repo_admin.py -F -r 5.0 -p ifeffit_1.2.11d-10.2.dsc repo_admin.py -F -r 5.0 -p janus_0.6.1-1.dsc repo_admin.py -F -r 5.0 -p libgdf_0.1.2-2.1.dsc repo_admin.py -F -r 5.0 -p mali-midgard_16.0+pristine-4.dsc repo_admin.py -F -r 5.0 -p msttcorefonts_3.7.dsc repo_admin.py -F -r 5.0 -p mtj_0.9.14+dfsg-5.dsc repo_admin.py -F -r 5.0 -p pycuda_2018.1.1-3.dsc repo_admin.py -F -r 5.0 -p python-uflash_1.2.4+dfsg-1.dsc repo_admin.py -F -r 5.0 -p sus_7.20180621.dsc repo_admin.py -F -r 5.0 -p zfs-linux_0.7.12-2+deb10u1.dsc Exceptions: repo_admin.py -F -r 5.0 -p gemrb_0.8.5-1.dsc This is "Section: contrib/game", but "gemrb-doc" is "Section: contrib/doc" $ grep-dctrl -s Package,Section -FSource,Package -X gemrb /var/lib/apt/lists/deb.debian.org_debian_dists_buster*_Packages
Created attachment 10268 [details] Fixed qa.py Ignore all "Section: *games" packages Handle packages without "Source" correctly. Handle packages with customized binary version correctly. Skip game "gemrb"
Reopen: The package state still seems inconsistent. I fixed the qa script again, it had a hardcoded packages.gz. # check_package_import_in_repodb.py | wc -l 20674 I tried to rerun the import with --dry-run, similar to comment 1. I wanted to check if the --sql parameter would help, but the script fails even without it in the dry-run: $ PYTHONPATH=~/repo-ng/src python -m univention.repong.debmirror -s buster -r 4.4-3 --into ucs_5.0-0 --dry-run --save ucs50.save -v Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/edamrose/repo-ng/src/univention/repong/debmirror.py", line 1758, in <module> main() File "/home/edamrose/repo-ng/src/univention/repong/debmirror.py", line 1753, in main app.main() File "/home/edamrose/repo-ng/src/univention/repong/debmirror.py", line 151, in main stats = self.process_all() File "/home/edamrose/repo-ng/src/univention/repong/debmirror.py", line 388, in process_all db_cur.execute(SQL_INSERT, (pkg, action, NEW, None, None, None, None, debian_ver, None, NEW)) UnboundLocalError: local variable 'db_cur' referenced before assignment I want to discuss the current state before making any modifications
(In reply to Erik Damrose from comment #8) > Reopen: The package state still seems inconsistent. > > I fixed the qa script again, it had a hardcoded packages.gz. Thanks, I missed that. > # check_package_import_in_repodb.py | wc -l > 20674 I initially forgot to add the option `--sql`, so the packages were imported correctly into 'apt/ucs_5.0-0/', but where not added to the 'repo-ng'-databases, which is used by `repo_stat.py`. I imported all packages, which the QA scripts misses AND wich are already in 'apt/ucs_5.0-0/sources/*.dsc'. But this already is out-of-date, as Debian received two point-updates since my initial import, which the QA scrips also detects as missing, but are not yet imported into UCS-5. PYTHONPATH=~/REPOS/repo-ng/src python -m univention.repong.debmirror -s buster -r 5.0-0 --save ucs50.save -vv -a amd64 PYTHONPATH=~/REPOS/repo-ng/src python -m univention.repong.debmirror -s buster -r 5.0-0 --load ucs50.save -vv -a amd64 --sql --dbout ucs50.sqlite --process COPY repo-check-duplicate-debs2 -s -d /var/univention/buildsystem2/apt/ucs_5.0-0 > I tried to rerun the import with --dry-run, similar to comment 1. I wanted > to check if the --sql parameter would help, but the script fails even > without it in the dry-run: > > $ PYTHONPATH=~/repo-ng/src python -m univention.repong.debmirror -s buster > -r 4.4-3 --into ucs_5.0-0 --dry-run --save ucs50.save -v > Traceback (most recent call last): ... > File "repo-ng/src/univention/repong/debmirror.py", line 388, in process_all > db_cur.execute(SQL_INSERT, (pkg, action, NEW, None, None, None, None, debian_ver, None, NEW)) > UnboundLocalError: local variable 'db_cur' referenced before assignment Fixed: [4.2] 206bb7c0 Bug #49900 debmirror: Fix saving to SQLite
Created attachment 10271 [details] Fixed^2 qa.py Ignore Debian Security Updates Ignore more games Scan all Packages files - not only contrib
OK: all current packages from buster are imported OK~: latest sec-updates still pending, but not relevant for first import OK: QA script can find all packages in repo-ng DB OK: apt/ucs_5.0-0/ - no i386 dir Verified
Important: I forgot the following option in all calls: --override ~/REPOS/repo-ng/mirror/update_ucs44_mirror_from_debian.yml Some game packages (gnome-video-arcade) have been imported because of it, because they are "Section: contrib/gnome".
Created attachment 10275 [details] Fixed^3 qa.py Ignore even more game packages: (('chess.app', '2.8-2')) (('debian-junior', '1.29')) (('fathom', '1.0+git.20190120.0439ca-1')) (('freecell-solver', '5.0.0-2')) (('gnome-video-arcade', '0.8.8-4')) (('haskell-lambdahack', '0.8.3.0-4')) (('haskell-werewolf', '1.5.2.0-4')) (('libkdegames', '4:18.04.0-1')) (('libkf5kmahjongg', '4:17.08.3-1'))