Univention Bugzilla – Bug 53108
Make UCS build reproducible
Last modified: 2021-05-25 15:58:45 CEST
Use $Timestamp from debian/changelog instead of $NOW: - .po generated for UMC - .png converted from .svg - ...
Isn't there a debhelper which does this automatically?
Created attachment 10692 [details] docker run --rm -ti -u $UID -v $PWD:/work:ro -w /work -v /var/build/temp:/out registry.salsa.debian.org/reproducible-builds/diffoscope --exclude-directory-metadata=recursive (In reply to Florian Best from comment #1) > Isn't there a debhelper which does this automatically? There is `dh_strip_nondeterminism`, but it only handles specific files: /usr/share/perl5/File/StripNondeterminism.pm From the fact, that I created this bug, you can deduce that it is not enough. Test ==== I pushed a branch twice to GitLab and let GitLab build all packages twice. Comparing the result with `diffoscope` shows the following differences - complete report attached: - .po files are normally compiled to .mo files, which `dh_s_nd` processes, but not the .po files itself. They contain a time-stamp from when our "msgmerge" is executed, which differs for two consecutive builds. The traversal order of files is also not stable, so the files are listed in different order inside the .po file. - bin:univention-management-console-module-appcenter contains usr/lib/python2.7/dist-packages/univention/appcenter/actions/umc_update.pyc .pyc files should not be included at all in addition to containing the time stamp of the original .py file. - bin:univention-portal contains differences in usr/share/univention-portal/js/app.js (compiled webapp?) - src:univention-connector-s4 has unstable timestamps in data.tar.xz - src:univention-squid has unstable timestamps in data.tar.xz dh_install internally used `cp -a`, which preserves the timestamp. As `git` does not track time stamps per file a `git checkout` uses $NOW for - *all* files on the initial checkout - for any *changed* file on a subsequent `git checkout` <https://git.wiki.kernel.org/index.php/Git_FAQ#Why_isn.27t_Git_preserving_modification_time_on_files.3F> For Debian packages and UCS packages built via repo-ng the source package is an intermediate .tar file, which preserves the time of import. If you would redo the import at a later time or build directly from a `git checkout` (without the time-preserving intermediate .tar) you will get a different result. Either `git archive $commit` should be used or the timestamp should be set to the time of last change to debian/changelog or last commit: find -exec touch -c -h -d "@${SOURCE_DATE_EPOCH:-$(dpkg-parsechangelog -STimestamp || git log -1 --pretty=%ct)}" {} + Diffing ======= ```sh docker run --rm -ti \ -u $UID \ -v /var/univention/buildsystem2/apt/git:/work:ro \ -w /work \ -v /var/build/temp:/out \ registry.salsa.debian.org/reproducible-builds/diffoscope \ --exclude-directory-metadata=recursive \ --json=/out/diff.json \ --text=/out/diff.text \ phahn-uss[12]/pool/ ```
(In reply to Philipp Hahn from comment #0) > Use $Timestamp from debian/changelog instead of $NOW: > > - .po generated for UMC There are many packages installing `.po` files by accident by calling `dh_install` for a parent directory: ucs-test/ucs-test-framework_10.0.4-57_all.deb /usr/lib/python2.7/dist-packages/univention/testing/selenium/de.po /usr/lib/python3/dist-packages/univention/testing/selenium/de.po test/ucs-test/debian/ucs-test-framework.install ucs-test/ucs-test-selenium_10.0.4-57_all.deb /usr/share/ucs-test/86_selenium/locale/de.po test/ucs-test/debian/ucs-test-selenium.install univention-app-appliance/univention-app-appliance_4.0.0-2_all.deb /usr/share/univention-web/js/umc/hooks/de.po base/univention-app-appliance/debian/univention-app-appliance.install univention-management-console/python-univention-management-console_12.0.11-2_all.deb /usr/lib/python2.7/dist-packages/univention/management/console/de.po management/univention-management-console/debian/python-univention-management-console.install /usr/lib/python3/dist-packages/univention/management/console/de.po management/univention-management-console/debian/python3-univention-management-console.install univention-management-console/univention-management-console-frontend_12.0.11-2_all.deb /usr/share/univention-management-console-frontend/de.po management/univention-management-console/debian/univention-management-console-frontend.install univention-management-console/univention-management-console-login_12.0.11-2_all.deb /usr/share/univention-management-console-login/de.po management/univention-management-console/debian/univention-management-console-login.install univention-saml/univention-saml-schema_7.0.4-5_all.deb /usr/lib/python2.7/dist-packages/univention/admin/handlers/saml/de.po /usr/lib/python3/dist-packages/univention/admin/handlers/saml/de.po saml/univention-saml/debian/univention-saml-schema.install univention-self-service/univention-self-service_5.0.1-5_all.deb /usr/share/univention-self-service/www/de.po management/univention-self-service/debian/univention-self-service.install univention-server-overview/univention-server-overview_3.0.1-1_all.deb /usr/share/univention-server-overview/de.po management/univention-server-overview/debian/univention-server-overview.install univention-system-activation/univention-system-activation_5.0.1-1_all.deb /usr/share/univention-system-activation/de.po base/univention-system-activation/debian/univention-system-activation.install univention-system-setup/univention-system-setup_13.0.3-21_all.deb /usr/share/locale/de/LC_MESSAGES/univention-system-setup-scripts.po /usr/share/univention-system-setup/www/de.po base/univention-system-setup/debian/univention-system-setup.install univention-system-setup/univention-system-setup-boot_13.0.3-21_all.deb /usr/share/univention-system-setup-boot/www/de.po base/univention-system-setup/debian/univention-system-setup-boot.install The only package which should include `.po` files is: univention-web/univention-web-src_4.0.1-24_all.deb /usr/share/univention-web-src/js/de.po /usr/share/univention-web-src/js/en.po management/univention-web/debian/univention-web-src.install PoC @ <https://git.knut.univention.de/univention/ucs/-/merge_requests/93>
OK: Built branch multiple times in a row and compared equal as in comment 2
OK: UMC builds are reproducible again
UCS 5.0 has been released: https://docs.software-univention.de/release-notes-5.0-0-en.html https://docs.software-univention.de/release-notes-5.0-0-de.html If this error occurs again, please use "Clone This Bug".