Bug 43341 - Update to UCS 4.2 failed: ImportError: No module named pipes
Update to UCS 4.2 failed: ImportError: No module named pipes
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: General
UCS 4.2
Other Linux
: P5 normal (vote)
: UCS 4.1-4-errata
Assigned To: Philipp Hahn
Stefan Gohmann
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-01-16 09:38 CET by Stefan Gohmann
Modified: 2017-02-15 14:57 CET (History)
2 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
updater.log (943.47 KB, text/plain)
2017-02-03 21:47 CET, Stefan Gohmann
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2017-01-16 09:38:06 CET
Update to UCS 4.2 failed:

I can't access the system anymore but I was able to copy the following updater.log part:

Vorbereitung zum Ersetzen von hpijs-ppds 3.12.6-3.1.23.201403191745 (durch .../hpijs-ppds_3.14.6-1_all.deb) ...
Ersatz für hpijs-ppds wird entpackt ...
Trigger für man-db werden verarbeitet ...
Trigger für install-info werden verarbeitet ...
Trigger für cracklib-runtime werden verarbeitet ...
Trigger für postgresql-common werden verarbeitet ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
  de_de
insserv: warning: script 'K01univention-directory-listener' missing LSB tags and overrides
insserv: warning: script 'K01univention-directory-notifier' missing LSB tags and overrides
insserv: warning: script 'K02bind9' missing LSB tags and overrides
insserv: warning: script 'K01univention-firewall' missing LSB tags and overrides
insserv: warning: script 'K01univention-system-setup-boot' missing LSB tags and overrides
insserv: warning: script 'K01univention-management-console-web-server' missing LSB tags and overrides
insserv: warning: script 'K01univention-management-console-server' missing LSB tags and overrides
insserv: warning: script 'K01univention-saml' missing LSB tags and overrides
insserv: warning: script 'K01univention-welcome-screen' missing LSB tags and overrides
insserv: warning: script 'K01univention-network-common' missing LSB tags and overrides
insserv: warning: script 'K01univention-runit' missing LSB tags and overrides
insserv: warning: script 'K01rdate' missing LSB tags and overrides
insserv: warning: script 'K01univention-maintenance' missing LSB tags and overrides
insserv: warning: script 'K01univention-directory-policy' missing LSB tags and overrides
insserv: warning: script 'S02resize2fs' missing LSB tags and overrides
insserv: warning: script 'univention-management-console-server' missing LSB tags and overrides
insserv: warning: script 'univention-directory-notifier' missing LSB tags and overrides
insserv: warning: script 'univention-system-setup-boot' missing LSB tags and overrides
insserv: warning: script 'univention-saml' missing LSB tags and overrides
insserv: warning: script 'univention-directory-listener' missing LSB tags and overrides
insserv: warning: script 'univention-network-common' missing LSB tags and overrides
insserv: warning: script 'rdate' missing LSB tags and overrides
insserv: warning: script 'univention-firewall' missing LSB tags and overrides
insserv: warning: script 'univention-maintenance' missing LSB tags and overrides
insserv: warning: script 'klogd' missing LSB tags and overrides
insserv: warning: script 'univention-welcome-screen' missing LSB tags and overrides
insserv: warning: script 'univention-system-setup-boot-prepare' missing LSB tags and overrides
insserv: warning: script 'univention-runit' missing LSB tags and overrides
insserv: warning: script 'univention-directory-policy' missing LSB tags and overrides
insserv: warning: script 'bind9' missing LSB tags and overrides
insserv: warning: script 'univention-management-console-web-server' missing LSB tags and overrides
insserv: warning: script 'resize2fs' missing LSB tags and overrides
Traceback (most recent call last):
  File "/usr/sbin/univention-config-registry", line 45, in <module>
    import univention.config_registry as ub
  File "/usr/lib/pymodules/python2.7/univention/config_registry/__init__.py", line 39, in <module>
    from univention.config_registry.filters import \
  File "/usr/lib/pymodules/python2.7/univention/config_registry/filters.py", line 33, in <module>
    from pipes import quote as escape_value
ImportError: No module named pipes
invoke-rc.d: initscript postgresql, action "start" failed.
dpkg: Fehler beim Bearbeiten von postgresql-common (--unpack):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück

root@master070:~# /etc/init.d/cups restart
[....] Restarting Common Unix Printing System: cupsdTraceback (most recent call last):
  File "/usr/sbin/univention-config-registry", line 45, in <module>
    import univention.config_registry as ub
  File "/usr/lib/pymodules/python2.7/univention/config_registry/__init__.py", line 39, in <module>
    from univention.config_registry.filters import \
  File "/usr/lib/pymodules/python2.7/univention/config_registry/filters.py", line 33, in <module>
    from pipes import quote as escape_value
ImportError: No module named pipes
Comment 2 Philipp Hahn univentionstaff 2017-01-17 10:23:42 CET
During the update Python is updated and temporarily not configured:

/var/lib/dpkg/info/postgresql-common.postinst:89         invoke-rc.d postgresql start || exit $?
/etc/init.d/postgresql:34            check_autostart postgres8 postgres8/autostart
/usr/share/univention-config-registry/init-autostart.lib:16     autostart=$("$ucr" get "$variable")
/usr/lib/pymodules/python2.7/univention/config_registry/filters.py:33 from pipes import quote as escape_value

The problem here is, that univention-* packages use the UCR template mechanism to replace files from other packages (here: postgresql-common) adding dependencies on third-party packages (here: UCR), which are themselves not tracked using the Debian-"Depends" mechanism for packages. Thus for APT it is okay to de-configure Python and PostgreSQL as the same-time and to upgrade PostgreSQL *before* Python, as there is no dependency from PostgreSQL on Python. But with UCS there now is the implicit dependency through UCR and the upgrade fails.

Making python2.7-minimal "Essential: yes" might be one option, but read <https://www.debian.org/doc/debian-policy/ch-binary.html#s3.8> carefully as this has severe consequences.
Then pipes.py (and probably others) need to be moved to libpython2.7-minimal (or must be re-implemented / copied with proper attribution).
The main problem might probably be, that we're still using pysupport (Bug #28497), which is incompatible with the requirement, that packages must work when still "unconfigured".
This also would pull in additional packages:
 # dpkg -s python-univention-config-registry
 Depends: python (<< 2.8), python (>= 2.6), python-support (>= 0.90.0), python-ipaddr, ifupdown
which might create even more severe upgrade problems (in the future).
(Currently the following packages are marked as essential: aptitude search '~E')


Another option might be to declare "python2.7" and "python-univention-config-registry" as a Pre-Depends somewhere clever™, as that would force APT to update Python in a separate step.


The bug is actually triggered by the update of hunspell (or some other dictionary), which triggers the interest of PostgreSQL. As the #DEBHELPER# block is also called for "$1"=triggered, the PostgreSQL service gets started and leads to the error situation from above

/var/lib/dpkg/info/postgresql-common.triggers:1
 interest /usr/share/myspell/dicts
 interest /usr/share/hunspell
/var/lib/dpkg/info/postgresql-common.postinst:82
 if [ "$1" = triggered ]; then
    pg_updatedicts || true
 fi
 # Automatically added by dh_installinit
 if [ -x "/etc/init.d/postgresql" ]; then
        update-rc.d postgresql defaults 19 21 >/dev/null
        invoke-rc.d postgresql start || exit $?
 fi


Manually configuring Python fixes the problem:

# dpkg -l \*python\* | grep ^i[A-Z]
iU  libpython2.7:amd64                                  2.7.9-2+deb8u1
iU  libpython2.7-minimal:amd64                          2.7.9-2+deb8u1
iU  python2.7                                           2.7.9-2+deb8u1
iU  python2.7-minimal                                   2.7.9-2+deb8u1

# dpkg --configure libpython2.7 libpython2.7-minimal python2.7 python2.7-minimal
dpkg: Abhängigkeitsprobleme verhindern Konfiguration von libpython2.7:amd64:                                        
 libpython2.7:amd64 hängt ab von libpython2.7-stdlib (= 2.7.9-2+deb8u1); aber:
  Paket libpython2.7-stdlib ist nicht installiert

# dpkg -i libpython2.7-stdlib_2.7.9-2+deb8u1_amd64.deb 
...
dpkg: Abhängigkeitsprobleme verhindern Konfiguration von libpython2.7-stdlib:amd64:
 libpython2.7-stdlib:amd64 hängt ab von libexpat1 (>= 2.1~beta3); aber:
  Paket libexpat1:amd64 ist noch nicht konfiguriert.
 libpython2.7-stdlib:amd64 hängt ab von libffi6 (>= 3.0.4); aber:
  Paket libffi6:amd64 ist noch nicht konfiguriert.
 libpython2.7-stdlib:amd64 hängt ab von libsqlite3-0 (>= 3.5.9); aber:
  Paket libsqlite3-0:amd64 ist noch nicht konfiguriert.

# dpkg --configure libexpat1 libffi6 libsqlite3-0 libpython2.7-stdlib
Comment 3 Philipp Hahn univentionstaff 2017-02-03 15:13:13 CET
r76400 | Bug #43341 ucr: Provide fallback quote() implementation
 The underlying is much more complex, but let's take the boss' road

Package: univention-config-registry
Version: 12.0.1-2A~4.2.0.201702031507
Branch: ucs_4.2-0
Comment 4 Stefan Gohmann univentionstaff 2017-02-03 21:47:28 CET
Created attachment 8399 [details]
updater.log
Comment 5 Stefan Gohmann univentionstaff 2017-02-03 22:02:21 CET
The update still fails, see updater.log.

>  The underlying is much more complex, but let's take the boss' road

Since, we don't want a preup update of UCR, I think it would be the best to merge your changes to UCS 4.1-4-errata. If you see other solutions, feel free to make suggestions.
Comment 6 Philipp Hahn univentionstaff 2017-02-10 16:38:50 CET
(In reply to Stefan Gohmann from comment #5)
> The update still fails, see updater.log.
> 
> >  The underlying is much more complex, but let's take the boss' road
> 
> Since, we don't want a preup update of UCR, I think it would be the best to
> merge your changes to UCS 4.1-4-errata. If you see other solutions, feel
> free to make suggestions.

My tests with an pre-updated version of UCR with this patch have been fine so far.
I've merged the change ucs-4.2-0/76608 back to UCS-4.1-4 as ucs-4.1-4/76608.
I'm changing this bug to to errata4.1-4, as this change must be done before the upgrade to 4.2-0 and the change is the same for both releases.


r76608 | Bug #43341 ucr: Provide fallback quote() implementation

Package: univention-config-registry
Version: 11.0.0-10.506.201702101631
Branch: ucs_4.1-0
Scope: errata4.1-4

r76609 | Bug #43341 ucr: Provide fallback quote() implementation YAML
Comment 7 Stefan Gohmann univentionstaff 2017-02-13 07:03:33 CET
r76617:
 ucs-ec2-tools/examples/jenkins
  Upgrade to test errata before updating to UCS 4.2 in order to test Bug #43341
Comment 8 Stefan Gohmann univentionstaff 2017-02-13 09:52:13 CET
(In reply to Stefan Gohmann from comment #7)
> r76617:
>  ucs-ec2-tools/examples/jenkins
>   Upgrade to test errata before updating to UCS 4.2 in order to test Bug
> #43341

It looks like upgrade to upgrade_to_latest_test_errata does not work here:

Execute: . utils.sh; upgrade_to_latest_test_errata

Starting univention-upgrade. Current UCS version is 4.1-4 errata389

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           none
Execute: . utils.sh; jenkins_updates && touch UCS_UPDATED
Starting from 4.1-4+389 to 4.2-0..
Comment 9 Stefan Gohmann univentionstaff 2017-02-14 07:44:52 CET
The update gets a step further and fails at a different point: Bug #43534. But this works now.

(In reply to Stefan Gohmann from comment #8)
> It looks like upgrade to upgrade_to_latest_test_errata does not work here:
> 
> Execute: . utils.sh; upgrade_to_latest_test_errata
> 
> Starting univention-upgrade. Current UCS version is 4.1-4 errata389
> 
> Checking for local repository:                          none
> Checking for package updates:                           none
> Checking for app updates:                               none
> Checking for release updates:                           none
> Execute: . utils.sh; jenkins_updates && touch UCS_UPDATED
> Starting from 4.1-4+389 to 4.2-0..

The script activate-errata-test-scope.sh was not copied, fixed via r76641.


YAML: OK

Code review: OK

UCS 4.2 merge: OK
Comment 10 Janek Walkenhorst univentionstaff 2017-02-15 14:57:45 CET
<http://errata.software-univention.de/ucs/4.1/390.html>