Bug 53034 - PostgreSQL not starting after update to UCS 5.0
PostgreSQL not starting after update to UCS 5.0
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: PostgreSQL
UCS 5.0
Other Linux
: P5 normal (vote)
: UCS 5.0
Assigned To: Arvid Requate
Felix Botner
:
Depends on: 51997
Blocks:
  Show dependency treegraph
 
Reported: 2021-04-07 10:10 CEST by Felix Botner
Modified: 2021-05-25 15:59 CEST (History)
2 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Felix Botner univentionstaff 2021-04-07 10:10:11 CEST
Updated a UCS 4.4-7 primary node with postgresql to UCS 5.0-0.

Rebooted the system (postgresql still runs directly after the update) and now the postgresql service seems to be dead:

-> dpkg -l| grep post
ii  postfix                                             3.4.14-0+deb10u1                                 amd64        High-performance mail transport agent
ii  postfix-ldap                                        3.4.14-0+deb10u1                                 amd64        LDAP map support for Postfix
ii  postfix-sqlite                                      3.4.14-0+deb10u1                                 amd64        SQLite map support for Postfix
ii  postgresql-11                                       11.10-0+deb10u1                                  amd64        object-relational SQL database, version 11 server
ii  postgresql-9.6                                      9.6.20-0+deb9u1                                  amd64        object-relational SQL database, version 9.6 server
ii  postgresql-client                                   11+200+deb10u4                                   all          front-end programs for PostgreSQL (supported version)
ii  postgresql-client-11                                11.10-0+deb10u1                                  amd64        front-end programs for PostgreSQL 11
ii  postgresql-client-9.6                               9.6.20-0+deb9u1                                  amd64        front-end programs for PostgreSQL 9.6
ii  postgresql-client-common                            200+deb10u4                                      all          manager for multiple PostgreSQL client versions
ii  postgresql-common                                   200+deb10u4                                      all          PostgreSQL database-cluster manager
ii  postgresql-contrib-9.6                              9.6.20-0+deb9u1                                  amd64        additional facilities for PostgreSQL
ii  univention-mail-postfix                             14.0.1-1A~5.0.0.202103311206                     all          UCS - postfix configuration
ii  univention-postgresql                               12.0.1-1A~5.0.0.202010151307                     all          UCS - PostgreSQL configuration
ii  univention-postgresql-11                            12.0.1-1A~5.0.0.202010151307                     all          UCS - PostgreSQL 11 configuration
rc  univention-postgresql-9.6                           11.0.1-3A~4.4.0.202004161257                     all          UCS - PostgreSQL 9.6 configuration

systemctl status postgresql@9.6-main.servicepostgresql@9.6-main.service - PostgreSQL Cluster 9.6-main
   Loaded: loaded (/lib/systemd/system/postgresql@.service; enabled-runtime; vendor preset: enabled)
   Active: failed (Result: protocol) since Wed 2021-04-07 09:48:46 CEST; 19min ago
  Process: 504 ExecStart=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 9.6-main start (code=exited, status=1/FAILURE)

Apr 07 09:48:44 master systemd[1]: Starting PostgreSQL Cluster 9.6-main...
Apr 07 09:48:46 master postgresql@9.6-main[504]: The PostgreSQL server failed to start. Please check the log output:
Apr 07 09:48:46 master postgresql@9.6-main[504]: 2021-04-07 09:48:46.306 CEST [717] LOG:  konnte Konfigurationsdatei »/etc/postgresql/9.6/main/pg_hba.conf« nicht öffnen: Datei oder Verzeichnis nicht gefunden
Apr 07 09:48:46 master postgresql@9.6-main[504]: 2021-04-07 09:48:46.307 CEST [717] FATAL:  konnte pg_hba.conf nicht laden
Apr 07 09:48:46 master postgresql@9.6-main[504]: 2021-04-07 09:48:46.323 CEST [717] LOG:  Datenbanksystem ist heruntergefahren
Apr 07 09:48:46 master systemd[1]: postgresql@9.6-main.service: Can't open PID file /run/postgresql/9.6-main.pid (yet?) after start: No such file or directory
Apr 07 09:48:46 master systemd[1]: postgresql@9.6-main.service: Failed with result 'protocol'.
Apr 07 09:48:46 master systemd[1]: Failed to start PostgreSQL Cluster 9.6-main.
root@master:~# systemctl status postgresql@11-main.servicepostgresql@11-main.service
   Loaded: masked (Reason: Unit postgresql@11-main.service is masked.)
   Active: inactive (dead)

So univention-postgresql-9.6 has been removed, but the postgres has not been updated to 11.

This is a blocker and must be fixed.
Comment 1 Arvid Requate univentionstaff 2021-04-13 14:57:34 CEST
Couldn't reproduce during manual update from 4.4-7 (with univention-postgresql-9.6, which has been updated from univention-postgresql-9.4 in the past). The package stays installed and Postgresql runs after the update.

root@master10:~# dpkg -l univention-postgresql-11 univention-postgresql-9.6
Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten
| Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/
         Halb installiert/Trigger erWartet/Trigger anhängig
|/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: GROSS=schlecht)
||/ Name                      Version                      Architektur  Beschreibung
+++-=========================-============================-============-==================================
ii  univention-postgresql-11  12.0.1-1A~5.0.0.202010151307 all          UCS - PostgreSQL 11 configuration
ii  univention-postgresql-9.6 11.0.1-3A~4.4.0.202004161257 all          UCS - PostgreSQL 9.6 configuration


root@master10:~# systemctl status postgresql@9.6-main.servicepostgresql@9.6-main.service - PostgreSQL Cluster 9.6-main
   Loaded: loaded (/lib/systemd/system/postgresql@.service; enabled-runtime; vendor preset: enabled)
   Active: active (running) since Mon 2021-02-01 21:59:23 CET; 16min ago
 Main PID: 14143 (postgres)
    Tasks: 6 (limit: 4691)
   Memory: 136.0M
   CGroup: /system.slice/system-postgresql.slice/postgresql@9.6-main.service
           ├─14143 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
           ├─14145 postgres: checkpointer process   
           ├─14146 postgres: writer process   
           ├─14147 postgres: wal writer process   
           ├─14148 postgres: autovacuum launcher process   
           └─14149 postgres: stats collector process   

Feb 01 21:59:21 master10 systemd[1]: Starting PostgreSQL Cluster 9.6-main...
Feb 01 21:59:23 master10 systemd[1]: Started PostgreSQL Cluster 9.6-main.

root@master10:~# systemctl status postgresql@11-main.servicepostgresql@11-main.service
   Loaded: masked (Reason: Unit postgresql@11-main.service is masked.)
   Active: inactive (dead)
Comment 2 Felix Botner univentionstaff 2021-04-13 22:12:10 CEST
still fails for me

* UCS 4.4-7 primary
* installed nextcloud (which install postgres)
* update to UCS 5.0
* reboot
* su postgres
* psql 
psql: konnte nicht mit dem Server verbinden: Datei oder Verzeichnis nicht gefunden
	Läuft der Server lokal und akzeptiert er Verbindungen
	auf dem Unix-Domain-Socket »/var/run/postgresql/.s.PGSQL.5432«?

ii  postgresql-11                                       11.11-0+deb10u1                                  amd64        object-relational SQL database, version 11 server
ii  postgresql-9.6                                      9.6.20-0+deb9u1                                  amd64        object-relational SQL database, version 9.6 server
ii  univention-postgresql                               12.0.1-1A~5.0.0.202010151307                     all          UCS - PostgreSQL configuration
ii  univention-postgresql-11                            12.0.1-1A~5.0.0.202010151307                     all          UCS - PostgreSQL 11 configuration
rc  univention-postgresql-9.6                           11.0.1-3A~4.4.0.202004161257                     all          UCS - PostgreSQL 9.6 configuration

Problem seems to be that we update the univention-postgresql to "univention-postgresql-11" but not the actual database, postgresql is still in "9.6" mode, but can't be started anymore because the update removes univention-postgresql-9.6

systemctl status postgresql@9.6-main.servicepostgresql@9.6-main.service - PostgreSQL Cluster 9.6-main
   Loaded: loaded (/lib/systemd/system/postgresql@.service; enabled-runtime; vendor preset: enabled)
   Active: failed (Result: protocol) since Tue 2021-04-13 22:00:32 CEST; 6min ago
  Process: 530 ExecStart=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 9.6-main start (code=exited, status=1/FAILURE)

Apr 13 22:00:30 master systemd[1]: Starting PostgreSQL Cluster 9.6-main...
Apr 13 22:00:32 master postgresql@9.6-main[530]: The PostgreSQL server failed to start. Please check the log output:
Apr 13 22:00:32 master postgresql@9.6-main[530]: 2021-04-13 22:00:31.791 CEST [663] LOG:  konnte Konfigurationsdatei »/etc/postgresql/9.6/main/pg_hba.conf« nicht öffnen: Datei oder Verzeichnis nicht gefunden
Apr 13 22:00:32 master postgresql@9.6-main[530]: 2021-04-13 22:00:31.791 CEST [663] FATAL:  konnte pg_hba.conf nicht laden
Apr 13 22:00:32 master postgresql@9.6-main[530]: 2021-04-13 22:00:31.792 CEST [663] LOG:  Datenbanksystem ist heruntergefahren
Apr 13 22:00:32 master systemd[1]: postgresql@9.6-main.service: Can't open PID file /run/postgresql/9.6-main.pid (yet?) after start: No such file or directory
Apr 13 22:00:32 master systemd[1]: postgresql@9.6-main.service: Failed with result 'protocol'.
Apr 13 22:00:32 master systemd[1]: Failed to start PostgreSQL Cluster 9.6-main.

See 10.200.7.90, this is an updated system.
Comment 3 Felix Botner univentionstaff 2021-04-14 10:17:53 CEST
seems that the postup removes univention-postgresql-9.6

Di 13. Apr 16:20:36 CEST 2021
Running postup.sh script:Paketlisten werden gelesen...
Abhängigkeitsbaum wird aufgebaut....
Statusinformationen werden eingelesen....
univention-server-master ist schon die neueste Version (15.0.3-5A~5.0.0.202103311145).
Die folgenden Pakete wurden automatisch installiert und werden nicht mehr benötigt:
  apt-transport-https cpp-6 firmware-atheros firmware-bnx2 firmware-bnx2x
  firmware-brcm80211 firmware-cavium firmware-intelwimax firmware-iwlwifi
  firmware-libertas firmware-myricom firmware-netxen firmware-qlogic
  firmware-realtek firmware-ti-connectivity golang-docker-credential-helpers
  libegl-mesa0 libegl1 libegl1-mesa libevent-2.0-5 libgbm1 libglapi-mesa
  libglvnd0 libisl15 liblvm2app2.2 liblvm2cmd2.02 libmpfr4 libperl5.24
  libsecret-1-0 libsecret-common libwayland-egl1-mesa libwayland-server0
  libxcb-dri2-0 libxcb-dri3-0 libxcb-present0 libxcb-sync1 libxcb-xfixes0
  libxshmfence1 mrtg php-mcrypt php7.0-mcrypt python-decorator python-imaging
  python-ruamel.ordereddict python-support python-yaml python3-dockerpycreds
  runit runit-systemd sysuser-helper sysvinit univention-ifplugd
  univention-postgresql-9.6 univention-skel
Comment 4 Arvid Requate univentionstaff 2021-04-20 13:54:14 CEST
Bug 51997 removed the alternatives from the "Depends" field of univention-postgresql.


d30da5d582 | For updates allow univention-postgresql-9.6 and univention-postgresql-9.4 to fulfill univention-postgresql


I've tested it with univention-postgresql-9.6:

root@prima101:~# dpkg -l univention-postgre*
Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten
| Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/
         Halb installiert/Trigger erWartet/Trigger anhängig
|/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: GROSS=schlecht)
||/ Name                      Version                      Architektur  Beschreibung
+++-=========================-============================-============-==================================
ii  univention-postgresql     12.0.2-1A~5.0.0.202104191825 all          UCS - PostgreSQL configuration
un  univention-postgresql-11  <keine>                      <keine>      (keine Beschreibung vorhanden)
un  univention-postgresql-9.4 <keine>                      <keine>      (keine Beschreibung vorhanden)
ii  univention-postgresql-9.6 11.0.1-3A~4.4.0.202004161257 all          UCS - PostgreSQL 9.6 configuration
         
root@prima101:~# systemctl status postgresql@9.6-main.servicepostgresql@9.6-main.service - PostgreSQL Cluster 9.6-main
   Loaded: loaded (/lib/systemd/system/postgresql@.service; enabled-runtime; vendor preset: enabled)
   Active: active (running) since Tue 2021-04-20 13:17:12 CEST; 31min ago
 Main PID: 3200 (postgres)
    Tasks: 6 (limit: 2350)
   Memory: 21.5M
   CGroup: /system.slice/system-postgresql.slice/postgresql@9.6-main.service
           ├─3200 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
           ├─3202 postgres: checkpointer process   
           ├─3203 postgres: writer process   
           ├─3204 postgres: wal writer process   
           ├─3205 postgres: autovacuum launcher process   
           └─3206 postgres: stats collector process   

Apr 20 13:17:10 prima101 systemd[1]: Starting PostgreSQL Cluster 9.6-main...
Apr 20 13:17:12 prima101 systemd[1]: Started PostgreSQL Cluster 9.6-main.

Interim Bug, no changelog required.
Comment 5 Arvid Requate univentionstaff 2021-04-21 12:09:48 CEST
As discussed:

131eef9a4e | Block update for postgresql-9.4

Package: univention-updater
Version: 15.0.3-41A~5.0.0.202104211207
Branch: ucs_5.0-0
Comment 6 Felix Botner univentionstaff 2021-04-21 16:50:54 CEST
OK - update with postgres 9.6
OK - update with postgres 9.4 is blocked
OK - migration from 9.4 to 9.6 (https://help.univention.com/t/8075)
OK - update with nextcloud (and postgres 9.6)
Comment 7 Florian Best univentionstaff 2021-05-25 15:59:19 CEST
UCS 5.0 has been released:
 https://docs.software-univention.de/release-notes-5.0-0-en.html
 https://docs.software-univention.de/release-notes-5.0-0-de.html

If this error occurs again, please use "Clone This Bug".