Bug 51655 - Remove LDAP objects no longer supported by UCS-5
Remove LDAP objects no longer supported by UCS-5
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: LDAP
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 5.0
Assigned To: Erik Damrose
Florian Best
: interim-2
Depends on:
Blocks: 31048 51497 51531 51973 52180 52872 52961
  Show dependency treegraph
 
Reported: 2020-07-09 13:13 CEST by Philipp Hahn
Modified: 2021-09-07 09:06 CEST (History)
9 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): API change, Cleanup, Release Goal
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2020-07-09 13:13:04 CEST
For UCS-5 (and earlier) several UDM modules and ACLs for them were removed:
 fatclient
 thinclient
 managedclient
 policy/thinclient
 admin-settings
 …

Their data might still be stored in LDAP. When their ACLs are removed, they might become visible to all users and might leak sensitive data.

We should write a UMC diagnostic module, which 
- similar to "univention-object-type-migrate" looks for unhandled LDAP objects
- looks for known deprecated objects / containers
and provides an option to dump them to a backup file and then delete them.

The script should also be callable from command line.

The script should also be included into the preup.sh script of UCS-5.0-0.

The script should also be included into <https://updates.software-univention.de/download/univention-update-checks/pre-update-checks-5.0> mechanism.

+++ This bug was initially created as a clone of Bug #31048 +++
Comment 1 Florian Best univentionstaff 2020-09-16 17:18:50 CEST
objectClass=univentionSamba4WinsHost Bug #51497
Comment 3 Florian Best univentionstaff 2021-01-26 13:35:19 CET
The following commits have been done so far.

univention-updater (15.0.1-8)
135e11742726 | Bug #51655 up: Improve delete_legacy_objects
4eadbadd3f43 | Bug #51655: Implement delete_legacy_objects
ad2d88025fc2 | Bug #51655 up: Unify preup.sh and pre-check.sh
219d304804c1 | Bug #51655: Bug #51973: add UCC object classes to be removed
911359059199 | Bug #51655: add more objects to be removed
ca984b245487 | Bug #51655: add UCR variable to skip check
a467f09a49ea | Bug #51497: Bug #51973: Bug #31048: Bug #51655: block upgrade if LDAP contains old objects
Comment 4 Sönke Schwardt-Krummrich univentionstaff 2021-01-26 13:43:49 CET
* if still missing, please add an entry to release notes
Comment 5 Erik Damrose univentionstaff 2021-02-02 23:14:52 CET
I think this needs improvement in either UCS 4 or in the preup script leading up to UCS 5. Currently the following objects are detected in every UCS domain, which means every update attempt will fail with the following message in updater.log, and manual steps are required to update to UCS 5:

legacy_objects:
	The following objects are no longer supported with UCS-5:
		dn: cn=default-settings,cn=thinclient,cn=policies,dc=mydomain,dc=intranet
		dn: cn=OpenStack,cn=CloudType,cn=Virtual Machine Manager,dc=mydomain,dc=intranet
	They must be removed before the update can be done.

	See <https://help.univention.com/t/16227> for details.
Comment 6 Philipp Hahn univentionstaff 2021-02-05 09:00:42 CET
(In reply to Erik Damrose from comment #5)
> I think this needs improvement in either UCS 4 or in the preup script
> leading up to UCS 5. Currently the following objects are detected in every
> UCS domain, which means every update attempt will fail with the following
> message in updater.log, and manual steps are required to update to UCS 5:
...
> 	See <https://help.univention.com/t/16227> for details.

Pleas do not delete objects automatically.
Similar to Debians policy for conffiles below /etc/ they should only be deleted automatically if they are unmodified: "Your never should loose data touched manually by an user".
Maybe add an UCRV to make this configurable:
- warn only
- remove unmodified
- remove always
At least make a backup somewhere so they can be restored easily. (with instructions on how to restore them, e.g. "sudo ldapadd -Y GSSAPI -H ldapi:/// -f /var/univention-backup/legacy500.ldif")
Comment 7 Felix Botner univentionstaff 2021-02-16 13:53:14 CET
Two issuse with the update_check_legacy_objects check
 * runs on un-joined systems (
 * does not check if univention-ldapsearch actually works

Checking ldap_schema ...                          OK
Checking legacy_objects ...                       ldap_start_tls: Can't contact LDAP server (-1)
ldap_start_tls: Can't contact LDAP server (-1)
ldap_start_tls: Can't contact LDAP server (-1)
ld...
ldap_start_tls: Can't contact LDAP server (-1)
ldap_start_tls: Can't contact LDAP server (-1)
ldap_start_tls: Can't contact LDAP server (-1)
ldap_start_tls: Can't contact LDAP server (-1)
ldap_start_tls: Can't contact LDAP server (-1)
OK

 -> this should not be OK
Comment 8 Philipp Hahn univentionstaff 2021-02-16 18:18:56 CET
(In reply to Felix Botner from comment #7)
> Two issuse with the update_check_legacy_objects check
>  * runs on un-joined systems (
>  * does not check if univention-ldapsearch actually works
...
>  -> this should not be OK

[5.0-0] c02140eb0f Bug #51655 up: Handle errors in update_check_legacy_objects
 base/univention-updater/debian/changelog |  6 ++++++
 base/univention-updater/script/check.sh  | 13 +++++++++++--
 2 files changed, 17 insertions(+), 2 deletions(-)

Package: univention-updater
Version: 15.0.3-8A~5.0.0.202102161817
Branch: ucs_5.0-0
Comment 9 Erik Damrose univentionstaff 2021-02-22 18:54:21 CET
d3dc28b8eb preup.sh: delete obsolete LDAP objects during update to UCS 5.0

univention-updater 15.0.3-14A~5.0.0.202102221854
Comment 10 Erik Damrose univentionstaff 2021-02-24 18:52:23 CET
3901eb66 Delete obsolete LDAP only on primary dc
univention-updater 15.0.3-16A~5.0.0.202102241851
Comment 11 Felix Botner univentionstaff 2021-02-25 10:04:28 CET
delete_legacy_objects is currently not called in the preup, is that OK?

delete_obsolete_objects is called but on all server roles!

s4 member
Checking term ...                                 OK
Checking usr_mountpoint ...                       OK
Checking valid_machine_credentials ...            OK
preupC4IGSM.sh: Cannot get LDAP credentials from '/etc/ldap.secret'
Error: Update aborted by pre-update script of release 5.0-0
exitcode of univention-updater: 1
ERROR: update failed. Please check /var/log
Comment 12 Erik Damrose univentionstaff 2021-02-25 10:33:52 CET
1d467369 postup.sh: wait for listener postrun
Also fix check for server role, move to correct function.

univention-updater 15.0.3-17A~5.0.0.202102251033
Comment 13 Erik Damrose univentionstaff 2021-03-05 12:32:05 CET
I adapted the help article to reflect the changes to the objectClasses that are detected as blocking the update.

https://help.univention.com/t/16227
Comment 14 Felix Botner univentionstaff 2021-03-09 10:46:46 CET
FAIL - delete_obsolete_objects
made some changes, please have a look - 97ac2b94bf770021bae871ddb45b383a0ed71331
 * check if updateLogDir is set, abort
 * timestamp for backupfile (in case update fails and is restarted)
 * removed >&3 2>&3, if we want to redirect the output we should change the preup
   delete_obsolete_objects  >&3 2>&3
 * -a for tee, otherwise every search overwrites the backup
 * die if ldapdelete fails

But there is still a problem: 
Some of the supposed to be structural objectClass's are in fact auxiliary.

legacy_ocs_structural:
structuralObjectClass=univentionCorporateClient -> aux

obsolete_ocs_structural:
structuralObjectClass=univentionThinClient -> aux
structuralObjectClass=univentionMobileClient -> aux

These filter will never find something and my ucc object is not found
# ucc1, four.five
dn: cn=ucc1,dc=four,dc=five
macAddress: 00:00:00:00:00:00
loginShell: /bin/bash
displayName: ucc1
cn: ucc1
krb5PrincipalName: host/ucc1.four.five@FOUR.FIVE
objectClass: krb5KDCEntry
objectClass: top
objectClass: univentionHost
objectClass: univentionObject
objectClass: krb5Principal
objectClass: person
objectClass: shadowAccount
objectClass: univentionCorporateClient
objectClass: sambaSamAccount
objectClass: posixAccount
univentionCorporateClientBootVariant: overlayfs
univentionObjectType: computers/ucc
uidNumber: 2011
krb5KDCFlags: 126
sambaAcctFlags: [W          ]
krb5MaxRenew: 604800
krb5KeyVersionNumber: 1
sn: ucc1
homeDirectory: /dev/null
uid: ucc1$
sambaSID: S-1-5-21-2862311440-2092257086-639877991-5022
krb5MaxLife: 86400
univentionCorporateClientBootRepartitioning: FALSE
gidNumber: 5007
sambaPrimaryGroupSID: S-1-5-21-2862311440-2092257086-639877991-11015

What do we do about that?

update_check_legacy_objects
 OK - finds str
 OK - finds aux
 OK - aborts update
 OK - bash check.sh delete_legacy_objects
Comment 15 Florian Best univentionstaff 2021-03-15 20:40:01 CET
(In reply to Erik Damrose from comment #13)
> I adapted the help article to reflect the changes to the objectClasses that
> are detected as blocking the update.
> 
> https://help.univention.com/t/16227

The list of object classes is not complete.
Comment 16 Florian Best univentionstaff 2021-03-15 20:40:41 CET
During the upgrade the following error message is printed:
/tmp/tmp13KZeR/postupjCGYIX.sh: Zeile 151: /var/lib/univention-directory-listener/handlers/udm_extension: Datei oder Verzeichnis nicht gefunden                                                                                               
/tmp/tmp13KZeR/postupjCGYIX.sh: Zeile 151: /var/lib/univention-directory-listener/handlers/udm_extension: Datei oder Verzeichnis nicht gefunden
Comment 17 Florian Best univentionstaff 2021-03-15 20:43:16 CET
git:ad2d88025fc24aded2181eae98db21b2adec21ba removed the function fail_if_role_package_will_be_removed.

A new function update_check_role_package_removed exists.

The old function is still called and shows an error message during upgrade:
/tmp/tmp8Ri7un/preupS8z8KG.sh: Zeile 845: fail_if_role_package_will_be_removed: Kommando nicht gefunden.
Comment 18 Felix Botner univentionstaff 2021-03-16 09:57:18 CET
(In reply to Florian Best from comment #15)
> (In reply to Erik Damrose from comment #13)
> > I adapted the help article to reflect the changes to the objectClasses that
> > are detected as blocking the update.
> > 
> > https://help.univention.com/t/16227
> 
> The list of object classes is not complete.

What is missing?

(In reply to Florian Best from comment #16)
> During the upgrade the following error message is printed:
> /tmp/tmp13KZeR/postupjCGYIX.sh: Zeile 151:
> /var/lib/univention-directory-listener/handlers/udm_extension: Datei oder
> Verzeichnis nicht gefunden                                                  
> 
> /tmp/tmp13KZeR/postupjCGYIX.sh: Zeile 151:
> /var/lib/univention-directory-listener/handlers/udm_extension: Datei oder
> Verzeichnis nicht gefunden

yes is see it, updater.log on a replica node

Removing univention-postgresql-9.6 (11.0.1-3A~4.4.0.202004161257) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...
Traceback (most recent call last):
  File "/usr/sbin/univention-pkgdb-scan", line 37, in <module>
    univention.pkgdb.main()
  File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 582, in main
    connection = open_database_connection(config_registry, pkgdbu=False)
  File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 564, in open_database_connection
    connection = pgdb.connect(database=connectstring)
  File "/usr/lib/python2.7/dist-packages/pgdb.py", line 1619, in connect
    cnx = _connect(dbname, dbhost, dbport, dbopt, dbuser, dbpasswd)
pg.InternalError: FATAL:  kein pg_hba.conf-Eintrag für Host »10.207.61.117«, Benutzer »slave075$«, Datenbank »pkgdb«, SSL an

W: --force-yes is deprecated, use one of the options starting with --allow instead.
Custom postupdate script /var/lib/local-postup.sh not found
Object modified: cn=slave075,cn=dc,cn=computers,dc=autotest075,dc=local
listener shutdown done
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
/tmp/tmpyXL_H7/postupawHUnf.sh: line 151: /var/lib/univention-directory-listener/handlers/udm_extension: No such file or directory
repository/online/component/4.4-7-errata-test/description: Preview errata updates for UCS 4.4-7

not sure how this can happen though



(In reply to Florian Best from comment #17)
> git:ad2d88025fc24aded2181eae98db21b2adec21ba removed the function
> fail_if_role_package_will_be_removed.
> 
> A new function update_check_role_package_removed exists.
> 
> The old function is still called and shows an error message during upgrade:
> /tmp/tmp8Ri7un/preupS8z8KG.sh: Zeile 845:
> fail_if_role_package_will_be_removed: Kommando nicht gefunden.

yep, this needs to be fixed
Comment 19 Florian Best univentionstaff 2021-03-16 10:24:59 CET
(In reply to Felix Botner from comment #18)
> (In reply to Florian Best from comment #15)
> > (In reply to Erik Damrose from comment #13)
> > > I adapted the help article to reflect the changes to the objectClasses that
> > > are detected as blocking the update.
> > > 
> > > https://help.univention.com/t/16227
> > 
> > The list of object classes is not complete.
> 
> What is missing?

The obsolete_ocs_structural from base/univention-updater/script/check.sh.
Comment 20 Felix Botner univentionstaff 2021-03-19 12:20:22 CET
see also #51531, seems that this "wait for udm_extension ldap_extension" does not work on members
Comment 21 Florian Best univentionstaff 2021-03-19 13:34:02 CET
(In reply to Florian Best from comment #16)
> During the upgrade the following error message is printed:
> /tmp/tmp13KZeR/postupjCGYIX.sh: Zeile 151:
> /var/lib/univention-directory-listener/handlers/udm_extension: Datei oder
> Verzeichnis nicht gefunden                                                  
> 
> /tmp/tmp13KZeR/postupjCGYIX.sh: Zeile 151:
> /var/lib/univention-directory-listener/handlers/udm_extension: Datei oder
> Verzeichnis nicht gefunden

I fixed that in git:b58844130fda. The reason was that the listener removed the file during re-initialization.
Comment 22 Florian Best univentionstaff 2021-03-23 20:47:12 CET
cn=default-settings,cn=thinclient,cn=policies,$ldap_base is removed but still referenced in:

dn: $ldap_base
univentionPolicyReference: cn=default-settings,cn=thinclient,cn=policies,$ldap_base

and
dn: cn=default containers,cn=univention,$ldap_base
univentionPolicyObject: cn=default-settings,cn=thinclient,cn=policies,$ldap_base
Comment 23 Florian Best univentionstaff 2021-03-31 19:02:01 CEST
(In reply to Felix Botner from comment #18)
> (In reply to Florian Best from comment #17)
> > git:ad2d88025fc24aded2181eae98db21b2adec21ba removed the function
> > fail_if_role_package_will_be_removed.
> > 
> > A new function update_check_role_package_removed exists.
> > 
> > The old function is still called and shows an error message during upgrade:
> > /tmp/tmp8Ri7un/preupS8z8KG.sh: Zeile 845:
> > fail_if_role_package_will_be_removed: Kommando nicht gefunden.
> 
> yep, this needs to be fixed
Fixed:
a08abac125 fixup! Bug #51655 up: Unify preup.sh and pre-check.sh
Comment 24 Erik Damrose univentionstaff 2021-05-11 17:03:07 CEST
9f8725a932 Bug #51655: Also remove policy references for automatically deleted objects
548ed20355 Bug #51655: preup.sh: Fixes and improvements for delete_obsolete_objects

univention-updater 15.0.3-58A~5.0.0.202105111659
Comment 25 Philipp Hahn univentionstaff 2021-05-12 09:08:16 CEST
(In reply to Erik Damrose from comment #24)
> 548ed20355 Bug #51655: preup.sh: Fixes and improvements for delete_obsolete_objects

Why?
You broke it, especially the backup file if now an invalid LDIF file!
Comment 26 Erik Damrose univentionstaff 2021-05-12 09:37:02 CEST
(In reply to Philipp Hahn from comment #25)
> You broke it, especially the backup file if now an invalid LDIF file!

The comment is not really helpful. Please clarify what has been broken, your wording makes it sound like not only the backup logfile is affected?

Regarding the logfile: The file could not be imported directly since the beginning as all attributes are logged, including operational attributes. I suggest we simply rename its ending to avoid disappointed expectations.

2d1fd596 + 4740bd03 preup.sh: Rename logfile created by delete_obsolete_objects
univention-updater 15.0.3-60A~5.0.0.202105120936
Comment 27 Erik Damrose univentionstaff 2021-05-12 12:09:27 CEST
git 1a7fbb3b

Explicitely use structuralObjectClass
Backupfile has ending .ldif
Insert Note about removed policy references as comment in LDIF

univention-updater 15.0.3-61A~5.0.0.202105121204
Comment 28 Florian Best univentionstaff 2021-05-12 14:04:52 CEST
OK: removal of legacy objects (structural object class, object class)
OK: backup ldif file
REOPEN: missing changelog entry
Comment 29 Erik Damrose univentionstaff 2021-05-12 16:23:04 CEST
[5.0-0 5975203a02] Bug #51655: changelog
Comment 30 Florian Best univentionstaff 2021-05-12 16:29:33 CEST
OK: changelog entry
Comment 31 Florian Best univentionstaff 2021-05-25 16:03:20 CEST
UCS 5.0 has been released:
 https://docs.software-univention.de/release-notes-5.0-0-en.html
 https://docs.software-univention.de/release-notes-5.0-0-de.html

If this error occurs again, please use "Clone This Bug".