Bug 45467 - user_deletion:delete and user_deletion:expiration do not work as expected
user_deletion:delete and user_deletion:expiration do not work as expected
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: Ucsschool-lib
UCS@school 4.2
Other Linux
: P5 normal (vote)
: UCS@school 4.2 v4
Assigned To: Daniel Tröder
Sönke Schwardt-Krummrich
:
Depends on: 24185 45287
Blocks:
  Show dependency treegraph
 
Reported: 2017-09-28 16:15 CEST by Sönke Schwardt-Krummrich
Modified: 2018-10-30 13:41 CET (History)
2 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 4: Minor Usability: Impairs usability in secondary scenarios
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.206
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional): External feedback
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2017-09-28 16:15:22 CEST
While working on another bug it became clear that the import options user_deletion:delete and user_deletion:expiration do not work as expected, documented or specified.

Status of current implementation:
1) delete=True, expiration=0
→ the user object is removed immediately from LDAP

2) delete=True, expiration=30
→ the user object is not removed, remains active (not deactivated) and the 
  user_expiry date (Kontoablaufdatum) is set to $NOW+30

3) delete=False, expiration=0
→ an exception is raised on purpose

4) delete=False, expiration=30
→ the user object is not removed, immediately deactivated and the user_expiry 
  date (Kontoablaufdatum) is set to $NOW+30

Additional notes:
- Without manual configuration, case 1) is used by default.
- Case 2) requires an additional script that removes users some time in the 
  future; this script does not exist yet!
- Case 4) is broken since deactivation and user_expiry date use (partially) the 
  same LDAP attribute(s), therefore the user is partially deactivated
- The description of these 2 options in the UCS@school admin manual does not 
  match the behavior described above.

During discussion with professional services it became clear that 7-8 school customers are using these flags or are about to use them:
- the actual removal of users is important (→ missing script/cron job)
- case 2 and 4 are both (about to be) in use



The following solution is the result of the discussion:

Instead of the two options above, two new options are implemented:
→ user_deletion:grace_period:deactivation (Integer) (value range >= 0)
→ user_deletion:grace_period:deletion (Integer) (value range >= 0)

Both values specify a period in the unit "days".
By default, both options default to "0" if not set manually in import config file.

New behaviour:     ("user_deletion:grace_period" == "UD:GP")
==============

UD:GP:deactivation=???,  UD:GP:deletion=0
→ the user object is removed immediately from LDAP
→ the value of "UD:GP:deactivation" does not matter in this case

UD:GP:deactivation=0,  UD:GP:deletion=90
→ the user object is not removed
→ the user object is immediately deactivated
→ the new property "purge_timestamp" is set to $NOW+90   (see below)
→ the user object is deleted automatically after 90 days via a cron job script

UD:GP:deactivation=30,  UD:GP:deletion=90
→ the user object is not removed
→ the "user_expiry" date (Kontoablaufdatum) is set to $NOW+30, so the user 
  object remains active/valid for 30 days
→ the new property "purge_timestamp" is set to $NOW+90   (see below)
→ the user object is deleted automatically after 90 days via a cron job script

Implementation:
===============

A) extend the LDAP schema of UCS@school and add a new attribute for UCS@school users that stores a date. Schema suggestion:

attributetype ( .........OID.............
        NAME 'ucsschoolPurgeTimestamp'
		DESC 'Specifies the point in time after which the object is deleted automatically'
        EQUALITY generalizedTimeMatch
        ORDERING generalizedTimeOrderingMatch
        SYNTAX 1.3.6.1.4.1.1466.115.121.1.24
        SINGLE-VALUE )

(derived from krb5PasswordEnd)
The equality and ordering options are important to perform quick lookups of expired user accounts in LDAP.

B) Create an extended attribute (EA) for UCS@school user objects

The EA should use a date syntax: date picker in UMC, ISO8601 date on CLI ("20170929000000Z" ?).
Suggestion for EA's UDM-CLI name: "purge_timestamp"

C) Update at least do_delete() in 
   ucs-school-import/modules/ucsschool/importer/mass_import/user_import.py

do_delete() should implement the behavior described above and use "ucsschoolPurgeTimestamp" to store the date, when the user object shall be removed.
Please make sure, that the calculated purge timestamp is correctly converted to Z-time before the value is stored in LDAP.

Please also make sure that if a previously "deleted" user with given ucsschoolPurgeTimestamp is recreated again via ucs-school-import, the user object has to be reactivated! That means that the account deactivation is reverted (depending on the import data!), the expiration date is removed and the purge timestamp is also removed.

D) Create a purge script

The script should use a proper LDAP filter for searching UCS@school user accounts with ucsschoolPurgeTimestamp<=$NOW, so the LDAP search only returns relevant user objects (and not all!).
After receiving a list of user DNs, the script should use the ucs-school-import lib to open() the user objects and call user.remove() on each object, so hooks and other import logic is also automatically considered.
The script should have a dry run mode. And it should be possible to call the script manually on the command line.

E) cron job

A new cron job has to be implemented that calls the script from D) regulary.
By default, I would suggest that the cron job is executed every night at 4:50 AM.
The cron job interval should be configurable via UCR (see e.g. update/check/cron/entry).

F) test script

The 3 cases described in "New behaviour" shall be covered by one (or more) ucs-test script(s). User reactivation should also be covered by ucs-test. Also the cronjob/script has to be covered by ucs-test.

G) manual update

The import-CLI manual has to be updated accordingly.

Please note: implement everything in a feature branch and DO NOT MERGE it to origin/4.2 for now!
Comment 1 Daniel Tröder univentionstaff 2017-10-06 13:31:51 CEST
Everything was implemented in branch dtroeder/45467_user_deletion.

8ca8bdd1 A), B) add schema, ext attribute and udm_hook for storing a sortable timestamp in LDAP
36043689 B) raise join script version
9d5b7396 G) adapt documentation
92b4841c B) handle unset and already converted timestamps more safely
358b3a89 C) new deactivation, deletion and expiration code
98bbeae7 D) E) script to remove expired user accounts
bbec0edb F) add compatibility for cmdline and ucs-test name for teacher_and_staff role
69c11ce4 F) add purge script test and adapt tests to config changes, 216_import-users_delete_variants now tests the new behavior

No changelog, no advisory, no build.
Please reopen when this should be merged and the above will be created.
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2017-10-14 23:56:32 CEST
During QA I noticed, that "purge_timestamp" for the CLI name of the extended attribute did not match the names of all other extended attributes. I renamed it to ucsschoolPurgeTimestamp, so LDAP attribute and CLI name are identical (similar to all other ext attrs).

I did some manual tests by calling the following command several times, that creates 2 new student accounts and removed the 2 accounts of the previous run:
# /usr/share/ucs-school-import/scripts/ucs-school-testuser-import \
   --staff 0 --students 2 --teachers 0 --staffteachers 0 --classes=1 \
   --inclasses 1 --schools 1 --create-email-addresses gsmitte

Systemdate at time of tests was "Fr 13. Okt 13:00:17 CEST 2017".
Each run has been performed with different settings of deletion_grace_period::deactivation and deletion_grace_period::deletion:

DGP::deactivation=0, DGP::deletion=0
====================================
→ user accounts have been removed immediately

DGP::deactivation=10, DGP::deletion=0
=====================================
→ user accounts have been removed immediately

DGP::deactivation=0, DGP::deletion=10
=====================================
→ user accounts have been deactivated immediately and marked for deletion
 dn: uid=neithard.bar,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local
+ucsschoolPurgeTimestamp: 20171023000000Z
+shadowExpire: 1
+sambaAcctFlags: [UD         ]
-sambaAcctFlags: [U          ]
+krb5KDCFlags: 254
-krb5KDCFlags: 126

DGP::deactivation=5, DGP::deletion=10
=====================================
→ user accounts have been marked for deactivation in 5 days and marked for deletion in 10 days
 dn: uid=soeren.vette,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local
+ucsschoolPurgeTimestamp: 20171023000000Z
+shadowExpire: 17457
+sambaKickoffTime: 1508277600
+krb5ValidEnd: 20171018000000Z
→ Please note: already deactivated user accounts (see previous test runs) remain deactivated and marked for deactivation.

DGP::deactivation=7, DGP::deletion=12345
========================================
→ user accounts have been marked for deactivation in 7 days and marked for deletion in 12345 days
 dn: uid=elina.haske,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local
+ucsschoolPurgeTimestamp: 20510801000000Z
+shadowExpire: 17459
+sambaKickoffTime: 1508450400
+krb5ValidEnd: 20171020000000Z

DGP::deactivation=0, DGP::deletion=12345
========================================
→ user accounts already marked for deactivation in 7 days (see previous test runs) have been immediately deactivated
→ I think, this is ok because an immediate deactivation is requested.
 dn: uid=elina.haske,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local
-shadowExpire: 17459
+shadowExpire: 1
+sambaAcctFlags: [UD         ]
-sambaAcctFlags: [U          ]
+krb5KDCFlags: 254
-krb5KDCFlags: 126

Tested the actual deletion of user accounts with different system dates. ucsschoolPurgeTimestamp was "20171023000000Z".
Systemtime and result:
2017-10-20 21:51:48 INFO  ucs-school-purge-expired-users.main:200  Found 0 expired accounts.
2017-10-22 21:24:56 INFO  ucs-school-purge-expired-users.main:200  Found 0 expired accounts.
2017-10-22 23:50:44 INFO  ucs-school-purge-expired-users.main:200  Found 0 expired accounts.
2017-10-22 23:58:44 INFO  ucs-school-purge-expired-users.main:200  Found 0 expired accounts.
2017-10-23 00:00:03 INFO  ucs-school-purge-expired-users.main:200  Found 4 expired accounts.
→ OK


I did some extra changes and merged the branch to 4.2:

ucs-school-import (15.0.2-1):
ad27cba39e1c | Bug #45467: Merge branch 'sschwardt/45467/42/import-user-deletion' into 4.2
e16579afdf71 | Bug #45467: add changelog entry
92db62d181dd | Bug #45467: cronjob creates two logfiles per day - use a custom logdir
17742816b5b5 | add compatibility for cmdline and ucs-test name for teacher_and_staff role (Bug #45467)
0f1542675839 | Bug #45467: use a custom UCR cron.d template instead of the customer UCR variables
5a3abd5fd88b | script to remove expired user accounts (Bug #45467)
513b1408e9e2 | Bug #45467: do not reactivate user accounts if already disabled
ff3d8a99a3c3 | new deactivation, deletion and expiration code (Bug #45467)
9122bbe12907 | handle unset and already converted timestamps more safely (Bug #45467)
ad1c9a3c9550 | adapt documentation (Bug #45467)
972975b84644 | Bug #45467: raise join script version
8c7045fd08ed | add schema, ext attribute and udm_hook for storing a sortable timestamp in LDAP (Bug #45467)

ucs-test-ucsschool (4.0.4-31.1):
ad27cba39e1c | Bug #45467: Merge branch 'sschwardt/45467/42/import-user-deletion' into 4.2
fc6a117f80ea | add purge script test and adapt tests to config changes (Bug #45467)

ucs-school-import.yaml:
ad27cba39e1c | Bug #45467: Merge branch 'sschwardt/45467/42/import-user-deletion' into 4.2
3d664bf26164 | Bug #45467: add advisory

Package: ucs-school-import
Version: 15.0.2-1A~4.2.0.201710142334
Branch: ucs_4.2-0
Scope: ucs-school-4.2
Comment 3 Sönke Schwardt-Krummrich univentionstaff 2017-10-15 22:03:56 CEST
If the next automatic test is also successful, consider this bug as verified :-)
Comment 4 Sönke Schwardt-Krummrich univentionstaff 2017-10-16 09:17:28 CEST
(In reply to Sönke Schwardt-Krummrich from comment #3)
> If the next automatic test is also successful, consider this bug as verified
> :-)

Jenkins tests showed no errors.
Comment 5 Sönke Schwardt-Krummrich univentionstaff 2017-10-16 21:32:03 CEST
UCS@school 4.2 v4 has been released.

http://docs.software-univention.de/changelog-ucsschool-4.2v4-de.html

If this error occurs again, please clone this bug.