Bug 57194 - Error 20: Authentication token manipulation error until UMC restart
Summary: Error 20: Authentication token manipulation error until UMC restart
Status: CLOSED FIXED
Alias: None
Product: UCS
Classification: Unclassified
Component: UMC - Change password
Version: UCS 5.0
Hardware: Other Linux
: P5 normal
Target Milestone: UCS 5.0-8-errata
Assignee: Ole Schwiegert
QA Contact: Dirk Wiesenthal
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-03-26 19:39 CET by Julia Bremer
Modified: 2024-07-23 18:03 CEST (History)
8 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 5: Will affect all installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.429
Enterprise Customer affected?: Yes
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2024011821000381, 2023121521000279, 2024051721000107, 2024052921000077, 2024061221000276, 2024061421000021, 2024022721000148,2024072221000193
Bug group (optional): Regression, Workaround is available
Customer ID: 08507
Max CVSS v3 score:


Attachments
umc-client-close.patch (847 bytes, patch)
2024-05-27 09:36 CEST, Felix Botner
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Julia Bremer univentionstaff 2024-03-26 19:39:30 CET
It happened increasingly often at customers, that no user could change their password any more using the self-service or the UMC.
The UMC then displays
Password change failed. The reason could not be determined. In case it helps, the raw error message will be displayed. Errorcode 20: The password could not be saved. 


Looking into the management-console-server.log you see a lot of this:
18.03.24 07:53:16.995  AUTH        ( ERROR   ) : PAM: authentication error: ('Authentication token is no longer valid; new one required', 12)
18.03.24 07:53:36.132  AUTH        ( ERROR   ) : PAM: authentication error: ('Authentication token is no longer valid; new one required', 12)
18.03.24 07:53:36.132  AUTH        ( WARN    ) : Changing password failed (('Authentication token manipulation error', 20)). Prompts: [('Current Kerberos password: ', 1)]

So the password change already fails at the authentication step.

The krb5 logs complain about 
Decrypt integrity check failed for checksum type hmac-sha1-96-aes256, key type aes256-cts-hmac-sha1-96

It seems like malformed kerberos tickets are send by the PAM stack, maybe?
We haven't seen this problem live yet, so it is not clear where the problem lies.
In UMC, PAM or in Heimdal.

But usually, a UMC restart was enough to make the password change work again.


Important: This bug is not a catchall for all "Errorcode 20" errors.
If the UMC output contains "Unable to reach any changepw server in realm", this is not the bug for you.
Comment 10 Felix Botner univentionstaff 2024-05-24 14:48:57 CEST
(at least one) reason for this is the schoollists UMC module, every time this module is started one FD is created and not closed, even after the module terminates

In the customer env this module is regularly opened.


Can be reproduced with

# ucr set umc/module/timeout=3
# ucr set umc/http/processes=1 # also happens with 0 or >1, but more predictable with just one process

for i in $(seq 1 1030); do 
	wait_for=$((i%30))

	test $wait_for -eq 0 && sleep 4
	umc-command  -U Administrator -P univention schoollists/csvlist
done

even though this is an invalid request and the module is terminated after 3s, the number of open FD increases

$ lsof -p $(pidof -x univention-management-console-server)| grep type=STREAM| wc -l
1074

Now that i have more than 1024, i can no longer change the password via UMC.


$ umc-set -U "Administrator" -P "univention" -e -o '{"password": {"password": "univention", "new_password": "univention.99"}}'
Response: SET
  ---
  MIMETYPE : application/json
  STATUS   : 400
  OPTIONS  : password={'password': 'univention', 'new_password': 'univention.99'}
  MESSAGE  : Changing password failed. The reason could not be determined. In case it helps, the raw error message will be displayed: Errorcode 20: The new password could not be set.
  RESULT   : {'new_password': 'Changing password failed. The reason could not be determined. In case it helps, the raw error message will be displayed: Errorcode 20: The new password could not be set.'}

So i guess "schoollists" has to be fixed.
Comment 11 Felix Botner univentionstaff 2024-05-24 14:54:13 CEST
and if we try to change the password and have this krb5 config in place

$ more /root/.krb5/config
[logging]
krb5 = 0-/FILE:/tmp/krb.log

we get 

2024-05-24T14:52:37 error message: Did not find a plugin for ccache_ops: 2
2024-05-24T14:52:37 krb5_get_init_creds: loop 1
2024-05-24T14:52:37 KDC send 0 patypes
2024-05-24T14:52:37 error message: Did not find a plugin for send_to_kdc: 2
2024-05-24T14:52:37 Trying to find service kdc for realm SCHOOL.TEST flags 0
2024-05-24T14:52:37 error message: Did not find a plugin for service_locator: 2
2024-05-24T14:52:37 configuration file for realm SCHOOL.TEST not found
2024-05-24T14:52:37 searching DNS for realm SCHOOL.TEST udp.kerberos -> 0
2024-05-24T14:52:37 submissing new requests to new host
2024-05-24T14:52:37 error message: Did not find a plugin for send_to_kdc: 2
2024-05-24T14:52:37 fd too large for select
2024-05-24T14:52:37 searching DNS for realm SCHOOL.TEST tcp.kerberos -> 0
2024-05-24T14:52:37 submissing new requests to new host
2024-05-24T14:52:37 error message: Did not find a plugin for send_to_kdc: 2
2024-05-24T14:52:37 fd too large for select
2024-05-24T14:52:37 DNS lookup failed domain: _kerberos._http.SCHOOL.TEST.
2024-05-24T14:52:37 searching DNS for realm SCHOOL.TEST http.kerberos -> -1765328228
2024-05-24T14:52:37 fallback lookup 0 for realm SCHOOL.TEST (service kerberos)
2024-05-24T14:52:37 No KDC entries found for SCHOOL.TEST
2024-05-24T14:52:37 out of hosts, waiting for replies
2024-05-24T14:52:37 no more hosts to send/recv packets to/from trying to pulling more hosts
2024-05-24T14:52:37 error message: unable to reach any KDC in realm SCHOOL.TEST: -1765328228
2024-05-24T14:52:37 unknown-function SCHOOL.TEST done: -1765328228 hosts: 2 packets: 0 wc: 0.001631 nr: 0.000105 kh: 0.001346 tid: 00000000
Comment 12 Felix Botner univentionstaff 2024-05-24 15:14:56 CEST
probably not related to the schoollists UMC module, also happens with other modules 

$ umc-command  -U Administrator -P univention sysinfo/general
$ lsof -p $(pidof -x univention-management-console-server)| grep type=STREAM| wc -l
1083

$ umc-command  -U Administrator -P univention sysinfo/general
$ lsof -p $(pidof -x univention-management-console-server)| grep type=STREAM| wc -l
1084

no more a general UMC problem
Comment 13 Felix Botner univentionstaff 2024-05-27 09:36:03 CEST
Created attachment 11212 [details]
umc-client-close.patch

We use a "tornado.curl_httpclient.CurlAsyncHTTPClient" object for the connection between the server and module processes. Seems that we have to explicitly close this object in case of timeout to free all the resources.
Comment 15 Mirac Erdemiroglu univentionstaff 2024-06-03 13:01:27 CEST
Another customer affected 2024052921000077
Comment 16 Mirac Erdemiroglu univentionstaff 2024-06-12 16:46:19 CEST
Another customer affected 2024061221000276
Comment 17 Mirac Erdemiroglu univentionstaff 2024-06-14 10:51:42 CEST
Mit dem folgenden Befehl laden Sie die raw Daten herunter und speichern diese im File mit dem Namen self-service.patch. Danch erfolgt der Patch-Befehl mit dem Inhalt des self-service.patch zu dem richtigen Pfad des Skripts.

1. wget 'https://forge.univention.org/bugzilla/attachment.cgi?id=11212&action=diff&context=patch&collapsed=&headers=1&format=raw' -O self-service.patch
    
2. patch /usr/lib/python3/dist-packages/univention/management/console/resources.py < self-service.patch
    
3. systemctl restart univention-management-console-server.service
Comment 18 Dirk Wiesenthal univentionstaff 2024-06-14 17:03:42 CEST
Package: univention-management-console
Version: 12.0.34-2
Branch: 5.0-0
Scope: errata5.0-8
Comment 19 Dirk Wiesenthal univentionstaff 2024-06-14 18:01:02 CEST
Manual tests: OK
Jenkins: OK
YAML: OK